User interface for manipulating artificial intelligence experiments

Abstract

When training agents of artificial intelligence in, for example, a racing game environment, a user interface (UI) for running, tracking, and visualizing various aspects of a machine learning experiment may be trained using an analytical model. The UI may be network-based and may enable researchers to easily see the state of their experiments. The UI may include an experiment synchronization event viewer that may synchronize visual displays, videos, and timelines / metrics lines in the experiment. The viewer enables a researcher to see how the experiment is developed more detailedly. The UI may also include experimental event annotations that may generate event annotations. These annotations may be displayed via a synchronization event viewer. The UI may be used to account for merging results across experiments, and may also account for videos. For example, a UI may provide a reusable dial that may capture and compare metrics across multiple experiments.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to artificial intelligence training systems and methods. More specifically, the present invention relates to user interfaces for operating, monitoring, running, and evaluating artificial intelligence experiments such as machine learning experiments. Background Art

[0002] The following background information may present examples of specific aspects of the prior art (e.g., but not limited to methods, facts, or common knowledge), and while these examples are expected to help further educate the reader about additional aspects of the prior art, they are not to be construed as limiting the present invention or any embodiment thereof to anything stated or implied or inferred therefrom.

[0003] Video game players often look to improve their game by practicing and playing against other players. However, once a gamer develops significant skill in a given game, the availability of suitable challengers decreases significantly. While such a player may be able to improve their game by playing against less skilled players, it is often more helpful to play against players who can provide a significant challenge.

[0004] Many games offer players that the game provides who can participate. However, these players may just follow certain procedures that a skilled player can figure out and beat.

[0005] In view of the above, there is a need for a user interface for analyzing model training runs, tracking, and visualizing various aspects of machine learning experiments that can be used when training artificial intelligence agents with capabilities that challenge even the most skilled video game players. Summary of the invention

[0006] An embodiment of the present invention provides a method for providing a user interface for analyzing model training runs and tracking and visualizing various aspects of machine learning experiments, the method comprising: displaying a timeline of selected metrics of the machine learning experiment; displaying a video synchronized with a selected portion of the timeline; and displaying a visualization tool that shows a global representation of data collected from it during the machine learning experiment.

[0007] Embodiments of the present invention also provide a method for providing a user interface for analyzing model training runs and tracking and visualizing various aspects of machine learning experiments, the method comprising: displaying a video of a visual representation available in data collected by a data collector of the machine learning experiment; and displaying a visualization tool that provides a global representation of the data collected from it during the machine learning experiment, wherein the visualization tool includes one or more event annotations of one or more key events from the machine learning experiment.

[0008] An embodiment of the present invention also provides a user interface for a computing architecture of a machine learning training system for training an artificial intelligence agent for a racing game, the user interface comprising: displaying a timeline of selected indicators of the machine learning experiment; displaying a video synchronized with a selected portion of the timeline; displaying a visualization tool showing a global representation of data collected during the machine learning experiment; and displaying one or more event annotations of one or more key events from the machine learning experiment on the visualization tool.

[0009]

[0011] These and other features, aspects and advantages of the present invention will become better understood with reference to the following drawings, description and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Some embodiments of the present invention are illustrated by way of example and not limitation in the figures of the accompanying drawings in which like reference numerals may indicate similar elements.

[0011] Figure 1 illustrates an overview page of a user interface showing a list of resources, active experiments, and completed experiments according to an exemplary embodiment of the present invention;

[0012] Figure 2 A screen shot of a user interface showing parallel execution of Figure 2 138 active experiments in this activity experiment in the example;

[0013] Figure 3 FIG. 1 shows an exemplary embodiment of the present invention. Figure 1 How the search bar in the user interface can be used to filter experiments;

[0014] Figure 4 The diagram shows the user interface used for all completed experiments. Figure 4 In the example, the number of completed experiments is 34,921 completed experiments;

[0015] Figure 5 illustrates an experiment overview page of a user interface according to an exemplary embodiment of the present invention, the experiment overview page showing key features about an experiment;

[0016] Figure 6 illustrates an experiment overview page of a user interface according to an exemplary embodiment of the present invention, the experiment overview page showing artifacts (aritfact) and videos below an overview dial;

[0017] Figure 7illustrates an experiment overview page of a user interface showing a list of trainers and data collectors that are part of an experiment below artifacts and videos according to an exemplary embodiment of the present invention;

[0018] Fig. 8A and Figure 8B illustrates an exemplary trainer / data collector log page of a user interface according to an embodiment of the present invention;

[0019] Fig. 9 illustrates an example experimental dial page of a user interface according to an exemplary embodiment of the present invention;

[0020] Fig.10 An exemplary dial with controls for comparing multiple experiments and metrics is illustrated;

[0021] Fig.11 illustrates a user interface for updating a script that populates a widget;

[0022] Fig.12 An experimental synchronization event viewer of a user interface according to an exemplary embodiment of the present invention is illustrated;

[0023] Fig.13 Illustration of a vehicle (vehicle 7) coming from a vehicle before it leaves the road in an example racing game. Fig.12 An exemplary visualization tool for; and

[0024] Fig.14 The diagram shows the vehicle 7 after it has left the road. Fig.13 A visualization tool showing small colored spheres as event annotations for indicating off-road events.

[0025] Unless otherwise indicated, the illustrations in these figures are not necessarily drawn to scale.

[0026] Now, by turning to the following detailed description, the present invention and its various embodiments can be better understood, wherein the illustrated embodiment is described. It is to be clearly understood that the illustrated embodiment is set forth as an example and not as a limitation of the present invention, as ultimately defined in the claims. DETAILED DESCRIPTION

[0027] The terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. As used herein, the term "and / or" includes any and all combinations of one or more associated listed items. As used herein, the singular forms "one", "an" and "the" are also intended to include plural forms in addition to the singular forms, unless the context clearly indicates otherwise. It should also be understood that the term "includes" and / or its variations when used in this specification specify the presence of the features, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, steps, operations, elements, parts and / or groups thereof.

[0028] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology and the present disclosure, and should not be interpreted in an idealized or overly formal sense, unless so explicitly defined herein.

[0029] In describing the present invention, it will be understood that a plurality of techniques and steps are disclosed. Each of these has independent benefits, and each may also be used in combination with one or more or in some cases all of the other disclosed techniques. Therefore, for the sake of clarity, this specification will avoid repeating each possible combination of the various steps in an unnecessary manner. However, it should be understood when reading the specification and claims that such combinations are fully within the scope of the present invention and claims.

[0030] In the following description, for the purpose of explanation, numerous specific details are set forth in order to provide a more thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention can be practiced without these specific details.

[0031] The present disclosure is to be considered as an exemplification of the invention and is not intended to limit the invention to the specific embodiments illustrated by the following figures or description.

[0032] Devices or system modules that are at least generally in communication with each other need not be in continuous communication with each other unless explicitly specified otherwise. In addition, devices or system modules that are at least generally in communication with each other may communicate directly or indirectly through one or more intermediaries.

[0033] The description of an embodiment with multiple components in communication with each other does not imply that all such components are required. Rather, various optional components are described to illustrate various possible embodiments of the present invention.

[0034] A "computer" or "computing device" may refer to one or more devices and / or one or more systems that can accept structured input, process the structured input according to specified rules, and produce processed results as output. Examples of computers or computing devices may include: computers; fixed and / or portable computers; computers with a single processor, multiple processors that can operate in parallel and / or non-parallel, or multi-core processors; supercomputers; mainframes; supermicrocomputers; workstations; microcomputers; servers; clients; interactive televisions; network equipment; telecommunications equipment with Internet access; hybrid combinations of computers and interactive televisions; portable computers; tablet personal computers (PCs); personal digital assistants (PDAs); portable phones; special-purpose hardware for emulating computers and / or software, such as (for example) digital signal processors (DSPs), field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific instruction set processors (ASIPs), one chip, multiple chips, systems on chips, or chipsets; data acquisition equipment; optical computers; quantum computers; biological computers; generally, a device that can accept data, process the data according to one or more stored software programs, generate results, and generally includes input, output, storage, operation, logic, and control units.

[0035] "Software" or "application" may refer to the specified rules for operating a computer. Examples of software or applications may include code segments in one or more computer-readable languages; graphical and / or textual instructions; applets; precompiled code; translated code; compiled code; and computer programs.

[0036] These computer program instructions may also be stored in a computer-readable medium, which may direct a computer, other programmable data processing apparatus, or other device to function in a specific manner so that the instructions stored in the computer-readable medium produce a product including instructions for implementing the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0037] In addition, although the processing steps, method steps, algorithms, etc. may be described in a sequential order, these processes, methods, and algorithms may be configured to work in an alternative order. In other words, any order or sequence of steps that may be described does not necessarily indicate a requirement to perform the steps in that order. The steps of the processes described herein may be performed in any order that is practical. In addition, some steps may be performed simultaneously.

[0038] It will be readily apparent that the various methods and algorithms described herein can be implemented by, for example, appropriately programmed general-purpose computers and computing devices. Typically, a processor (e.g., a microprocessor) will receive instructions from a memory or similar device and execute these instructions, thereby performing the processing defined by these instructions. In addition, various known media can be used to store and transmit programs that implement such methods and algorithms.

[0039] As used herein, the term "computer-readable medium" refers to any medium that participates in providing data (e.g., instructions) that can be read by a computer, a processor, or a similar device. Such a medium can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical or magnetic disks and other permanent memories. Volatile media include dynamic random access memory (DRAM) that usually constitutes the main memory. Transmission media include coaxial cables, copper wires, and optical fibers, including wires that include a system bus connected to the processor. Transmission media may include or transmit sound waves, light waves, and electromagnetic emissions such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs, any other optical media, punch cards, paper tapes, any other physical media with hole patterns, RAMs, PROMs, EPROMs, FLASH EPROMs, any other storage chips or cassettes, carrier waves as described below, or any other media from which a computer can read.

[0040] Various forms of computer-readable media may be involved in carrying the sequence of instructions to the processor. For example, the sequence of instructions (i) may be transferred from a RAM to the processor, (ii) may be carried via a wireless transmission medium, and / or (iii) may be formatted according to numerous formats, standards, or protocols such as Bluetooth, TDMA, CDMA, 3G, 4G, 5G, etc.

[0041] Embodiments of the present invention may include means for performing the operations disclosed herein.A device may be specially constructed for the intended purposes, or it may comprise a general purpose device selectively activated or reconfigured by a program stored in the device.

[0042] Unless otherwise specifically stated, and as will become apparent from the following specification and claims, it should be understood that throughout the specification, terms such as "processing," "computing," "calculating," "determining," etc., are utilized to refer to the action and / or processing of a computer or computing system or similar electronic computing device that manipulates data represented as physical (e.g., electronic) quantities such as within a computer system's registers and / or memories and transforms it into other data similarly represented as physical quantities within a computer system's memories, registers, or other such information storage, transmission, or display devices.

[0043] In a similar manner, the term "processor" may refer to any device or part of a device that processes electronic data from registers and / or memory to convert the electronic data into other electronic data that can be stored in registers and / or memory or can be transmitted to an external device to cause a physical change or actuation of an external device. The term "agent" or "intelligent agent" or "artificial intelligence agent" is intended to refer to any artificial entity that selects an action in response to an observation. "Agent" may refer to, without limitation, a robot, a simulated robot, a software agent or a "bot", an adaptive agent, an Internet or networked humanoid robot.

[0044] In summary, embodiments of the present invention provide a user interface for analyzing model training runs, tracking, and visualizing various aspects of machine learning experiments that can be used when training an agent of artificial intelligence in, for example, a racing game environment. The user interface can be web-based and can enable researchers to easily see the status of their experiments. The user interface can include an experiment synchronization event viewer that can synchronize visual displays, videos, and timeline / metric graphs in the experiment. The viewer enables researchers to see in more detail how the experiment unfolds. The user interface can also include experiment event annotations that can generate event annotations. These annotations can be displayed via the synchronization event viewer.

[0045] In some embodiments, the user interface can be used to consider integrated results across experiments, and can also consider videos. For example, the user interface can provide a reusable dial that can capture metrics across multiple experiments and compare them; the user interface can provide a graphical representation of how experiments are linked together; and the user interface can be used as an integrated video player that weaves multiple videos for seamless viewing. More specifically, in some embodiments, the user interface can include the ability to capture video from a game console in the cloud and display it in the user interface.

[0046] As used herein, the term "environment" or "data center" refers to a cluster of resources. In the example described below, three static environments and one auto-scaling environment are described.

[0047] As used herein, the term "experiment" refers to a single experiment conducted by a single trainer and multiple data collectors. In most figures, it is referred to as a "run".

[0048] As used herein, the term "trainer" refers to a single centralized trainer that trains a model using data collected by a data collector. There is typically one trainer for an experiment.

[0049] As used herein, the term "data collector" refers to a distributed worker that collects data. An experiment typically has multiple data collectors. In most screenshots described herein, data collectors are represented as "rollout workers" or "R".

[0050] Reference Figures 1 to 3 , the upper frame 10 shows the resources (current usage / total available) across multiple cloud data centers / environments available in the user interface system. The resources are shown together with the usage amount and the total available amount. Figure 1 In the example of , there are four separate environments, namely fixed environments 1-3 and an autoscaling environment. From left to right, each environment includes the number of graphics processing units (GPUs), central processing units (CPUs) on GPU machines, memory on GPU machines, CPUs on non-GPU machines, and memory on non-GPU machines, any of which may become a bottleneck for starting a new experiment. Figure 1 As can be seen in FIG. 1 , such resource usage can be easily monitored in a user interface according to an embodiment of the present invention.

[0051] As shown below relative to Figure 3 As discussed in more detail, the list of experiments can be filtered using the search bar 12. Figure 2 As discussed in more detail, the lower frame 14 shows the actively running experiments in pagination.

[0052] specifically refer to Figure 2 The first column shows the experiment name, with a link to the experiment page, e.g. Figure 5 , and discussed in more detail below. The second column shows the name of the user who submitted the experiment. The third column shows the status of the experiment, which can include states such as (1) Building - an image containing the user's code is being built for the experiment; (2) Queued - an experiment with the built image is waiting for resources; or (3) Running - the experiment is actively running in one or more environments and consuming resources. The fourth column shows the time when the experiment started.

[0053] The fifth column shows the resources used by the experiment and which environments these resources come from. In this column, on the left hand side, the first and second rows can show the number of GPUs and CPUs used by the experiment trainer, while the third row shows the number of data collectors being used in the experiment. Depending on the experiment the user is running, each data collector may or may not communicate with the game console. Some experiments are run in simulations that do not require a game console. The right side of the fifth column shows which environments the trainer (T) and data collector (R) are running in. An experiment can be divided into multiple environments. For example, Figure 2 The third experiment in uses resources from three environments. The GPU type that the experiment trainer is using is also shown, for example, the GPU type can be Nvidia VI 00 or Nvidia A 100 GPU.

[0054] Figure 2 The sixth column shows the labels that the user uses to categorize the runs. A single experiment can provide multiple labels. The user can give labels arbitrary colors. The seventh column shows a text description of the experiment.

[0055] Reference Figure 3 , the search bar 14 may enable the user to filter experiments. The user may filter experiments by, for example, full or partial experiment name, full or partial username of the user running the experiment, full or partial tags used by one or more experiments, or full or partial descriptions used by one or more experiments. When filtering, the resource view shows three values ​​per entry—resources used by the filtered experiments, resources used by all experiments, and all available resources.

[0056] Reference Figure 4 , you can display all completed experiments. Typically, these are displayed below the active experiment. Typically, the same information as provided for the active experiment can be displayed (see above for Figure 2 Discussion), except that the experiment status can take one of three values: (1) Success; (2) Error - if an error during the experiment execution caused it to fail prematurely; or (3) Cancelled - the user manually canceled the experiment. The Completed Experiments display can also provide the total duration of each experiment.

[0057] Reference Figure 5 , showing an experiment overview page of a user interface. The experiment overview page may show key features about the experiment.

[0058] The first portion 50 of the display may show the following: (1) the username of the user running the experiment; (2) the current state of the experiment; (3) a link to the configuration showing the running experiment (config.yml); (4) a link to the code repository containing the experiment code (git); (5) the script used to run the experiment; (6) the experiment start time; and (7) the experiment duration. This will be different from the start time after the experiment is completed.

[0059] The second portion 52 of the display may show the experimental provenance. The system may easily enable the user to use a model generated from one experiment in another experiment. This is useful to the user because they may need to design the next experiment in a series of experiments after analyzing the results of a previous experiment. Each successive experiment is designed independently because it may not be possible to design it a priori. The user interface may provide the user with a simple graphical representation that allows tracking and easy navigation to experiments that build on each other, and may display experiments that contribute to the current experiment as well as new experiments that use the results of the current experiment. Figure 5 In the screenshot of , the current experiment uses models and artifacts from four previous experiments. The user can click on a predecessor or successor run in the graphic and open that experiment page. When the list of predecessors is long, a condensed summary can be presented.

[0060] The third portion 54A of the display may show all resources used by the experimental trainer. This includes the number of CPUs, the number of GPUs, and system memory. If the trainer itself interacts directly with the game console, that number is also displayed. Typically, the trainer does not interact directly with the game console, but has the ability to do so if necessary.

[0061] The fourth section 54B shows all resources for use by a single data collector and the number of such data collectors. This includes the number of CPUs, the number of GPUs, and system memory. If the data collector interacts with a game console, the number of game consoles that the single data collector interacts with is also displayed. Depending on the experiment the user is running, each data collector may or may not communicate with a game console. Some experiments are run in simulations that do not require a game console.

[0062] The fifth section 56 shows the status event history of the experiment and may include states such as Building, Queued, Running, and Paused.

[0063] The sixth section 58 shows the main overview dial of the experiment. This section 58 may be configurable by the user. The overview dial displays key indicators that help the user quickly determine how the experiment is going. As discussed in more detail below, an experiment may have multiple dials, and the user may edit the dashboard to include information such as Figure 5 An overview dial is shown in FIG. 5 , in which five graph lines are shown.

[0064] Reference Figure 6 , users can configure experiments to store arbitrary artifacts - these can include things like videos, checkpoints, and models. An expandable list of artifacts is displayed on the left side of the user interface. The entire list of artifacts can also be searched.

[0065] On the right side of the user interface, an optional video player is included. If the experiment saves videos and information about those videos, the user interface can parse that information and display the video. The video can show any visual representation available in the data collected by the data collector. The video can be generated by a simulation application being run by the trainer or data collector, or it can be a video feed from a game console when communicating.

[0066] Additionally, the user interface can enable information from multiple video sources to be woven together into a single coherent video stream. For example, video files are often each chunked in 2-5 minute intervals, and a researcher may need to analyze an experiment across video boundaries. The user interface can enable a user to seamlessly transition from one video to another across video files. The video player can also be viewed on a separate "Experiment Video" page if desired.

[0067] Reference Figure 7 , the experiment overview page can show a list of trainers and data collectors that are part of the experiment below the artifacts and videos. First, a list of trainers can be displayed. Typically, there is a single trainer - but an experiment can use more than one trainer. Next, a paginated list of data collectors can be displayed.

[0068] Each row contains the following information: (1) the name of the trainer or data collector, linked to the log from it (as discussed further below with reference to the log page); (2) a bar graph that visually represents how long the trainer or data collector has been running, as well as the time it started and the time it was last observed. The starting point and length of the bar graph depend on the total duration of the experiment. If a data collector fails partway through the experiment, the user interface can start a new data collector in its place. The failed data collector can take up the left half of the total bar width, and the new data collector can take up the right half of the total bar width; (3) a textual description of the same information in the bar; (4) the status of the trainer or data collector. Typically, the status can be one of "running", "failed", or "cancelled"; (5) a complete description of the status; and (6) the environment in which the trainer or data collector is running.

[0069] Reference Fig. 8A and Figure 8B , shows the trainer / data collector log page. The trainer or data collector log page shows a list of log lines from that component. The log page helps users and engineers better analyze experiments via text records. Some key features of the log page include (1) the user can select the source (trainer or specific data collector) from the top of the log page; (2) the user can specify the duration they want to extract logs; (3) the user can also specify optional filters when extracting logs. Fig. 8A and Figure 8B In the A filter is provided by an open source software called a "reverb server". Other filtering software may be used as appropriate. The log includes only the lines where the filter exists. The user interface may also support negated filters, where all lines not containing the filter are displayed; and (4) options may be provided that enable specification of how many log lines to extract and whether to extract / display them in ascending / descending order. Typically, the log page may automatically refresh when more logs are available.

[0070] Reference Fig. 9 , shows an exemplary experiment dial page. An experiment can include one or many dials, and the user interface can enable a user to create multiple reusable dials that can be easily applied to one or more experiments. Since a single experiment can have multiple dials, each dial enables the user to understand the experiment from a different perspective.

[0071] The dial itself can include multiple widgets. Each widget can display the following: (1) a graphical representation of the metrics recorded by the experiment; (2) a table containing summary information from the run; and (3) any text that the user wishes to be displayed there.

[0072] The dial also enables the user to set drop-down "controls". The user can use one or many controls to select an experiment name, an experiment label, and a metric name. These controls provide interactivity in the dial, allowing the user to dynamically compare different experiments and different metrics as needed. Fig.10 An example of a controlled dial is shown in .

[0073] Each widget can be driven by a user script, allowing the user to easily control the widget. Fig.11 Screen shot showing a user editing a script that controls a particular widget.

[0074] Reference Fig.12 , shows the experimental synchronization event viewer. Figure 6Video from the video player of the description can also be built into a separate synchronized viewer page so that the indicators, visual displays and video can be synchronized up to the timeline of events, e.g. Fig.12 This feature provides researchers with a synchronized, comprehensive view of the data collected during an experiment.

[0075] In this view, (1) the video on the lower left provides a specific view. In this specific example of a racing game, the video shows video from a specific vehicle, and any vehicle in the collected data can be selected; (2) the visual display on the upper right provides a global view. The visualizer shows a global representation of the application from which data was collected during the experiment. In this specific example of a racing game, a representation of the track, eight vehicles and their trajectories is shown. Event annotations can also be displayed on this visual display, as described in detail below; (3) Both the video and the visual display are synchronized with the timeline on the upper left. In this specific example of a racing game, the timeline shows vehicle positions. In general, a timeline can show any arbitrary metric or event annotation from an experiment over time. The user can navigate to a certain moment by clicking on the timeline.

[0076] Provides deep analytics in the machine learning user interface by combining synchronized experiment visual displays, videos, and metrics.

[0077] Reference Fig.13 and Fig.14 In addition, the user interface system can analyze the experiment even further and generate event annotations. After the experiment is completed, the key events in the data stream can be illustrated. In a racing game, the key events may look like the following: position gained, position lost, car off the track, etc. However, events can be generated for any data stream collected by the data collector for any application. The event annotations can then be displayed on the visualization tool.

[0078] On the upper right visualizer, various event annotations are shown over time. In the racing game example, Fig.12 and Fig.13 The diagrams show the vehicle 7 before and after it deviates from the road. Fig.13 Off-road event annotations can be displayed in the visual display via a sphere of a specific color. Other events can also be visualized similarly.

[0079] The above user interface can be used in a system for training machine learning models such as a racing game artificial intelligence player. The user interface can be part of a training system computing architecture as described, for example, in U.S. Patent Application No. 17 / 650,275, the contents of which are incorporated herein by reference.

[0080] Various changes and modifications may be made by those of ordinary skill in the art without departing from the spirit and scope of the present invention. Therefore, it must be understood that the illustrated embodiments are set forth for illustrative purposes only, and they should not be taken as limiting the present invention as defined in the following claims. For example, despite the fact that the elements of the claims are set forth below in some combination, it must be clearly understood that the present invention includes other combinations of fewer, more, or different elements in the disclosed elements.

[0081] The words used in this specification to describe the present invention and its various embodiments will be understood not only in the sense of their commonly defined meanings but also will include their common structures, materials, or actions representing a single species through special definitions in this specification.

[0082] Therefore, in this specification, the definitions of the words or elements of the following claims include not only the combinations of elements literally set forth. Therefore, in this sense, equivalent substitutions of two or more elements may be made for any one element in the following claims, or substitutions of a single element may be made for two or more elements in the claims. Although elements may be described above as functioning in certain combinations and even initially declared as such, it is to be clearly understood that in some cases, one or more elements in a declared combination may be deleted from the combination, and that a declared combination may involve sub-combinations or variations of sub-combinations.

[0083] Insubstantial changes to the claimed subject matter as seen by one of ordinary skill in the art, as now known or later devised, are expressly considered to be equivalently within the scope of the claims. Therefore, obvious substitutions now or later known to one of ordinary skill in the art are defined to be within the scope of the defined elements.

[0084] Therefore, the claims should be understood to include the contents specifically illustrated and described above, contents that are conceptually equivalent, contents that can be obviously substituted, and contents that contain the basic idea of ​​the present invention.

Claims

1. A method for providing a user interface for analyzing model training runs and tracking and visualizing aspects of machine learning experiments, the method comprising: Displaying a timeline of selected metrics of the machine learning experiment; displaying a video synchronized with a selected portion of the timeline; as well as A visualizer is displayed that shows a global representation of data collected during the machine learning experiment.

2. The method according to claim 1, wherein: The video is generated by a simulated application run by a trainer or data collector.

3. The method according to claim 1, wherein: The video is the video feed when communicating from a cloud-based gaming console.

4. The method according to claim 1, wherein: The video includes information from multiple video sources woven together in a single coherent video stream.

5. The method according to claim 1, wherein: The machine learning experiment was in the context of a racing game.

6. The method according to claim 5, wherein: The video shows vehicle video from a selected vehicle participating in the racing game.

7. The method according to claim 5, wherein: The visualization tool shows a representation of the racing track, each car, and the trajectory of each car.

8. The method according to claim 5, wherein: The timeline illustrates vehicle positions within the racing game environment.

9. The method of claim 1, further comprising displaying one or more event annotations of one or more key events from the machine learning experiment on the visualization tool.

10. The method according to claim 9, wherein: The one or more key events include a racing game key event including one or more of a position gained, a position lost, or a car going off the track.

11. The method according to claim 1, wherein: The visualization tool includes one or more event annotations of one or more key events from the machine learning experiment.

12. A method for providing a user interface for analyzing model training runs and tracking and visualizing aspects of machine learning experiments, the method comprising: providing a user with an interface for generating a plurality of user dial pages for visualizing an experiment, each user dial page comprising one or more dials, each of the one or more dials providing an alternative perspective on the experiment, wherein the plurality of user dial pages can be reused in a plurality of experiments; Wherein, at least one of the one or more dials provides a user with a graphically based representation for tracking and navigating to experiments that build upon each other, the at least one dial displaying one or more experiments that contribute to the current experiment and new experiments that use results from the current experiment.

13. The method of claim 12, further comprising displaying a timeline of selected metrics of the machine learning experiment. The method of claim 13 , further comprising synchronizing the video with the timeline.

15. The method of claim 14, further comprising displaying a visualization tool showing a global representation of data collected during the machine learning experiment.

16. The method according to claim 15, wherein: The machine learning experiment was in the context of a racing game.

17. The method according to claim 16, wherein: The visualization tool shows a representation of the racing track, each car, and the trajectory of each car.

18. The method according to claim 16, wherein: The timeline illustrates vehicle positions within the racing game environment.

19. A user interface for a machine learning training system computing architecture for training a racing game artificial intelligence agent, the user interface comprising: Displaying a timeline of selected metrics of the machine learning experiment; displaying a video synchronized with a selected portion of the timeline; displaying a visualization tool showing a global representation of data collected during the machine learning experiment; as well as One or more event annotations of one or more key events from the machine learning experiment are displayed on the visualization tool.

20. The method of claim 19, further comprising displaying an experimental watch face page with one or more watch faces, wherein: The one or more watch faces are user created watch faces suitable for a single experiment or multiple experiments.

Images