User interface for operating an artificial intelligence experiment

Abstract

For example, when training an artificial intelligence agent in a racing game environment, a user interface (UI) can be used to analyze the execution of model training and track and visualize various aspects of the machine learning experiment. The UI can be web-based and can enable researchers to easily view their experimental situation. The UI can include an experimental synchronization event viewer that can synchronize visualizations, videos, and timeline / metric graphs in the experiment. This viewer enables researchers to view in detail how the experiment unfolds. The UI can further include an experimental event annotation that can generate event annotations. These annotations can be displayed via the synchronized event viewer. The UI can be used to examine the combined results of the entire experiment and can also examine videos. For example, the UI can provide a reusable dashboard that can import metrics across multiple experiments and compare them.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to artificial intelligence training systems and methods. Specifically, the present invention relates to a user interface for operating, monitoring, executing, and evaluating artificial intelligence experiments such as machine learning experiments.

Background Art

[0002] The following background information can present examples of specific aspects of the prior art (e.g., without limitation, techniques, facts, or general concepts), and these examples are expected to help convey further aspects of the prior art to the reader, but the present invention or any of its embodiments should not be construed as being limited to any matter mentioned or implied within these examples, or inferred from these examples.

[0003] Video game players often desire to improve their games through practice and playing against other players. However, when a certain game player acquires excellent skills in a given game, the availability of challengers commensurate with those skills significantly decreases. Such players may be able to improve their games by playing against less skilled players, but usually it is more beneficial to play against challenging players.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Many games provide game-provided players that can participate. However, such players may simply be following specific programming, and skilled players may be able to figure out the programming and defeat them.

[0006] Therefore, there is a need for a user interface that can be used when training an artificial intelligence agent to have the ability to challenge even the most skilled video game players, which analyzes the execution of model training, tracks and visualizes various aspects of machine learning experiments.

Means for Solving the Problem

[0007] Embodiments of the present invention are methods for providing a user interface that analyzes the execution of model training and tracks and visualizes aspects of machine learning experiments, including displaying a timeline of selected metrics of a machine learning experiment, displaying a video synchronized with a selected portion of the timeline, and displaying a visualizer showing the global representation that is the source of the data during the machine learning experiment.

[0008] Embodiments of the present invention are methods for providing a user interface that analyzes the execution of model training and tracks and visualizes aspects of machine learning experiments, including displaying a video of a visual representation available in the data collected by a data collector of a machine learning experiment, and displaying a visualizer providing the global representation that is the source of the data during the machine learning experiment, and further providing a method in which the visualizer includes one or more event annotations for one or more key events from the machine learning experiment.

[0009] Embodiments of the present invention are user interfaces of a machine learning training system computing architecture for training a racing game artificial agent, including displaying a timeline of selected metrics of a machine learning experiment, displaying a video synchronized with a selected portion of the timeline, displaying a visualizer showing a global representation that is a source of data during the machine learning experiment, and displaying one or more event annotations for one or more key events from the machine learning experiment on the visualizer.

[0010] These and other features, aspects, and advantages of the present invention will be better understood with reference to the following drawings, description, and claims.

[0011] Some embodiments of the present invention are shown by way of example and not limitation in the figures of the accompanying drawings in which like reference numerals can indicate like elements.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8A

Figure 8B

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

[0013] Unless otherwise indicated, the illustrations in the figures are not necessarily to scale.

[0014] A better understanding of the present invention and its various embodiments can be achieved by referring to the following detailed description that explains the illustrated embodiments. It should be clearly understood that the illustrated embodiments are shown as examples and do not ultimately limit the invention as defined in the claims.

[0015] The terms used in this specification are for the purpose of describing particular embodiments only and are not intended to limit the present invention. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items. The singular forms "a," "an," and "the" as used in this specification are intended to include the plural forms as well, unless the context clearly indicates otherwise. Further, the terms "comprises" and / or "comprising" when used in this specification indicate the presence of the recited features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0016] Unless otherwise defined, all terms (including technical and scientific terms) used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Further, terms defined as in a commonly used dictionary should be interpreted to have a meaning consistent with their meaning in the context of the relevant art and this disclosure, and should not be interpreted in an idealized or overly formal sense unless clearly defined otherwise in this specification.

[0017] In the description of the present invention, it will be understood that a plurality of techniques and steps are disclosed. Each of these has its own individual advantages and can be used either alone or in combination with one or more, or in some cases all, of the other techniques disclosed. Therefore, in this specification, for the sake of clarity, we refrain from unnecessarily repeating all possible combinations of the individual steps. However, for this specification and the claims, such combinations should be read with the understanding that they are fully included within the scope of the present invention and the claims.

[0018] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one of ordinary skill in the art that the present invention may be practiced without these specific details.

[0019] This disclosure should be regarded as exemplary of the present invention and is not intended to limit the present invention to the specific embodiments shown by the figures and description.

[0020] Devices or system modules that at least generally communicate with each other do not necessarily need to communicate continuously with each other unless otherwise explicitly stated. Also, devices or system modules that at least generally communicate with each other can communicate directly or indirectly through one or more intermediate devices.

[0021] The description of embodiments that include a plurality of components that communicate with each other does not mean that all such components are necessary. Rather, various optional components are described to illustrate the wide variety of possible embodiments of the present invention.

[0022] "Computer" or "computer device" can mean one or more devices and / or one or more systems that can accept structured input, process the structured input according to defined rules, and generate the result of the processing as output. Examples of a computer or computer device include a computer, a fixed and / or portable computer, a single processor, multiple processors, or a computer having a multi-core processor that can operate in parallel and / or not in parallel, a supercomputer, a mainframe, a super-minicomputer, a minicomputer, a workstation, a microcomputer, a server, a client, a two-way television, a web appliance, a communication device having Internet access, a hybrid combination of a computer and a two-way television, a portable computer, a tablet personal computer (PC), a personal digital assistant (PDA), a mobile phone, for example, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an application specific instruction set processor (ASIP), a chip, multiple chips, a system on chip or a chip set, etc., a special-purpose hardware that emulates a computer and / or software, a data collector, an optical computer, a quantum computer, a bio-computer, and generally a device that can accept data, process the data according to one or more stored software programs, generate a result, and typically can include an input device, an output device, a storage device, an arithmetic unit, a logic unit, and a control unit.

[0023] "Software" or "application" can mean defined rules for operating a computer. Examples of software or an application include code segments in one or more computer-readable languages, graphic and / or text instructions, applets, pre-compiled code, interpreted code, compiled code, and computer programs.

[0024] Also, by storing these computer program instructions, which can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, in a computer-readable medium, an article of manufacture can be created in which the instructions stored in the computer-readable medium include instructions for performing the functions / operations specified in one or more blocks of a flowchart and / or block diagram.

[0025] Furthermore, although process steps, method steps, or algorithms, etc. may be described in a certain order, such processes, methods, and algorithms can also be configured to function in a different order. In other words, any order or sequence of steps that can be described does not necessarily indicate that these steps must be performed in this order. The process steps described herein can be performed in any practical order. Additionally, some steps can be performed simultaneously.

[0026] It will be readily apparent that the various methods and algorithms described herein can be executed, for example, by appropriately programmed general-purpose computers and computing devices. Typically, a processor (e.g., a microprocessor) receives instructions from memory or a similar device and executes these instructions to perform the processes defined by these instructions. Additionally, programs for executing such methods and algorithms can be stored and transmitted using various known media.

[0027] As used herein, the term "computer-readable medium" means any medium that participates in providing data (e.g., instructions) that can be read by a computer, a processor, or a similar device. Such a medium can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks and other permanent memories. Volatile media typically includes dynamic random access memory (DRAM) that constitutes main memory. Transmission media includes coaxial cables, copper wire, and fiber optics, including wires that include a system bus coupled to a processor. Transmission media can include or convey acoustic waves, light waves, and electromagnetic radiation, such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer-readable media can include floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic media, CD-ROM, DVD, any other optical media, punch cards, paper tape, any other physical media with hole patterns, RAM, PROM, EPROM, FLASHEEPROM, any other memory chip or cartridge, carrier waves as described hereinafter, or any other media that can be read by a computer.

[0028] Various forms of computer-readable media can be involved in conveying a series of instructions to a processor. For example, a series of instructions can be supplied to the processor from (i) RAM, (ii) conveyed via a wireless transmission medium, and / or (iii) formatted according to numerous formats, standards, or protocols such as Bluetooth, TDMA, CDMA, 3G, 4G, and 5G.

[0029] Embodiments of the present invention can include an apparatus that performs the operations disclosed herein. The apparatus can include a specially configured device for a desired purpose or can include a general-purpose device selectively activated or configured by a program stored therein.

[0030] Unless otherwise specified, and as may be apparent from the following description and claims, throughout this specification, descriptions using terms such as "process", "calculate", "compute", or "determine" refer to operations and / or processes of a computer, computer system, or similar electronic computing device that manipulate data represented as physical quantities such as electronic quantities in the registers and / or memories of the computer system and / or transform the data into other data similarly represented as physical quantities in the memories, registers, or other such information storage, transmission, or display devices of the computer system.

[0031] Similarly, the term "processor" can mean any device or part of a device that processes electronic data from registers and / or memories and can convert this electronic data into other electronic data that can be stored in registers and / or memories or transmitted to external devices to cause physical changes or operations of the external devices. The terms "agent" or "intelligent agent" or "artificial agent" or "artificial intelligence agent" are intended to mean any artificial entity that selects actions in response to observations. "Agent" can mean, without limitation, robots, simulated robots, software agents or "bots", adaptive agents, Internet or web bots.

[0032] Generally, embodiments of the present invention provide a user interface that can be used, for example, when training an artificial intelligence agent in a racing game environment, to analyze the execution of model training and track and visualize various aspects of a machine learning experiment. The user interface can be web-based and can enable researchers to easily view their experimental situation. The user interface can include an experiment synchronized event viewer that can synchronize visualizations, videos, and timeline / metric graphs in an experiment. This viewer enables researchers to view in detail how an experiment unfolds. The user interface can further include experimental event annotations that can generate event annotations. These annotations can be displayed via the synchronized event viewer.

[0033] In some embodiments, the user interface can be used to examine the consolidated results of an entire experiment, and the user interface can also examine videos. For example, the user interface can provide a reusable dashboard that can import metrics across multiple experiments and compare them, can provide a graphical representation of how the experiments are linked, and can be used as a concatenated video player that combines multiple videos for seamless viewing. Specifically, in some embodiments, the user interface can include the function of importing videos from game machines in the cloud and displaying them within the user interface.

[0034] As used herein, the terms "environment" or "data center" mean a cluster of resources. In the examples described below, three static environments and one auto-scaling environment will be described.

[0035] As used herein, the term "experiment" means a single experiment that includes a single trainer and multiple data gatherers. In most figures, this experiment is referred to as a "Run".

[0036] As used herein, the term "trainer" means a single centralized trainer that trains a model using data collected by data gatherers. Typically, there is one trainer per experiment.

[0037] As used herein, the term "data gatherer" means a distributed worker that collects data. Typically, there are multiple data gatherers per experiment. In most of the screenshots described herein, the data gatherer is represented as a "rollout worker" or "R".

[0038] Referring to FIGS. 1 - 3, the upper box 10 shows the (current / overall available) resources across multiple cloud data centers / environments available within the user interface system. The resources are shown along with the number in use and the total available number. In the example of FIG. 1, there are four independent environments: fixed environments 1 - 3 and an auto - scaling environment. Each environment includes, from left to right, the number of graphics processing units (GPUs), the number of central processing units (CPUs) on the GPU machines, the amount of memory on the GPU machines, the number of CPUs on non - GPU machines, and the amount of memory on non - GPU machines, any of which can be a bottleneck when starting a new experiment. As can be seen in FIG. 1, with the user interface according to an embodiment of the present invention, such resource usage can be easily monitored.

[0039] As will be described in more detail below with respect to FIG. 3, the search bar 12 can be used to filter the experiment list. As will be described in more detail below with respect to FIG. 2, the lower box 14 shows the experiments actively running along with pagination.

[0040] Referring specifically to FIG. 2, the first column shows the experiment name along with a link to an experimental page as shown in FIG. 5 and described in more detail below. The second column shows the name of the user who submitted the experiment. The third column shows the status of the experiment, which can include states such as (1) Building - the state where an image containing user code is being built for the experiment, (2) Queued - the state where the experiment containing the built image is waiting for resources, (3) Running - the state where the experiment is actively running in one or more environments and consuming resources. The fourth column shows the experiment start time.

[0041] The fifth column shows the resources used by the experiment and from which environments these resources are. The first and second lines on the left side of this column can show the number of GPUs and CPUs used by the experiment, and the third line shows the number of data collectors used in the experiment. Each data collector can communicate with the game console or not depending on the experiment being run by the user. There are also experiments that are run in simulations that do not require a game console. The right side of the fifth column shows in which environments the trainer (T) and data collector (R) are running. An experiment can be divided into multiple environments. For example, the third experiment in FIG. 2 uses resources from three environments. The GPU types used by the experiment trainer are also displayed, and these can be, for example, either Nvidia V100 GPUs or Nvidia A100 GPUs.

[0042] The sixth column in FIG. 2 shows the labels used by the user to classify the execution. One experiment can provide multiple labels. The user can assign any color to the labels. The seventh column shows the text description of the experiment.

[0043] Referring to FIG. 3, the search bar 14 enables the user to filter experiments. The user can filter experiments by, for example, the full or partial experiment name, the full or partial username of the user who ran the experiment, the full or partial label used by one or more experiments, or the full or partial description used by one or more experiments. When filtering, the resource view shows, for each item, three values: the resources used by the filtered experiments, the resources used by all experiments, and all available resources.

[0044] Referring to FIG. 4, all completed experiments can be displayed. Usually, these are displayed below the active experiments. Usually, the experiment state can take one of three values: (1) success, (2) error - if the experiment failed prematurely due to an error during execution, or (3) canceled - if the user manually canceled the experiment. Except for this, the same information as provided for the active experiments (described above with reference to FIG. 2) can be displayed. The display of completed experiments can further provide the total duration of each experiment.

[0045] FIG. 5 shows the experiment summary page of the user interface. The experiment summary page can show the main features of the experiment.

[0046] The first part 50 of the display can show: (1) the username of the user who ran the experiment, (2) the current experiment state, (3) a link to display the configuration (config.yml) used to run the experiment, (4) a link to the code repository (git) containing the experiment code, (5) the script used to run the experiment, (6) the start time of the experiment, and (7) the duration of the experiment. The first part 50 will be different from the start time when the experiment ends.

[0047] The second part 52 of the display can indicate the experiment provenance. The system can easily enable a user to use a model generated from one experiment in another experiment. This is useful to the user as the user may need to design the next experiment in a series of experiments after analyzing the results of the previous experiment. Each successive experiment is designed independently as it may not be possible to design it a priori. The user interface can provide a simple graph-based representation for the user to easily navigate by tracking experiments that build on each other, and can display experiments that contributed to the current experiment and new experiments that use the results from the current experiment. In the screenshot of FIG. 5, models and artifacts from four previous experiments are being used in the current experiment. The user can click on a predecessor run or successor run in the graph to open the experiment page. A summary can also be displayed if the list of predecessor runs is long.

[0048] The third part 54A of the display can indicate the complete resources used by the experiment trainer. The third part 54A includes the number of CPUs, the number of GPUs, and the system memory. If the trainer itself interacts directly with the game console, the number of times is also displayed. Usually, the trainer does not interact directly with the game console, but can interact if necessary.

[0049] The fourth part 54B indicates the complete resources used by a single data collector and the number of such data collectors. The fourth part 54B includes the number of CPUs, the number of GPUs, and the system memory. If the data collector interacts with the game console, the number of game consoles that this single data collector interacts with is also displayed. Each data collector can either communicate with the game console or not, depending on the experiment being run by the user. There are also experiments that are run in simulations that do not require a game console.

[0050] The fifth part 56 shows the state event history of the experiment and can include states such as being under construction, in the queue, running, and interrupted.

[0051] The sixth part 58 shows the main summary dashboard of the experiment. This part 58 can be configurable by the user. This summary dashboard displays the key metrics that help the user quickly determine how well the experiment is going. As will be explained in more detail below, the experiment can have multiple dashboards, and the user can edit the dashboard that includes a summary dashboard showing five graphs as shown in FIG. 5.

[0052] Referring to FIG. 6, the user can configure the experiment to remember any artifacts, which can include things such as videos, checkpoints, and models. An expandable list of artifacts is displayed on the left side of the user interface. It is also possible to perform a search from the list of artifacts.

[0053] On the right side of the user interface is included an arbitrary video player. If the experiment saves videos and information about these videos, the user interface can parse this information and display this video. The video can show any visual representation available in the data collected by the data collector. This video can be generated by a simulation application being run by a trainer or data collector, or be a video feed when communicating from a gaming console.

[0054] In addition, the user interface can also aggregate information from multiple video sources into a single coherent video stream. For example, usually, video files are chunked at intervals of 2 to 5 minutes each, and researchers may need to cross video boundaries to analyze experiments. The user interface can enable the user to seamlessly transition from one video to another across video files. The video player can also be viewed on another "Experiment Video" page when needed.

[0055] Referring to FIG. 7, the experiment summary page can display a list of trainers and data collectors that are part of the experiment below the artifacts and videos. The list of trainers can be displayed first. Usually, there is a single trainer, but the experiment can also use multiple trainers. Next, a paginated list of data collector page numbers can be displayed.

[0056] Each row contains (1) the name of the trainer or data collector, a link to the log from there (as will be explained with reference to the log page below), (2) a bar visually representing how long the trainer or data collector has been running, when they started, and when they were last observed (the start and length of the bar vary according to the total duration of the experiment. If a data collector fails during the experiment, the user interface can instead start a new data collector. The failed data collector can occupy the left half of the bar width, and the new data collector can occupy the right half of the bar width.), (3) a text description of the same information within the bar, (4) the status of the trainer or data collector (usually, this status can be either "Running", "Failed", or "Canceled"), (5) a full explanation of the status, and (6) information such as the environment in which this trainer or data collection tool is operating.

[0057] Figures 8A and 8B show the trainer / data collector log page. The trainer or data collector log page shows a list of log lines from these components. The log page helps users and engineers better analyze experiments via text records. The main features of the log page are: (1) the user can select the source (trainer or specific data collector) from the top of the log page; (2) the user can specify the period for which they want to pull the logs; (3) the user can also specify any filters while pulling the logs (in Figures 8A and 8B, a filter called "reverb_server" based on open source software by Google™ is applied. Other filtering software can be used as needed. The logs include only the lines where the filter exists. The user interface can also support negated filters where all lines without the filter are displayed.); and (4) the option to provide the ability to specify how many log lines should be pulled and whether to pull / display them in ascending / descending order (usually, the log page is automatically updated as more logs become available).

[0058] Figure 9 shows an exemplary experimental dashboard page. An experiment can include one or more dashboards, and the user interface can enable the user to create multiple reusable dashboards that can be easily applied to one or two or more experiments. Since one experiment can have multiple dashboards, each dashboard can enable the user to obtain different perspectives on the experiment.

[0059] A dashboard can itself include multiple widgets. Each widget can display: (1) a graphical representation of metrics recorded by the experiment; (2) a table containing summary information from the execution; (3) text that the user desires to display there.

[0060] The dashboard can also enable the user to set the dropdown "Control". The user can use one or more controls to select the experiment name, experiment label, and metric name. These controls provide interactivity within the dashboard that enables the user to dynamically compare different experiments and different metrics on demand. Figure 10 shows an example of a controlled dashboard.

[0061] Each widget can be driven by a user script that enables the user to easily control the widget. Figure 11 shows a screenshot of the user editing a script that controls a particular widget.

[0062] Figure 12 shows the experiment synchronization event viewer. As shown in Figure 12, by embedding the video from the video player described above with respect to Figure 6 into separate synchronized viewer pages, the metrics, visualizations, and videos can be synchronized to the timeline of the event. This feature can provide researchers with a synchronized comprehensive view of the data collected within the experiment.

[0063] In this view, (1) the video in the lower left provides a specific view. In this specific example of a racing game, the video shows the video from a specific vehicle, and any vehicle within the collected data can be selected. (2) The visualization in the upper right provides a global view. The visualizer shows a global representation of the application that was the source of the data collection during the experiment. In this specific example of a racing game, the course, eight vehicles, and the trajectories of these vehicles are shown. As will be detailed below, event annotations can also be displayed on this visualization. (3) Both the video and the visualization are synchronized with the timeline in the upper left. In this specific example of a racing game, this timeline indicates the vehicle positions. In a general case, the timeline can display any metric or event annotation from the experiment over time. The user can move to a particular moment by clicking on the timeline. By combining the synchronized experiment visualization, video, and metrics, a detailed analysis is provided within the machine learning user interface.

[0064] Also, referring to FIGS. 13 and 14, the user interface system can further analyze the experiment to generate event annotations. After the experiment is complete, key events within the data stream can be indicated. In a racing game, key events can appear to be things like position gains, position losses, or a vehicle leaving the course. However, events can be generated for any data stream collected by a data collector of any application. Thereafter, event annotations can be displayed on the visualizer.

[0065] On the visualizer in the upper right, various event annotations are shown over time. In the example of the racing game, FIGS. 12 and 13 show before and after vehicle 7 leaves the road, respectively. FIG. 13 can show a road - out event annotation within the visualizer, for example, via a specifically - colored sphere. Other events can be visualized similarly.

[0066] The above-described user interface can be used in a system for training a machine learning model, such as a racing game artificial intelligence player. The user interface can be part of a training system computing architecture, such as that described in U.S. Patent Application No. 17 / 650,275, the contents of which are incorporated herein by reference.

[0067] Those skilled in the art can make many changes and modifications without departing from the spirit and scope of the present invention. Therefore, the illustrated embodiments are shown by way of example only and should not be construed as limiting the present invention as defined by the following claims. For example, even though the elements of the claims are shown in a particular combination below, it should be clearly understood that the present invention includes other combinations of fewer, more, or different elements than the disclosed elements.

[0068] The words used herein to describe the present invention and its various embodiments are to be understood as including not only their generally defined meanings, but also special definitions herein that include any inclusive structures, materials, or acts that represent a single species.

[0069] Therefore, it is defined herein that the definitions of the words or elements in the following claims do not only include the combinations of elements expressly recited in the language. Thus, in this sense, it is contemplated that an equivalent substitute of two or more elements can be used in place of any one of the elements within the following claims, or a single element can be used in place of two or more elements within the claims. In the above, the elements are described as functioning in a particular combination and may initially be claimed as such, but in some cases, one or more elements resulting from the claimed combination can also be deleted from these combinations, and the claimed combination can be directed to a sub-combination or a variation of a sub-combination.

[0070] It is clearly contemplated that slight variations from the present subject matter, known now or later devised, from the perspective of those skilled in the art, are equally included within the scope of the claims. Accordingly, obvious substitutions known to those skilled in the art now and in the future are defined as being included within the scope of the defined elements.

[0071] Accordingly, the claims are to be understood to include what has been specifically illustrated and described above, what is conceptually equivalent, what is obviously substitutable, and what incorporates the basic idea of the present invention.

Claims

1. A method for providing a user interface that analyzes the execution of model training and tracks and visualizes aspects of a machine learning experiment, comprising: Displaying a timeline of selected metrics of the machine learning experiment; Displaying a video synchronized with a selected portion of the timeline; Displaying a visualizer showing the global representation that is the source of data during the machine learning experiment; A method characterized by including the above.

2. The video is generated by a simulation application executed by a trainer or a data collector, The method according to claim 1.

3. The video is a video feed during communication from a cloud-based gaming console, The method according to claim 1.

4. The video includes information from multiple video sources aggregated into a single coherent video stream, The method according to claim 1.

5. The machine learning experiment is in a racing game environment, The method according to claim 1.

6. The video shows vehicle video from a selected vehicle participating in the racing game, The method according to claim 5.

7. The visualizer shows a representation of a racing track, each racing vehicle, and the trajectory of each racing vehicle, The method according to claim 5.

8. The timeline shows the vehicle positions in the racing game environment, The method according to claim 5.

9. Further comprising displaying, on the visualizer, one or more event annotations for one or more key events from the machine learning experiment The method according to claim 1.

10. The one or more key events include racing game key events including one or more of acquisition of a position, loss of a position, or departure from the course of the vehicle The method according to claim 9.

11. The visualizer includes one or more event annotations for one or more key events from the machine learning experiment The method according to claim 1.

12. A method of providing a user interface for analyzing the execution of model training and tracking and visualizing aspects of a machine learning experiment, comprising providing an interface for a user to generate a plurality of user dashboard pages for visualizing the experiment, each user dashboard page including one or more dashboards, each of the one or more dashboards providing a different perspective on the experiment, and the plurality of user dashboard pages being reusable across a plurality of experiments at least one of the one or more dashboards provides a graph-based representation for the user to track and navigate experiments that build on each other, and the at least one dashboard displays one or more experiments that contribute to the current experiment and new experiments that use the results from the current experiment A method characterized by the above.

13. Further comprising displaying a timeline of selected metrics of the machine learning experiment The method according to claim 12.

14. Further comprising synchronizing a video with the timeline The method according to claim 13.

15. Further comprising displaying a visualizer showing a global representation that is a data collection source during the machine learning experiment, The method according to claim 14.

16. The machine learning experiment is in a racing game environment, The method according to claim 15.

17. The visualizer shows a representation of a racing course, each racing vehicle, and the trajectory of each racing vehicle, The method according to claim 16.

18. The timeline shows vehicle positions in the racing game environment, The method according to claim 16.

19. A user interface of a machine learning training system computing architecture for training a racing game artificial agent, comprising: Displaying a timeline of selected metrics of a machine learning experiment; Displaying a video synchronized with a selected portion of the timeline; Displaying a visualizer showing a global representation that is a data collection source during the machine learning experiment; Displaying one or more event annotations for one or more key events from the machine learning experiment on the visualizer; A user interface characterized by including the above.

20. Further comprising displaying an experimental dashboard page including one or more dashboards, wherein the one or more dashboards are user-configurable dashboards applicable to a single experiment or multiple experiments, The method according to claim 19.

Images