Training Architecture Using Game Consoles

Abstract

An artificial intelligence agent can act as a player of a video game, such as a racing video game. The agent can compete against the best players in the world and often beat them. The game is completely external to the agent and can run in real time. In this way, the training system closely resembles a real-world system. Consoles that run the games to train the agents are provided in a cloud computing environment. Agents and trainers can run on other computing devices in the cloud, and the system can select trainers and agents calculated, for example, based on their proximity to the console. Users can select the game they want to run and submit code that can be built and deployed to the cloud system. Game metrics, logs, and artifacts can be sent to cloud storage.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE

[0001] Embodiments of the present invention relate generally to artificial intelligence training. In particular, the present invention relates to a system for training artificial agents using a game console. [Background technology]

[0002]

[0002] The following background information may present examples of certain aspects of the prior art (e.g., but not limited to, techniques, facts or conventional wisdom) that are expected to serve to further inform the reader of additional aspects of the prior art, but should not be construed as limiting the invention or any embodiment thereof to anything mentioned or implied therein or inferred therefrom.

[0003]

[0003] Video game players often wish to improve their game through practice and playing against other players. However, as a game player develops greater skill at a given game, the number of suitable challengers decreases significantly. Although such players may be able to improve their game by playing against less skilled players, it is usually more beneficial for them to play against players who can pose a greater challenge.

[0004]

[0004] Many games allow players to participate as the game offers them, but these players may only be following a specific program that a skilled player can understand and defeat. Summary of the Invention [Problem to be solved by the invention]

[0005] In view of the above, what is needed is a system and method for training artificial intelligence agents so that they have the capabilities to challenge even the most skilled video game players. [Means for solving the problem]

[0006]

[0006] An embodiment of the present invention provides a training system computing architecture including: a build environment that enables a user to build data gatherers, trainers, and experiment definition programs, where the data gatherers are configured to interact with games on cloud-based game consoles, and the trainers are configured to review experiences from the data gatherers and improve the data gatherers' strategies for interacting with the games; a development source code control service for managing code for the data gatherers, the trainers, and the experiment definition programs and creating Docker images thereof; a production source code control service for managing the development source code control service and building Docker images for experiments; an experiment manager component configured to monitor a state of an experiment and determine whether the experiment should be run once the experiment is in a scheduled state, where the experiment manager component starts the experiment on a predetermined number of the cloud-based game consoles using a predetermined number of data gatherers; and a monitoring service that enables a user to monitor the experiment.

[0007]

[0007] An embodiment of the present invention further provides a method for training an artificial intelligence agent to play a video game on a cloud-based game console, the method including: programming the artificial intelligence agent to interact within the video game; configuring a trainer to review experience from the artificial intelligence agent and improve the artificial intelligence agent's strategy for interacting with the video game; storing and sharing code of the artificial intelligence agent, the trainer, and an experiment definition program using a development source code control service and creating Docker images thereof; and managing the development source code control service using a production source code control service within a build environment of a game console system. In some embodiments, the development source code control service and the production source code control service can be the same. The method may further include building a Docker image for an experiment; monitoring an experiment state with an experiment manager component and determining whether to run the experiment once the experiment reaches a scheduled state; initiating the experiment on a predetermined number of the cloud-based game consoles with a predetermined number of data gatherers; receiving experience from the data gatherers regarding playing of the video game; and running one or more artificial intelligence learning algorithms to update game playing strategies of the data gatherers.

[0008]

[0008] An embodiment of the present invention also provides an artificial intelligence agent configured to compete in a video game, the artificial intelligence agent being trained on a cloud based game console, the artificial intelligence agent being trained by a method including the steps of programming the artificial intelligence agent to interact within the video game; configuring a trainer to review experience from the artificial intelligence agent and improve the artificial intelligence agent's strategy for interacting with the video game; reviewing code of the artificial intelligence agent, the trainer, and an experiment definition program using a development source code control service and creating a Docker image thereof; mirroring the development source code control service and building a Docker image for the experiment using a production source code control service within a build environment of a game console system; monitoring a status of an experiment using an experiment manager component and determining whether the experiment should be run once the experiment is in a scheduled state; initiating the experiment on a predetermined number of the cloud based game consoles using a predetermined number of data gatherers; receiving experience from the data gatherers regarding the play of the video game; and executing one or more artificial intelligence algorithms to update the game play strategies of the data gatherers.

[0009]

[0009] These and other features, aspects and advantages of the present invention will become better understood with reference to the following drawings, description and claims.

[0010]

[0010] Some embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference symbols can indicate similar elements and in which: [Brief description of the drawings]

[0011] [Figure 1]FIG. 1 illustrates an exemplary system architecture for training agents using a game console, according to an embodiment of the present invention. [Diagram 2] FIG. 2 illustrates resources used in the system architecture of FIG. 1. [Diagram 3] FIG. 2 illustrates a schematic diagram of a user computing device for use in the architecture and method according to an exemplary embodiment of the present invention. [Figure 4] FIG. 2 illustrates services provided by a landlord service for controlling resource usage in an architecture and method according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012]

[0015] Unless otherwise indicated, the figures in the drawings are not necessarily drawn to scale.

[0013]

[0016] A better understanding of the invention and its various embodiments can be obtained by reference to the following detailed description which describes illustrative embodiments, with the express understanding that the illustrated embodiments are presented by way of example and are not intended to limit the invention as ultimately defined by the claims.

[0014]

[0017] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting of the present invention. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. As used herein, the singular forms "a," "an," and "the" are intended to include the plural in addition to the singular, unless the context clearly indicates otherwise. Furthermore, it should be understood that the terms "comprises" and / or "comprising," as used herein, indicate the presence of stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0015]

[0018] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by those skilled in the art to which the present invention belongs. Furthermore, terms as defined in commonly used dictionaries should be interpreted to have a meaning in accordance with these meanings in the context of the relevant art and this disclosure, and should not be interpreted in an ideal or overly formal sense unless expressly defined herein.

[0016]

[0019] In the description of the present invention, it will be understood that multiple techniques and steps are disclosed. Each of these has its own advantages and can also be used with one or more, or in some cases all, of the other techniques disclosed. Thus, for the sake of clarity, the description will not unnecessarily repeat all possible combinations of the individual steps. However, the specification and claims should be read with the understanding that such combinations are fully within the scope of the present invention and the claims.

[0017]

[0020] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without these specific details.

[0018]

[0021] The present disclosure should be considered as an exemplification of the invention and is not intended to limit the invention to the specific embodiments shown in the following figures or description.

[0019]

[0022] Devices or system modules that are in at least general communication with each other need not be in continuous communication with each other, unless otherwise specified, and devices or system modules that are in at least general communication with each other may communicate directly or indirectly through one or more intermediary devices.

[0020]

[0023] A description of an embodiment including multiple components in communication with each other does not imply that all such components are required, but rather a variety of optional components are described to illustrate the wide range of possible embodiments of the present invention.

[0021]

[0024] A "computer" or "computing device" may mean one or more devices and / or one or more systems that can accept structured input, process the structured input according to prescribed rules, and produce the results of the processing as output. Examples of computers or computing devices may include computers, fixed and / or portable computers, computers with a single processor, multiple processors, or multi-core processors capable of operating in parallel and / or non-parallel, supercomputers, mainframes, superminicomputers, minicomputers, workstations, microcomputers, servers, clients, interactive televisions, web appliances, communications devices with Internet access, hybrid combinations of computers and interactive televisions, portable computers, tablet personal computers (PCs), personal digital assistants (PDAs), mobile phones, application specific hardware that emulates a computer and / or software, such as digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific instruction set processors (ASIPs), chips, chips, systems on chips or chipsets, data acquisition equipment, optical computers, quantum computers, biocomputers, and generally devices that accept data, process the data according to one or more stored software programs, and generate results, and may typically include input devices, output devices, storage devices, arithmetic units, logic units, and control units.

[0022]

[0025] "Software" or "application" can mean prescribed rules for operating a computer. Examples of software or applications can include code segments, graphical and / or textual instructions, applets, precompiled code, interpreted code, compiled code, and computer programs in one or more computer readable languages.

[0023]

[0026] Furthermore, these computer program instructions, which can direct a computer, other programmable data processing apparatus, or other device to function in a particular manner, can be stored on a computer-readable medium such that the instructions stored on the computer-readable medium produce an article of manufacture that includes instructions that implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0024]

[0027] Additionally, although process steps, method steps, algorithms, or the like may be described in a certain order, such processes, methods, and algorithms may be configured to work in other orders. In other words, any order or sequence of steps that may be described does not necessarily indicate that the steps must be performed in this order. Steps of processes described herein may be performed in any order that is practical. Additionally, some steps may be performed simultaneously.

[0025]

[0028] It will be readily apparent that the various methods and algorithms described herein can be implemented, for example, by appropriately programmed general-purpose computers and computing devices. Typically, a processor (e.g., a microprocessor) receives instructions from a memory or similar device and executes those instructions to perform the process defined by those instructions. Furthermore, programs implementing such methods and algorithms can be stored and transmitted using a variety of known media.

[0026]

[0029] The term "computer-readable medium" as used herein means any medium that participates in providing data (e.g., instructions) that may be read by a computer, a processor, or a similar device. Such media may take many forms, including, but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical or magnetic disks and other persistent memory. Volatile media include dynamic random access memory (DRAM), which typically constitutes the main memory. Transmission media include coaxial cables, copper wire, and fiber optics, including the wires that comprise a system bus coupled to the processor. Transmission media may include or convey acoustic waves, light waves, and electromagnetic radiation, such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer readable media include, for example, a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, a DVD, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM, EPROM, FLASHEEPROM, any other memory chip or cartridge, a carrier wave as described below, or any other medium from which a computer can read.

[0027]

[0030] Various forms of computer readable media may be involved in carrying the set of instructions to the processor, for example, the set of instructions may (i) be provided to the processor from a RAM, (ii) be carried over a wireless transmission medium, and / or (iii) be formatted according to a number of formats, standards, or protocols, such as Bluetooth, TDMA, CDMA, 3G, 4G, 5G, etc.

[0028]

[0031] Embodiments of the present invention may include apparatuses for performing the operations disclosed herein, which may be specially constructed for the desired purposes, or which may comprise general-purpose devices selectively activated or reconfigured by programs stored in the devices.

[0029]

[0032] Unless otherwise stated, and as will become apparent from the following description and claims, throughout this specification, descriptions using terms such as "processing," "calculating," "computing," or "determining" should be understood to mean operations and / or processes of a computer, computer system, or similar electronic computing device that manipulates data represented as physical quantities, such as electronic quantities in a computer system's registers and / or memory, and / or transforms data into other data that is similarly represented as physical quantities in a computer system's memory, registers, or other such information storage, transmission, or display device.

[0030]

[0033] Similarly, the term "processor" may refer to any device or part of a device that processes electronic data from registers and / or memory and converts this electronic data into other electronic data that can be stored in registers and / or memory or transmitted to an external device so as to cause a physical change or action in an external device.

[0031]

[0034] The term "agent" or "intelligent agent" or "artificial agent" or "artificial intelligence agent" is intended to mean any artificial entity that selects an action in response to an observation. "Agent" can mean, without limitation, a robot, a simulated robot, a software agent or "bot", an adaptive agent, an internet or web bot.

[0032]

[0035] In general, embodiments of the present invention provide an artificial intelligence agent that can act as a player of a video game, such as a racing video game. The game is completely external to the agent and can run in real time. In this way, the training system closely resembles a real-world system. Consoles that run games to train the agent are provided in a cloud computing environment. The agents and trainers can run on other computing devices in the cloud, and the system can select trainers and agents calculated, for example, based on their proximity to the console. Users can select the game they want to run and submit code that can be built and deployed to the cloud system. A resource management service can monitor the resources of the game consoles among human users, review usage, and identify interrupted experiments to ensure there are enough game consoles for the human users.

[0033]

[0036] With reference to Figures 1-4, the basic workflow can be envisioned as follows:

[0034] On the user's local machine

[0037] On the user's local machine 10, the user can write a computer program (usually in, for example, Python) for the agent. This program is called a "data gatherer" 12, and such an agent can be programmed to interact with the game and know how to control it. The user can also write a computer program (usually in, for example, Python) for the "trainer" 14. The trainer 14 can be programmed to get experiences from the data gatherer 12 and use them to know how to improve the agent's (data gatherer 12) strategy. The trainer 14 can use any number of algorithms and neural network structures, such as may be present in an artificial intelligence (AI) library 16. The user can write a third program that defines the experiments 18. This program is usually in the form of a configuration file. The configuration file can be written in a human-readable data serialization language, such as YAML, and can define the number of data gatherers 12 to use, the computational power required for the data gatherers 12 and trainer 14, the algorithms that the trainer 14 should use, the set of tasks that the trainer 14 will have the data gatherer 12 perform, etc.

[0035]

[0038] A user can check in the code (data gatherer 12, trainer 14, and experiment definition 18) to a source code repository 22, such as GitHub. A user can run a command line program via a command line interface 23, which submits a request to build an experiment to a build system 26 in the build environment 20 if an existing Docker image cannot be reused. The user then requests a server 52 in the monitoring environment 57 via a data query interface 25 to run the experiment identified by its source code check-in reference hash. The system server 52 can store information about the requested experiment in a database 56 with a status <submitted>. In some embodiments, there can also be a web interface 24 that allows a user to request an execution. As shown in FIG. 1, the web interface 24 and the command line interface 23 can interact with a data query and manipulation interface 25, such as Hasura / GraphQL, to allow a user to review the experiment during or after its execution, as described below. Of course, other query interfaces can be utilized for the user's review of the data.

[0036] In the cloud

[0039] In a cloud computing environment, a build system 26 can build the user's code into a Docker image 28. The build system 26 can be any virtual machine imaging system, such as CircleCI. If the experiment requires resources from a cloud gaming system 30 (also referred to as a production build environment 30), the production build environment 30 can pull the code from the development build environment 20, and the build system can also use a source code repository 32, such as GitLab, to perform various secondary security assessments and then build the user's code into a Docker image 36 using a Docker build 34. The system can set the experiment state to <building> and record which environment (such as data center DC-1 (environment 38) and data center DC-2 (environment 40) as shown in FIG. 2) is building the experiment. While the description herein uses Docker images for illustration and FIG. 1 shows Kubernetes 44 as a container orchestration system for interfacing with Docker image 36, it should be understood that other types of architectures can be used to achieve the same purpose. For example, the Docker runtime can be replaced with a runtime that conforms to the Kubernetes container runtime interface. Similarly, the container orchestration system Kubernetes can be replaced by other orchestration systems such as Slurm.

[0037]

[0040] Periodically, the resource control service 42 for each environment 38, 40 can look in its view at the build system 30 for building experiments. Once complete, the resource control service 42 can transition the experiment state in that environment to the <built> state. The system monitors each environment for transitions to <build>, and once all required environments are complete, the overall experiment state can be changed to <build>. The system can monitor experiments in the <build> stage and transfer them to the <queued> state.

[0038]

[0041] Periodically, the system may evaluate experiments in a <waiting> state and determine whether the experiment should be started. When determining whether to start an experiment, the system may take into account factors such as the priority level of the experiment, the age of the experiment, whether the resources required by the experiment are available in any acceptable environment, and other criteria for scheduling experiments (such as quota limits by user or project).

[0039]

[0042] If the system decides to start an experiment, it can mark the experiment as <scheduling> and tag the experiment with an identifier of the resources to be consumed. For example, the system can determine that a particular experiment should be run in a particular environment 38, 40 using a game console 46 (e.g., PS4) and a data gatherer 12. The experiment can be run using a GPU (e.g., V100 GPU 48) for a trainer 14 in the same or a different environment, and add annotations to the experiment to record their decisions.

[0040]

[0043] Periodically, the resource control service 42 of each target environment can check to see if there are any experiments in the <scheduling> state tagged to start within that environment 38, 40. If there are, the required resources can be started.

[0041] When Data Gatherer starts

[0044] Technically, a data gatherer can be any program. In the context of an embodiment of the present invention, a data gatherer 12 can be playing a game (such as a PlayStation® game) within the network of a cloud gaming system production environment 50.

[0042]

[0045] The data gatherer 12 can find and connect to a trainer 14 running as specified by the system server 52. The data gatherer 12 can request a user ID for the gaming system from a service that manages user IDs for training agents. The data gatherer 12 requests available consoles 46 in the cloud gaming system 50 and requests that a particular game be loaded.

[0043]

[0046] Data gatherer 12 can then request tasks from trainer 14. A task is essentially a configuration of the game that data gatherer 12 should play. For example, in a racing game, one task might have data gatherer 12 launch clusters of five cars evenly spaced around the track. Each cluster contains one car controlled by an agent and three cars controlled by the game's built-in AI.

[0044]

[0047] The data gatherer 12 can start the game, communicate the scenario configuration to the game, and then begin playing. As the agent plays the game, it can transmit its experiences to the trainer 14.

[0045]

[0048] Periodically, data gatherer 12 can fetch updated models from trainer 14. Optionally, data gatherer 12 can also send metrics during or after the scenario to database 56 via data query interface 25. For example, data gatherer 12 can report its best lap time. Optionally, data gatherer 12 can store other data, such as complete race data, in a remote data store 58, such as S3. Optionally, data gatherer 12 can be configured to stream the video output of the cloud gaming console to S3 for later viewing by the experimenter.

[0046]

[0049] Once the task termination criteria are met, the data gatherer 12 may terminate the scenario on the cloud gaming console 46 and request a new task from the trainer 14.

[0047] When the trainer starts

[0050] The trainer 14 can initialize a buffer in which it can store experiences reported by the data gatherer 12. Optionally, it can load a buffer from a previous run. The trainer 14 can maintain a list of tasks from which it can pass new tasks to the data gatherer 12 when requested by the data gatherer 12.

[0048]

[0051] Periodically, the trainer 14 loads the experiences from the buffer and updates the neural network model using a learning algorithm. Optionally, the trainer 14 reports metrics to the system, and such metrics are stored in a metrics database 56. The updated neural network model can be sent to the data gatherer 12.

[0049] On the user's machine

[0052] While an experiment is being built and run, a user can monitor the experiment using, for example, a web browser 24 connected to the system server 52 via a data query interface 25. The system interface can show progress through the build and deployment stages of the experiment. Once the experiment is running, the system interface can allow the user to inspect metrics and create dashboards that display various graphs of performance. The system interface can also be used to graph metrics across multiple runs simultaneously, allowing the user to compare the performance of different experiments.

[0050] Suspension and resumption

[0053] The resource management service 60 (also simply referred to as the resource manager 60) is the name of a service that the cloud gaming system has to coordinate resources with external services. Since training is performed on an actual game console, the training system shares the game system (such as a PlayStation® network) with humans. When more humans want to play the game, the resource management service 60 instructs the training system 50 to reduce usage. When the humans stop playing, the resource management service 60 gives more resources to the training system 50. The system server 52 utilizes the resource control service 42 (also referred to as the experiment manager 42) to adjust the use of resources based on the goals set by the resource management service 60.

[0051]

[0054] As described in more detail below, some key features of the integration of the training system with the cloud gaming system are (1) a module 62 for measuring the load due to human activity, (2) a module 64 for predicting future load, and (3) a module 66 for determining how much of these resources can be given to the researcher. The resource control service 42 can provide functions including (4) a module 68 for reading the number of available resources, (5) a module 70 for starting and stopping experiments according to resource constraints and job priority / age / quota, and (6) a module 72 for restarting jobs in an environment where resources are available. The modules 70 and 72 can be part of the system server 52. In some embodiments, an experiment can be run in multiple environments, but the resource control service 42 (experiment manager 42) only controls resources in one environment. In some embodiments, for example, if the system server 52 does not operate or does not operate fast enough, the resource management service 60 can terminate the experiment according to a pre-programmed protocol, such as first in, first out.

[0052]

[0055] The training system can monitor the resource management service 60 of the cloud gaming system. When the system notices that the resources allocated to the training system (particularly the cloud gaming consoles) fall below the current usage of the system, the system can identify one or more experiments to suspend. When determining which experiments to suspend, the system can consider the location of the resources used by the experiment, the priority level of the experiment, the age of the experiment, the user ID of the experiment, and / or other attributes related to the experiment. The system can transition the selected experiments to a <suspending> state.

[0053]

[0056] Each resource control service 42 in each environment (locations 38, 40, etc.) may periodically check the system server to see if a running experiment has transitioned to a <suspended> state. If so, the resource control service may terminate the processes under its control. When a trainer is requested to suspend, state information (especially the experience buffer) may be saved to remote storage so that it can be later reloaded before graceful shutdown.

[0054]

[0057] When all processes under each control have terminated, the resource control service 42 changes its portion of the experiment to the <suspended> state. When all of the associated resource control services 42 have transitioned their portions to <suspended>, the system can transition the entire experiment to the <suspended> state.

[0055]

[0058] If the system determines in the resource control service 42 that the number of available resources is greater than the number of resources in use, it can look at the list of suspended runs. The system can then resume some of these experiments. The selection of which experiments to resume can take into account the location of the resources used by the experiment, the priority level of the experiment, the age of the experiment, the user ID of the experiment, and / or other attributes related to the experiment.

[0056]

[0059] To avoid thrashing, the system server may smooth signals regarding resource availability received from the resource management service 60. The system server may smooth these signals by applying any number of standard algorithms such as low pass filters, min-max time windows, etc. Optionally, the user may suspend a running experiment by clicking a button. The experiment transitions to a <manually suspended> state. The user may choose to reactivate a manually suspended experiment by pressing a button on the user interface. The system transitions the experiment to <suspended>. The system reactivates the experiment when resources become available, following the same conditions as above.

[0057] completion

[0060] The user can write exit conditions into the trainer script to report completion to the system and exit once certain conditions are met. The system changes the experiment state to <Success>. Alternatively, the user can use the system's interface to click the "Cancel" button. The system will shut down the experiment immediately after through a process similar to the abort process described above, but the current experiment state will not be saved. The system will set the experiment state to <Cancelled>.

[0058]

[0061] Those skilled in the art may make many changes and modifications without departing from the spirit and scope of the present invention. It is therefore to be understood that the illustrated embodiments are presented by way of example only and should not be construed as limiting the invention as defined by the following claims. For example, although the elements of the claims are presented below in specific combinations, it should be expressly understood that the invention includes other combinations of elements that are less, more, or different than those disclosed.

[0059]

[0062] The words used herein to describe the invention and its various embodiments are to be understood to include not only their generally defined meaning, but also the generic structures, materials, or acts for which they represent a single species by specific definitions herein.

[0060]

[0063] Therefore, the definition of the words or elements in the following claims is set forth herein to include more than just the combination of elements expressly stated in the text. In this sense, it is therefore considered that the equivalent substitute of two or more elements may be used in place of any one of the elements in the following claims, or a single element may be used in place of two or more elements in the claims. Although elements may be described above as functioning in a particular combination and may be initially claimed in this manner, it is to be clearly understood that in some cases, one or more elements resulting from the claimed combination may be deleted from these combinations, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0061]

[0064] Slight variations from the subject matter as viewed by one of ordinary skill in the art, whether now known or later devised, are expressly contemplated to be equally within the scope of the claims. Thus, obvious substitutions now known to those of ordinary skill in the art and those that will become known in the future are also defined as being within the scope of the specified elements.

[0062]

[0065] The claims are therefore to be understood to include what is specifically illustrated and described above, what is conceptually equivalent, what is obviously substituted, and what incorporates the basic idea of ​​the invention. [Explanation of symbols]

[0063] 10 User's local machine 12 Data Gatherer 14 Trainer 16 Artificial Intelligence (AI) Library 18 Experiment Definition 20 Build Environment 22 Source Code Repositories 23 Command Line Interface 24 Web Interface 25 Data Query Interface 26 Build System 28 Docker Images 30 Cloud gaming system / production build environment 32 Source Code Repositories 34 Docker Build 36 Docker Images 38 Environment 40 Environment 42 Resource Control Service / Experiment Manager 44 Container Orchestration System 46 Game Consoles 48V100 GPU 50 Cloud gaming system production environment 52 Server 56 Databases 57 Monitoring environment 58 Remote Data Store 60 Resource Management Service / Resource Manager 62 Measuring human user activity 64 Predicting future loads 66 Determine resources for training 68 Read the number of available resources 70 Starting and stopping experiments according to resource constraints 72 Restart the experiment

Claims

1. 1. A training system computing architecture, comprising: data gatherers configured to interact with games on the cloud-based game console; a trainer configured to review experiences from the data gatherers and improve the data gatherers' strategies for interacting with the game; an experiment manager component configured to monitor a state of an experiment and determine whether to run the experiment once the experiment reaches a scheduled state, the experiment manager component initiating the experiment on a predetermined number of the cloud-based game consoles with a predetermined number of data gatherers; a monitoring service that allows a user to monitor the experiment; 1. A training system computing architecture comprising:

2. a development source code control service for managing the code of the data gatherer, the trainer, and the experiment definition program and creating a Docker image thereof; A production source code control service that mirrors the development source code control service and builds experimental Docker images; and 10. The training system computing architecture of claim 1 further comprising:

3. 2. The training system computing architecture of claim 1, wherein the experiment definition program defines the number of data gatherers to use, the computing power required for the data gatherers and the trainer, the algorithms the trainer should use, and the set of tasks the trainer will have the data gatherers perform within the game.

4. 10. The training system computing architecture of claim 1, wherein the cloud-based game console is shared with human users who play the games.

5. 2. The training system computing architecture of claim 1, wherein the data gatherer and the trainer are deployed in one or more environments.

6. 2. The training system computing architecture of claim 1, wherein the experiment manager component determines whether to run the experiment by considering the priority level of the experiment, the age of the experiment, and / or whether resources required by the experiment are available in any acceptable environment.

7. 2. The training system computing architecture of claim 1, wherein said trainer is programmed with one or more tasks for each of said data gatherers.

8. 8. The training system computing architecture of claim 7, wherein the experiment is complete when each of the one or more tasks is completed for each of the data gatherers.

9. 10. The training system computing architecture of claim 1, further comprising an execution and metrics database for storing information about the games played for the experiment, the information including at least one of artifacts, neural networks, replay buffers, and algorithm states.

10. 2. The training system computing architecture of claim 1, further comprising one or more artificial intelligence learning algorithms used by the trainer to receive experience from the data gatherer to update the game play strategy of the data gatherer.

11. 1. A method for training an artificial intelligence agent to play a video game on a cloud-based game console, comprising: programming said artificial intelligence agent to interact within said video game; configuring a trainer to review experiences from the artificial intelligence agent and improve the artificial intelligence agent's strategy for interacting with the video game; monitoring the state of an experiment using an experiment manager component and determining whether to run the experiment once the experiment reaches a scheduled state; initiating the experiment on a predetermined number of the cloud-based game consoles with a predetermined number of data gatherers; receiving experience from the data gatherer regarding playing of the video game; running one or more learning algorithms to update the game playing strategy of the data gatherer; The method according to claim 1, further comprising:

12. using a development source code control service to manage the code of the artificial intelligence agent, the trainer, and the experiment definition program and create a Docker image thereof; mirroring the development source code control service with a production source code control service within a build environment of the game console system to build an experimental Docker image; The method of claim 11 further comprising:

13. 12. The method of claim 11, further comprising the step of defining, by the experiment definition program, the number of data gatherers to be used, the computing power required for the data gatherers and the trainer, the algorithms to be used by the trainer, and the set of tasks and type of curriculum the trainer will have the data gatherers perform within the video game.

14. 12. The method of claim 11, wherein the cloud-based game console is shared with human users who play the games.

15. 12. The method of claim 11, further comprising deploying the data gatherer and the trainer to one or more environments.

16. 12. The method of claim 11, further comprising: determining, by the experiment manager component, whether to run the experiment by considering a priority level of the experiment, an age of the experiment, an experiment quota, and / or whether resources required by the experiment are available in any acceptable environment.

17. 12. The method of claim 11, further comprising the step of programming the trainer with one or more tasks for each of the data gatherers.

18. 12. The method of claim 11, further comprising storing information about the games played for the experiment in an execution and metrics database, the information including at least one of artifacts, neural networks, replay buffers, and algorithm states.

19. 1. An artificial intelligence agent configured to compete in a video game, the artificial intelligence agent being trained on a cloud-based game console shared with a human player, the artificial intelligence agent being trained by a method comprising: programming said artificial intelligence agent to interact within said video game; configuring a trainer to review experiences from the artificial intelligence agent and improve the artificial intelligence agent's strategy for interacting with the video game; monitoring the state of an experiment using an experiment manager component and determining whether to run the experiment once the experiment reaches a scheduled state; initiating the experiment on a predetermined number of the cloud-based game consoles with a predetermined number of data gatherers; receiving experience from the data gatherer regarding playing of the video game; running one or more learning algorithms to update the game playing strategy of the data gatherer; Including, An artificial intelligence agent characterized by

20. 20. The artificial intelligence agent of claim 19, wherein the artificial intelligence agent is trained by the method further comprising the step of the experiment manager component determining whether to run the experiment by considering the priority level of the experiment, the elapsed time of the experiment, and / or whether resources required by the experiment are available in any acceptable environment.

Images