Methods for controlling the use of computing resources such as virtual game consoles.

Abstract

The artificial intelligence agents can act as players of video games, such as racing video games. The games are completely external to the agents and can run in real time. In this way, the training system closely resembles a real-world system. Consoles that run the games to train the agents are provided in a cloud computing environment. The agents and trainers can run on other computing devices in the cloud, and the system can select trainers and agents, for example calculated based on their proximity to the console. Users can select the games they want to run and submit code that can be built and deployed to the cloud system. A resource management service can monitor the resources of the game consoles among human users, examine usage, and identify interrupted experiments to ensure there are enough game consoles for human users.

Description

Technical Field

[0001]

[0001] Embodiments of the present invention generally relate to managing the use of computing resources. Specifically, the present invention relates to a method for managing the use of a cloud-based game console to enable both research and human gaming use.

Background Art

[0002]

[0002] The following background information can present examples of specific aspects of the prior art (e.g., by way of illustration and not limitation, techniques, facts, or generalizations), which are expected to be useful in further informing the reader of additional aspects of the prior art, but should not be construed as limiting the present invention or any of its embodiments to any matter mentioned or implied or speculated about therein.

[0003]

[0003] Video game players often wish to improve their games through practice and playing against other players. However, when a game player acquires excellent skills in a given game, the number of suitable challengers significantly decreases. Such players may be able to improve their games by playing against less skilled players, but usually it is more beneficial to play against players who can pose a greater challenge.

[0004]

[0004] In many games, players can participate in what the game offers. However, these players may only be following a specific program that can be understood and defeated by skilled players.

Summary of the Invention

Problems to be Solved by the Invention

[0005] In light of the above, there is a need for methods to train artificial intelligence agents on cloud-based game consoles so that they can challenge even the most skilled video game players, while managing computing resources so as not to affect human players through research, development, and training of artificial intelligence agents. [Means for solving the problem]

[0006]

[0005] Embodiments of the present invention provide a method for managing computing resources between a human user and a research user, comprising the steps of: providing a resource management service for the computing resources, the resource management service comprising: a first resource management module for measuring the load caused by human activity; a second resource management module for predicting future loads caused by human activity; and a third resource management module for determining the computing resources to be given to the research user; providing a resource control service for the computing resources, the resource control service comprising: a first resource control module for reading the number of available computing resources; and a second resource control module for identifying one or more experiments to be suspended; and moving the one or more experiments identified to be suspended to a suspending state.

[0007]

[0006] Embodiments of the present invention are methods for training an artificial intelligence agent to play a video game on a cloud-based game console shared with a human user, comprising the steps of: providing the artificial intelligence agent for interacting within the video game; configuring a trainer to review the experiences from the artificial intelligence agent and improve the strategies of the artificial intelligence agent for interacting with the video game; reviewing the code of the artificial intelligence agent, the trainer, and the experiment definition program using a local source code repository and creating a Docker image thereof; mirroring the local source code control service using the console system source code control service in the build environment of the game console system and building a Docker image for the experiment; monitoring the status of the experiment using a resource control service and determining whether to run the experiment when it enters a scheduling state; and using one or more predetermined environments with a predetermined number of data gatherers (data The method further provides a step of starting the experiment on a predetermined number of the cloud-based game consoles using gatherers, and providing a resource management service for monitoring the amount of the cloud-based game consoles available in each of the environments, wherein the resource management service includes a first resource management module for measuring the load on the cloud-based game consoles in each of the environments due to the activity of the human users, a second resource management module for predicting future loads due to the activity of the human users, and a third resource management module for determining the cloud-based game consoles to be given to research users, and a method further comprising the steps of reading the number of available cloud-based game consoles by a resource control system, the resource control system identifying one or more experiments that should be suspended, and the one or more experiments identified as needing to be suspended being moved to a <suspended> state.

[0008]

[0007] Embodiments of the present invention also include a method for managing computing resources of a cloud-based game console between a human user and a research user, comprising the steps of: providing a resource management service for the computing resources, the resource management service comprising: a first resource management module for measuring the load caused by human activity; a second resource management module for predicting future loads caused by human activity; and a third resource management module for determining the computing resources to be given to the research user; and providing a resource control service for the computing resources, the resource control service comprising: a first resource control module for reading the number of available computing resources; and a second resource control module for identifying one or more experiments that should be interrupted; and the one or more experiments that have been identified as needing to be interrupted. The present invention provides a method comprising the steps of: transitioning an experiment to a <suspended> state; assigning the resource control service to each environment, including the cloud-based game console, wherein each resource control service monitors the one or more experiments in the <suspended> state; terminating each process under the control of the resource control service for the one or more experiments in the <suspended> state; changing the status of each process under the control of the resource control service to a <suspended> state when each process under the control of the resource control service has been terminated; and changing the one or more experiments to a <suspended> state when each of the resource control services in each environment, including the computing resources, has its respective process for the one or more experiments in a <suspended> state.

[0009]

[0008] These and other features, aspects and advantages of the present invention will be better understood by referring to the following drawings, description and claims.

[0010]

[0009] Several embodiments of the present invention are shown by illustration of the accompanying drawings, where the same reference numerals may indicate similar elements, not as limitations but as examples. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows an exemplary system architecture for training an agent using a game console, according to an embodiment of the present invention. [Figure 2] This figure shows the resources used in the system architecture shown in Figure 1. [Figure 3] This figure shows a schematic diagram of a user computing device used in an architecture and method according to an exemplary embodiment of the present invention. [Figure 4] This figure shows modules provided by resource management services and resource control services for controlling resource usage in an architecture and method according to an exemplary embodiment of the present invention. [Modes for carrying out the invention]

[0012]

[0014] Unless otherwise specified, the figures within the diagrams are not necessarily drawn to scale.

[0013]

[0015] The present invention and its various embodiments can be better understood by referring to the following detailed description of the illustrated embodiments. It should be clearly understood that the illustrated embodiments are for illustrative purposes only and do not limit the present invention to the final claims.

[0014]

[0016] The terms used herein are for the purpose of describing specific embodiments and are not intended to limit the invention. The terms “and / or” as used herein include any combination of one or more of the items relating to the description. The singular forms “a, an” and “the” as used herein are intended to include the plural form as well as the singular form unless otherwise explicitly indicated in the context. Furthermore, the terms “comprises and / or comprising” as used herein indicate the presence of the features, steps, actions, elements and / or components referred to, but should be understood not to exclude the presence or addition of one or more other features, steps, actions, elements, components and / or groups thereof.

[0015]

[0017] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as those generally understood by those skilled in the art to which the present invention pertains. Furthermore, terms defined in commonly used dictionaries should be interpreted to have the meanings corresponding to those meanings in the context of the relevant art and this disclosure, and should not be interpreted in an ideal or overly formal sense unless explicitly defined herein.

[0016]

[0018] It will be understood that the description of this invention discloses multiple techniques and steps. Each of these has its own individual benefit and can be used in conjunction with one or more, or possibly all, of the other techniques disclosed. Therefore, for clarity, this description avoids unnecessarily repeating all possible combinations of the individual steps. However, this specification and the claims should be read with the understanding that such combinations are fully included within the scope of the invention and the claims.

[0017]

[0019] The following description provides numerous specific details for illustrative purposes to provide a complete understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be carried out without these specific details.

[0018]

[0020] This disclosure should be considered illustrative of the present invention and is not intended to limit the invention to specific embodiments shown by the following figures or description.

[0019]

[0021] Devices or system modules that communicate with each other using general communication do not need to communicate with each other continuously unless otherwise specified. Furthermore, devices or system modules that communicate with each other using general communication can communicate directly or indirectly through one or more intermediary devices.

[0020]

[0022] The description of embodiments including multiple components that communicate with each other does not mean that all such components are necessary. Rather, various optional components are described to illustrate a wide range of possible embodiments of the present invention.

[0021]

[0023] "Computer" or "computing device" can mean one or more devices and / or one or more systems that can accept structured input, process the structured input according to defined rules, and generate the result of the processing as output. Examples of a computer or computing device include a computer, a fixed and / or portable computer, a single processor, multiple processors, or a computer having a multi-core processor that can operate in parallel and / or not in parallel, a supercomputer, a mainframe, a super-minicomputer, a minicomputer, a workstation, a microcomputer, a server, a client, a two-way television, a web appliance, a communication device having Internet access, a hybrid combination of a computer and a two-way television, a portable computer, a tablet personal computer (PC), a personal digital assistant (PDA), a mobile phone, such as a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an application specific instruction set processor (ASIP), a chip, multiple chips, a system on a chip or a chip set, etc., a specific-purpose hardware that emulates a computer and / or software, a data collection device, an optical computer, a quantum computer, a bio computer, and generally a device that can accept data, process the data according to one or more stored software programs, generate a result, and typically can include an input device, an output device, a storage device, an arithmetic unit, a logic unit, and a control unit.

[0022]

[0024] "Software" or "application" can mean defined rules for operating a computer. Examples of software or an application include code segments in one or more computer-readable languages, graphic and / or text instructions, applets, pre-compiled code, interpreted code, compiled code, and computer programs.

[0023]

[0025] Also, by storing these computer program instructions, which can be used to direct a computer, other programmable data processing apparatus, or other devices to function in a specific manner, in a computer-readable medium, it is possible to create an article of manufacture in which the instructions stored in the computer-readable medium include instructions for implementing the functions / operations specified within one or more blocks of a flowchart and / or block diagram.

[0024]

[0026] Further, process steps, method steps, algorithms, etc. can be described in a certain order, but such processes, methods, and algorithms can also be configured to function in a different order. In other words, any order or sequence of steps that can be described does not necessarily indicate that these steps must be executed in this order. The process steps described herein can be executed in any practical order. Additionally, several steps can be executed simultaneously.

[0025]

[0027] It will be readily apparent that the various methods and algorithms described herein can be implemented, for example, by appropriately programmed general-purpose computers and computing devices. Typically, a processor (e.g., a microprocessor) receives instructions from memory or a similar device and executes these instructions to perform the processes defined by these instructions. Further, programs implementing such methods and algorithms can be stored and transmitted using various known media.

[0026]

[0028] As used herein, the term “computer-readable medium” means any medium involved in providing data (e.g., instructions) that can be read by a computer, processor, or similar device. Such mediums can take many forms, including, but are not limited to, non-volatile, volatile, and transmitting media. Non-volatile media include, for example, optical or magnetic disks and other persistent memory. Volatile media include dynamic random-access memory (DRAM), which typically constitutes main memory. Transmitting media include coaxial cables, copper wires, and optical fibers, including wires that include a system bus coupled to a processor. Transmitting media include, or can transmit, sound waves, light waves, and electromagnetic radiation, such as those generated during radio frequency (RF) and infrared (IR) data communications. Examples of common forms of computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs, any other optical media, punch cards, paper tapes, any other physical media having a hole pattern, RAM, PROMs, EPROMs, FLASHEEPROMs, any other memory chips or cartridges, carrier waves as described below, or any other media that can be read by a computer.

[0027]

[0029] Various forms of computer-readable media can be involved in transporting a series of instructions to the processor. For example, a series of instructions can be (i) supplied to the processor from RAM, (ii) transported via a wireless transmission medium, and / or (iii) formatted according to numerous formats, standards, or protocols such as Bluetooth, TDMA, CDMA, 3G, 4G, and 5G.

[0028]

[0030] Embodiments of the present invention may include devices that perform the operations disclosed herein. These devices may be specifically configured for a desired purpose, or they may include general-purpose devices that are selectively operated or reconfigured by internally stored programs.

[0029]

[0031] Unless otherwise stated, and as may become apparent from the following description and claims, throughout this specification, any use of terms such as “process,” “calculate,” “calculate,” or “determine” should be understood to mean the operation and / or process of a computer, computer system, or similar electronic computing device that manipulates data represented as physical quantities, such as electron quantities in the registers and / or memory of a computer system, and / or transforms it into other data similarly represented as physical quantities in the memory, registers, or other such information storage, transmission, or display device of a computer system.

[0030]

[0032] Similarly, the term “processor” can mean any device or part of a device that processes electronic data from registers and / or memory and converts this electronic data into other electronic data that can be stored in registers and / or memory or transmitted to an external device, so as to cause a physical change or operation of an external device.

[0031]

[0033] The terms “agent,” “intelligent agent,” “artificial agent,” or “artificial intelligence agent” are intended to mean any artificial entity that chooses actions in response to observation. “Agent” can mean, without limitation, a robot, a simulated robot, a software agent or “bot,” an adaptive agent, an internet or webbot.

[0032]

[0034] In general, embodiments of the present invention provide an artificial intelligence agent capable of functioning as a player of a video game, such as a racing video game. The game is entirely external to the agent and can be run in real time. Thus, the training system is very similar to a real-world system. The console on which the game is run to train the agent is provided in a cloud computing environment. The agent and trainer can run on other computing devices in the cloud, and the system can select trainers and agents calculated based, for example, on proximity to the console. Users can select the game they want to run and submit code that can be built and deployed to the cloud system. A resource management service can monitor the resources of the game console among human users, investigate usage, and identify experiments for interruptions to ensure sufficient game consoles for human users.

[0033]

[0035] Referring to Figures 1-4, the basic workflow can be assumed to be as follows:

[0034] On the user's local machine

[0036] On the user's local machine 10, the user can write a computer program (usually in Python, for example) for an agent. This program is called a "data gatherer" 12, and such an agent can be programmed to interact with a game and learn how to control it. The user can also write a computer program (usually in Python, for example) for a "trainer" 14. The trainer 14 can be programmed to learn how to take experience from the data gatherer 12 and use it to improve the agent's (data gatherer 12's) strategy. The trainer 14 can use any number of algorithms and neural network structures that may exist in the artificial intelligence (AI) library 16. The user can write a third program that defines an experiment 18. This program is usually in the form of a configuration file. The configuration file is written in a human-readable data serialization language such as YAML, for example, and can define the number of data gatherers 12 to use, the computing power required for the data gatherers 12 and trainer 14, the algorithms that the trainer 14 should use, and the set of tasks that the trainer 14 should have the data gatherers 12 perform.

[0035]

[0037] The user can check in code (data gatherer 12, trainer 14, and experiment definition 18) to a source code repository 22 such as GitHub. The user can execute command-line programs via the command-line interface 23. This submits a request to the build system 26 in the build environment 20 to build the experiment if an existing Docker image cannot be reused. The user then requests the server 52 in the monitoring environment 57 to run the experiment identified by its source code check-in reference hash via the data query interface 25. The system server 52 can store information about the requested experiment in the database 56 with the status <submitted>. In some embodiments, there may also be a web interface 24 that allows the user to request execution. As shown in Figure 1, the web interface 24 and the command-line interface 23 can interact with a data query and manipulation interface 25 such as Hasura / GraphQL to allow the user to review the experiment during or after its execution, as described below. Of course, other query interfaces can also be used for the user to review the data.

[0036] inside the cloud

[0038] In a cloud computing environment, the build system 26 can build the user's code into a Docker image 28. The build system 26 can be any virtual machine imaging system, such as CircleCI. If the experiment requires resources from the cloud game system 30 (also called the production build environment 30), the production build environment 30 can pull code from the development build environment 20, and its build system can perform various secondary security assessments using a source code repository 32 such as GitLab, and then build the user's code into a Docker image 36 using a Docker build 34. The system can set the experiment state to <building> and record which environment (such as data center DC-1 (environment 38) and data center DC-2 (environment 40), as shown in Figure 2) is building the experiment. This specification uses Docker images for explanation, and Figure 1 shows Kubernetes 44 as a container orchestration system for interfaced with the Docker image 36, but it should be understood that other types of architectures can be used to achieve the same purpose. For example, the Docker runtime can be replaced with a runtime that conforms to the Kubernetes container runtime interface. Similarly, the container orchestration system Kubernetes can be replaced by other orchestration systems such as Slurm.

[0037]

[0039] Periodically, the resource control service 42 in each environment 38, 40 can examine the build system 30 within its view and find building experiments. Once complete, the resource control service 42 can transition the experiment state in that environment to the built state. The system monitors the transition to built in each environment, and once all necessary environments are complete, it can change the overall experiment state to built. The system monitors the experiments in the built state and can transfer them to the queued state.

[0038]

[0040] Periodically, the system can evaluate experiments in a <standby> state and decide whether to start them. When deciding whether to start an experiment, the system may consider factors such as the experiment's priority level, the elapsed time of the experiment, whether the resources required for the experiment are available in an acceptable environment, and other criteria for scheduling the experiment (such as quota limits imposed by the user or project).

[0039]

[0041] If the system decides to start an experiment, it can mark it as a <scheduled> and tag it with an identifier for the resources it should consume. For example, the system might decide that a particular experiment should be run in specific environments 38, 40 using a game console 46 (e.g., PS4) and a data gatherer 12. The experiment can be run using a GPU (e.g., V100 GPU 48) for a trainer 14 in the same or a different environment, and annotations can be added to the experiment to record those decisions.

[0040]

[0042] Periodically, the resource control service 42 in each target environment can check if there are any experiments in the <scheduled> state tagged to start within that environment 38, 40. If so, it can start the necessary resources.

[0041] When starting Data Gatherer

[0043] Technically, the data gatherer can be any program. In the context of the embodiments of the present invention, the data gatherer 12 may be playing a game (such as a PlayStation® game) within the network of the cloud game system production environment 50.

[0042]

[0044] The data gatherer 12 can find and connect to a trainer 14 that is operating as specified by the system server 52. The data gatherer 12 can request a user ID for the game system from a service that manages user IDs for training agents. The data gatherer 12 can request a console 46 available within the cloud game system 50 and also request that a specific game be loaded.

[0043]

[0045] Next, the data gatherer 12 can request a task from the trainer 14. A task is essentially a game configuration that the data gatherer 12 should play. For example, in a racing game, one task could cause the data gatherer 12 to start a cluster of five cars placed at equal intervals around a track. Each cluster would include one car controlled by an agent and three cars controlled by the game's built-in AI.

[0044]

[0046] Data Gatherer 12 can start a game, transmit the scenario structure to the game, and then begin playing. While an agent is playing the game, they can send their experience to Trainer 14.

[0045]

[0047] Periodically, the data gatherer 12 can fetch updated models from the trainer 14. Optionally, the data gatherer 12 can also send metrics to the database 56 via the data query interface 25 during or after the scenario. For example, the data gatherer 12 can report its best lap time. Optionally, the data gatherer 12 can store other data, such as complete race data, in a remote data store 58 such as S3. Optionally, the data gatherer 12 can be configured to stream the video output of a cloud game console to S3 for later viewing by the experimenter.

[0046]

[0048] Once the task completion criteria are met, the data gatherer 12 can complete the scenario on the cloud game console 46 and request a new task from the trainer 14.

[0047] When the trainer starts up

[0049] Trainer 14 can initialize a buffer that can store the experience reported by Data Gatherer 12. Optionally, it can load a buffer from a previous run. Trainer 14 maintains a list of tasks and can pass new tasks from there to Data Gatherer 12 when requested.

[0048]

[0050] Periodically, the trainer 14 loads experience from a buffer and updates the neural network model using a learning algorithm. Optionally, the trainer 14 reports metrics to the system, and such metrics are stored in the metrics database 56. The updated neural network model can then be sent to the data gatherer 12.

[0049] On the user's machine

[0051] While an experiment is being built and running, users can monitor it using, for example, a web browser 24 connected to the system server 52 via the data query interface 25. The system interface can show the progress throughout the experiment's build and deployment phases. Once the experiment is running, the system interface can enable users to inspect metrics and create dashboards that display various performance graphs. The system interface can also be used to graph metrics across multiple runouts simultaneously, allowing users to compare the performance of different experiments.

[0050] Suspension and resumption

[0052] The resource management service 60 (also simply called the resource manager 60) is the name of a service that the cloud gaming system has to coordinate resources with external services. Since training is performed on an actual game console, the training system shares the game system (such as the PlayStation® Network) with humans. If more people want to play the game, the resource management service 60 instructs the training system 50 to reduce its usage. When humans stop playing, the resource management service 60 provides the training system 50 with even more resources. The system server 52 uses the resource control service 42 (also called the experiment manager 42) to adjust resource usage based on the goals set by the resource management service 60.

[0051]

[0053] As will be described in more detail below, some key features of the integration of the training system and the cloud gaming system are (1) a module 62 that measures the load caused by human activity, (2) a module 64 that predicts future loads, and (3) a module 66 that determines how much of these resources can be given to researchers. The resource control service 42 may provide functions including (4) a module 68 that reads the number of available resources, (5) a module 70 that starts and stops experiments according to resource constraints and job priority / elapsed time / quota, and (6) a module 72 that restarts the job in an environment where resources are available. Modules 70 and 72 may be part of the system server 52. In some embodiments, experiments may run in multiple environments, but the resource control service 42 (experiment manager 42) controls resources in only one environment. In some embodiments, for example, if the system server 52 is not working or does not work fast enough, the resource management service 60 may terminate the experiment according to a pre-programmed protocol, such as first-in, first-out.

[0052]

[0054] The training system can monitor the cloud game system's resource management service 60. When the system detects that the resources allocated to the training system (particularly the cloud game console) have fallen below the system's current usage, it can identify one or more experiments that should be suspended. When deciding which experiments to suspend, the system may consider the location of the resources used by the experiment, the experiment's priority level, the experiment's elapsed time, the experiment's user ID, and / or other attributes related to the experiment. The system can then move the selected experiments to a <suspending> state.

[0053]

[0055] Each resource control service 42 in each environment (locations 38, 40, etc.) can periodically check the system server to see if a running experiment has transitioned to a <suspended> state. If so, the resource control service can terminate the processes under its control. When a trainer requests a suspension, state information (especially the experience buffer) can be saved to remote storage so that it can be reloaded later before a proper shutdown.

[0054]

[0056] Once all processes under each control have finished, the resource control service 42 changes that portion of the experiment to the <suspended> state. When all related resource control services 42 have transitioned that portion to <suspended>, the system can transition the entire experiment to the <suspended> state.

[0055]

[0057] The system, in the resource control service 42, can examine a list of suspended executions once it confirms that the number of available resources is greater than the number of resources currently in use. The system can then resume some of these experiments. When selecting which experiments to resume, the system may consider the location of the resources used by the experiment, the experiment's priority level, the experiment's elapsed time, the experiment's user ID, and / or other attributes related to the experiment.

[0056]

[0058] To avoid thrashing, the system server can smooth the resource availability signals it receives from the resource management service 60. The system server can smooth these signals by applying any number of standard algorithms, such as low-pass filters and minimum-maximum time windows. Optionally, the user can interrupt a running experiment by clicking a button. This experiment transitions to the <manually suspended> state. The user can choose to reactivate a manually suspended experiment by clicking a button on the user interface. The system transitions the experiment to <suspended>. The system reactivates the experiment according to the same conditions as above when the resource becomes available.

[0057] completion

[0059] The user can write termination conditions in the trainer script so that when certain conditions are met, the system reports completion and terminates. The system changes the experiment status to <Success>. Alternatively, the user can also click the "Cancel" button using the system interface. The system shuts down the experiment immediately after a process similar to the interruption process described above, but the current experiment status is not saved. The system sets the experiment status to <Cancelled>.

[0058]

[0060] Those skilled in the art can make numerous changes and modifications without departing from the spirit and scope of the present invention. Therefore, the illustrated embodiments are merely examples and should not be interpreted as limiting the present invention as defined by the following claims. For example, while the elements of the claims are shown below in specific combinations, it should be clearly understood that the present invention includes other combinations of elements, fewer, more, or different from those disclosed.

[0059]

[0061] The words used herein to describe the present invention and its various embodiments should be understood to include not only their generally defined meanings but also, by their specific definitions herein, comprehensive structures, materials, or actions that represent a single kind.

[0060]

[0062] Therefore, in this specification, the definitions of words or elements in the following claims are not limited to combinations of elements explicitly stated in the text. In this sense, it is conceivable that one of any of the elements in the following claims may be replaced by two or more equivalent substitutes of elements, or a single element may be replaced by two or more elements of the claims. While elements are described above as functioning in specific combinations, and claims may initially be made in this manner, it should be clearly understood that, in some cases, one or more elements resulting from the claimed combinations may be removed from these combinations, and the claimed combinations may be directed towards lower combinations or variations of lower combinations.

[0061]

[0063] It is explicitly assumed that any minor changes from the subject matter, as seen by those skilled in the art, whether currently known or later devised, are equally included in the claims. Therefore, obvious substitutions, both currently known and hereafter known to those skilled in the art, are also defined as being within the scope of the specified elements.

[0062]

[0064] Therefore, the claims should be understood to include those specifically illustrated and described above, those that are conceptually equivalent, those that are clearly substituted, and those that incorporate the fundamental ideas of the present invention. [Explanation of Symbols]

[0063] 10 users' local machines 12 Data Gatherer 14 Trainer 16. Artificial Intelligence (AI) Library 18. Experimental Definition 20 Build Environment 22 Source Code Repositories 23 Command Line Interface 24 Web Interface 25 Data Query Interface 26 Build Systems 28 Docker images 30 Cloud gaming system / production build environment 32 Source Code Repositories 34 Docker builds 36 Docker images 38 Environment 40 Environment 42 Resource Control Service / Experiment Manager 44 Container Orchestration System 46 Game Consoles 48 V100 GPU 50 Cloud Gaming System Production Environment 52 servers 56 Databases 57 Monitoring environment 58 Remote Datastore 60 Resource Management Services / Resource Manager 62 Measuring the activity of human users 64 Predicting future loads 66. Determine the resources for training. 68 Read the number of available resources. 70. Start and stop experiments according to resource constraints. 72. Restart the experiment.

Claims

1. A method for managing computing resources between a human user and a research user performing training experiments on an artificial intelligence agent, wherein the method is: Computers A step of providing a resource management service for the computing resources, wherein the resource management service is A first resource management module that measures the load caused by human activity, A second resource management module predicts future loads caused by human activity, A third resource management module that determines the computing resources to be provided to the research user, Steps including, A step of providing a resource control service for the computing resource, wherein the resource control service is A first resource control module reads the number of available computing resources, A second resource control module that identifies one or more artificial intelligence agent training experiments that should be suspended, Steps including, Steps include: transitioning the training experiment of the one or more artificial intelligence agents identified as needing to be interrupted to a <suspending> state; A method characterized by including the execution of

2. The method according to claim 1, characterized in that the computing resource is a cloud-based game console.

3. The method according to claim 1, further comprising the step of determining which of the one or more training experiments should be interrupted based on at least one of the following: the location of the computing resources used by the one or more training experiments, the priority level of the one or more training experiments, the elapsed time of the one or more training experiments, and / or the user ID of the one or more training experiments.

4. The method according to claim 1, further comprising the step of assigning the resource control service to each environment including the computing resources, wherein each resource control service monitors the one or more training experiments in the <suspended> state.

5. The method according to claim 4, further comprising the step of the computer terminating each process under the control of the resource control service for the one or more training experiments in the <suspended> state.

6. The method according to claim 5, further comprising the step of the computer saving state information to remote storage before terminating each process.

7. The method according to claim 1, further comprising the step of the computer changing the status of each process under the control of the resource control service to a <suspended> state when each process under the control of the resource control service has terminated.

8. The method according to claim 7, further comprising the step of changing the one or more training experiments to a <suspended> state when each of the resource control services in each environment including the computing resources has put each process for the one or more training experiments into a <suspended> state.

9. The method according to claim 1, further comprising the step of the computer resuming a suspended training experiment when it determines that the computing resources are available to the resource management service.

10. The method according to claim 9, further comprising the step of selecting the interrupted training experiment to be resumed based on at least one of the following: the location of the computing resources used by the one or more training experiments, the priority level of the one or more training experiments, the elapsed time of the one or more training experiments, and / or the user ID of the one or more training experiments.

11. The method according to claim 9, further comprising the step of the computer smoothing signals relating to the availability of computing resources in order to avoid thrashing.

12. The method according to claim 1, further comprising the step of the computer providing the user with the option to manually interrupt and manually resume a training experiment.

13. A method for training an artificial intelligence agent to play video games on a cloud-based game console shared with human users, Computers The steps include providing the artificial intelligence agent for interacting within the video game, A step of configuring a trainer to evaluate the experiential information obtained from the artificial intelligence agent and to improve the strategy of the artificial intelligence agent for interacting with the video game, The steps include: monitoring the status of the training experiment of the artificial intelligence agent using a resource control service, and determining whether to execute the training experiment when it enters a scheduling state; The steps include: starting the training experiment on a predetermined number of cloud-based game consoles using a predetermined number of data gatherers in one or more predetermined environments; A step of providing a resource management service for monitoring the number of cloud-based game consoles available in each of the aforementioned environments, wherein the resource management service is A first resource management module for measuring the load caused by the activity of the human user on the cloud-based game console in each of the aforementioned environments, A second resource management module that predicts future loads resulting from the activities of the aforementioned human users, A third resource management module that determines the cloud-based game console to be provided to research users who conduct training experiments on artificial intelligence agents, Steps including, The resource control system reads the number of available cloud-based game consoles, The resource control system identifies one or more training experiments that should be interrupted, A step of transitioning one or more training experiments identified as needing to be interrupted to a <suspended> state, A method characterized by including the execution of

14. The method according to claim 13, further comprising the step of determining which of the one or more training experiments should be interrupted based on at least one of the following: the location of the cloud-based game console used in the one or more training experiments, the priority level of the one or more training experiments, the elapsed time of the one or more training experiments, and / or the user ID of the one or more training experiments.

15. The method according to claim 13, further comprising the step of assigning the resource control service to each environment, including the cloud-based game console, wherein each resource control service monitors the one or more training experiments in the <suspended> state.

16. The computer terminates each process under the control of the resource control service for the one or more training experiments in the <suspended> state, The steps include: changing the status of each process under the control of the resource control service to the <suspended> state when each process under the control of the resource control service terminates; When each of the resource control services in each environment, including computing resources, has its respective process for the one or more training experiments in a <suspended> state, the step of changing the one or more training experiments to a <suspended> state, The method according to claim 15, further comprising performing the following:

17. The method according to claim 16, further comprising the step of the computer saving state information to remote storage before terminating each process.

18. The method according to claim 16, further comprising the step of the computer resuming a suspended training experiment when the computing resources are determined to be available by the resource management service.

19. A method for managing computing resources of a cloud-based game console between a human user and a research user performing training experiments on an artificial intelligence agent, wherein the method is: Computers A step of providing a resource management service for the computing resources, wherein the resource management service is A first resource management module that measures the load caused by human activity, A second resource management module predicts future loads caused by human activity, A third resource management module that determines the computing resources to be provided to the research user, Steps including, A step of providing a resource control service for the computing resource, wherein the resource control service is A first resource control module reads the number of available computing resources, A second resource control module that identifies one or more training experiments that should be interrupted, Steps including, A step of transitioning one or more training experiments identified as needing to be interrupted to a <suspended> state, A step of assigning the resource control service to each environment, including the cloud-based game console, wherein each resource control service monitors the one or more training experiments in the <suspended> state, With respect to the one or more training experiments in the <suspended> state, the steps include terminating each process under the control of the resource control service, The steps include: changing the status of each process under the control of the resource control service to the <suspended> state when each process under the control of the resource control service terminates; When each of the resource control services in each environment including the computing resources has put each of the processes for the one or more training experiments into a <suspended> state, the step of changing the one or more training experiments to a <suspended> state, A method characterized by including the execution of

20. The method according to claim 19, further comprising the step of a computer determining which of the one or more training experiments should be interrupted based on at least one of the following: the location of the cloud-based game console used by the one or more training experiments, the priority level of the one or more training experiments, the elapsed time of the one or more training experiments, and / or the user ID of the one or more training experiments.

Images