Methods for training artificial intelligence agents using curriculum and skills

Abstract

The method for training an agent uses a mix of scenarios designed to teach specific skills useful in a larger domain, such as a mix of general racing and very specific tactical racing scenarios. Aspects of the method can include one or more of: (1) training an agent to be very good at time trials with one or more cars distributed around a track, (2) running the agent in various racing scenarios with a variable number of opponents starting in different configurations around the track, (3) varying the opponents using agents provided in the game, agents trained according to aspects of the invention, or agents controlled to follow specific driving lines, (4) setting up specific short scenarios involving opponents in various racing situations with specific success criteria, and (5) having a dynamic curriculum based on how the agent performs in various evaluation scenarios.

Description

Technical Field

[0001]

[0001] Embodiments of the present invention generally relate to the training of artificial intelligence agents. More specifically, the present invention relates to a method for training game agents both in general gameplay and in placing agents in specific scenarios. Even more specifically, aspects of the present invention can use mixed scenario training for reinforcement learning in configurable environments such as racing game agents.

Background Art

[0002]

[0002] The following background information can present examples of specific aspects of the prior art (e.g., without limitation, methods, facts, or general ideas), and these examples are expected to help convey further aspects of the prior art to the reader, but should not be construed as limiting the present invention or any of its embodiments to any matter mentioned, implied, or speculated about therein.

[0003]

[0003] Referring to FIG. 1, a reinforcement learning (RL) agent 100 is a form of artificial intelligence that is trained (also called "learning") through interaction with an environment 102. At each time step of the agent's training, an observation 104 of its current state is provided to the agent. Next, the agent 100 executes an action 106, which causes the agent to transition to a new state and generate a reward condition. Various existing RL algorithms and models 108 provide routines for ultimately finding an optimal policy 110 (a mapping from states to actions) that maximizes some function of the reward condition (such as the expected total).

[0004]

[0004] In simple domains, it is expected that an RL agent can experience all possible states based on its own actions. However, in complex problems such as learning to drive or racing autonomous vehicles in traffic, a learning agent that requires data from useful learning scenarios, such as driving between two cars, faces many challenges. For example, if the environment is large enough, random or targeted search by a learning agent may have too much area to explore, making it likely that important scenarios will be missed.

[0005]

[0005] Furthermore, since reinforcement learning agents almost always have a finite horizon (or effective finite horizon) in their planning, even if they identify a scenario they want to visit, it may not be practical to execute a plan to reach it. For example, if the agent needs data between two cars, but the cars are far apart, the agent may not have a reliable way to reach the cars.

[0006]

[0006] Furthermore, when other agents share an environment, they may not perform actions that would lead to the experience the learning agent needs. For example, if an autonomous driving system is a racing simulator, the two cars will not slow down to allow the learning agent to get in between them. Even if the learning agent were to somehow experience a scenario in which it drives between the two cars, the amount of experience required to reach that scenario may be far greater than the experience gained in the scenario, and the impact on training would be minimal.

[0007]

[0007] Furthermore, in complex environments, there is often a strong prior knowledge gained from human experience about which scenarios are helpful for learning. For example, a driving instructor may be able to define many scenarios that are helpful for the agent's learning. However, encodings of these known learning scenarios are not available in the basic reinforcement learning formulation. [Overview of the Initiative] [Problems that the invention aims to solve]

[0008]

[0008] To effectively learn in such a complex domain without simply running the agent through a huge number of steps, these problems need to be solved.

[0009] A complex environment requiring multiple skills for an agent to succeed is simulated car racing. To skillfully control a race car, the driver needs a detailed understanding of the vehicle's dynamics and the characteristics (idiosyncrasies) of the track on which the race is being held. On top of this foundation, the driver must acquire the tactical skills necessary to overtake or defend against opponents, and perform high-speed, precise maneuvers with almost no room for error. [Means for solving the problem]

[0010]

[0009] Embodiments of the present invention provide a method for training a reinforcement learning agent in mixed scenario training, comprising the steps of: providing a rollout worker to an environment having one or more predetermined scenario characteristics; operating the rollout worker in the environment while focusing on one or more specific skills; providing a reward when the one or more specific skills are successfully achieved; and creating a policy for the rollout worker to optimize the reward.

[0011]

[0010] Embodiments of the present invention provide a deep reinforcement learning architecture that uses mixed scenario training, comprising a set of rollout workers, a trainer, and a set of scenario characteristics, wherein the trainer refines models and policies used to determine the behavior of the rollout workers in an environment, the rollout workers operate in the environment based on predetermined activation conditions taken from the scenario characteristics, and data from the rollout workers operating in the environment including the predetermined activation conditions is collected and stored in the trainer's experience replay buffer.

[0012]

[0011] Embodiments of the present invention also provide a method for training an agent using deep reinforcement learning to interact in a racing video game, comprising the steps of: learning a policy for selecting actions based on observations by the agent and on a value function that estimates future rewards for each possible action; mapping the agent's core actions to either a changing velocity dimension or a steering dimension, wherein both the changing velocity dimension and the steering dimension are continuous-valued dimensions; and training the agent in an environment including predetermined scenario characteristics, wherein the predetermined scenario characteristics include a start condition, opponent distribution options, replication count, stop condition, experience table mapping, and scenario weighting.

[0013]

[0012] In some embodiments, the method further includes the steps of providing the agent with position, velocity, and acceleration state information for the agent itself and each opponent, and providing the agent with a map of the track as a list of points defining the left end, the right end, and the centerline.

[0014]

[0013] In some embodiments, the method further includes the steps of training an agent in a race scenario involving a variable number of opponents starting in different configurations around a track, and training an agent against opponents selected from artificial agents provided in a game, other agents trained with various reward functions, and controller-controlled agents that follow specific driving lines.

[0015]

[0014] These and other features, aspects and advantages of the present invention will be better understood by referring to the following drawings, description and claims.

[0016]

[0015] Several embodiments of the present invention are shown by the figures in the attached drawings, where the same reference numerals may indicate similar elements, not as limitations but as examples. [Brief explanation of the drawing]

[0017] [Figure 1] This diagram shows a schematic representation of how a reinforcement learning agent interacts with its environment. [Figure 2] This figure shows a scenario component of a single environment for mixed-scenario training and its consumer, according to an exemplary embodiment of the present invention. [Figure 3] This figure shows a schematic diagram of a complete mixed-scenario training according to an exemplary embodiment of the present invention. [Figure 4] This diagram illustrates an exemplary system configuration in which a trainer distributes training scenarios to rollout workers, and each rollout worker controls a single game console running an instance of a racing game. [Figure 5a] This figure shows three different racing track scenario configurations used to train agents using mixed-scenario training. [Figure 5b] This figure shows three different racing track scenario configurations used to train agents using mixed-scenario training. [Figure 5c] A diagram showing the scenario configurations on three different racing tracks used to train an agent using mixed scenario training. [Figure 5d] A diagram showing the scenario configurations on three different racing tracks used to train an agent using mixed scenario training. [Figure 5e] A diagram showing the scenario configurations on three different racing tracks used to train an agent using mixed scenario training. [Figure 5f] A diagram showing the scenario configurations on three different racing tracks used to train an agent using mixed scenario training. [Figure 6a] A diagram showing an ablation study of various aspects of the scenarios used to train a racing agent. When the scenarios are changed or removed, the agent's ability to acquire and maintain important skills decreases. [Figure 6b] A diagram showing an ablation study of various aspects of the scenarios used to train a racing agent. When the scenarios are changed or removed, the agent's ability to acquire and maintain important skills decreases. [Figure 7] A diagram showing the content of an experience replay buffer that organizes data from different scenarios using multiple tables. [Figure 8a] A diagram showing the race results when an agent trained in a mixed scenario competes against four top drivers in a simulated racing game. [Figure 8b] A diagram showing the race results when an agent trained in a mixed scenario competes against four top drivers in a simulated racing game. [Figure 8c] A diagram showing the race results when an agent trained in a mixed scenario competes against four top drivers in a simulated racing game.

Best Mode for Carrying Out the Invention

[0018]

[0024] Unless otherwise indicated, the figures in the drawings are not necessarily drawn to scale.

[0019]

[0025] By referring to the following detailed description that explains the illustrated embodiments, the present invention and its various embodiments can be better understood. It should be clearly understood that the illustrated embodiments are shown as examples and do not limit the present invention as finally defined in the claims.

[0020]

[0026] The terms used in this specification are only for explaining specific embodiments and are not intended to limit the present invention. The term "and / or" used in this specification includes any and all combinations of one or more of the related listed items. The singular forms "a, an" and "the" used in this specification are intended to include the plural forms as well, unless otherwise clearly indicated in the context. Furthermore, the terms "comprises and / or comprising" when used in this specification indicate the presence of the recited features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0021]

[0027] Unless otherwise specified, all terms (including technical and scientific terms) used in this specification have the same meaning as commonly understood by those skilled in the art to which the present invention pertains. Furthermore, terms defined as in commonly used dictionaries should be interpreted to have meanings consistent with their meanings in the context of the related art and this disclosure, and should not be interpreted in an ideal or overly formal sense unless clearly defined in this specification.

[0022]

[0028] It will be understood that the description of this invention discloses multiple techniques and steps. Each of these has its own individual benefit and can be used in conjunction with one or more, or possibly all, of the other techniques disclosed. Therefore, for clarity, this description avoids unnecessarily repeating all possible combinations of the individual steps. However, this specification and the claims should be read with the understanding that such combinations are fully included within the scope of the invention and the claims.

[0023]

[0029] The following description provides numerous specific details for illustrative purposes to provide a complete understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be carried out without these specific details.

[0024]

[0030] This disclosure should be considered illustrative of the present invention and is not intended to limit the invention to specific embodiments shown by the following figures or description.

[0025]

[0031] Devices or system modules that communicate with each other using general communication do not need to communicate with each other continuously unless otherwise specified. Furthermore, devices or system modules that communicate with each other using general communication can communicate directly or indirectly through one or more intermediary devices.

[0026]

[0032] The description of embodiments including multiple components that communicate with each other does not mean that all such components are necessary. Rather, various optional components are described to illustrate a wide range of possible embodiments of the present invention.

[0027]

[0033] "Computer" or "computer device" can mean one or more devices and / or one or more systems that accept structured input, process the structured input according to prescribed rules, and produce the results of the processing as output. Examples of computers or computer devices include computers, fixed and / or portable computers, computers having a single processor, multiple processors, or multiple core processors that can operate in parallel and / or non-parallel, supercomputers, mainframes, superminicomputers, minicomputers, workstations, microcomputers, servers, clients, interactive televisions, web appliances, communication devices with internet access, hybrid combinations of computers and interactive televisions, portable computers, tablet personal computers (PCs), personal digital assistants (PDAs), mobile phones, application-specific hardware that emulates computers and / or software, such as digital signal processors (DSPs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific instruction set processors (ASIPs), chips, multiple chips, systems on a chip or chipsets, data acquisition devices, optical computers, quantum computers, biocomputers, and devices that generally accept data, process the data according to one or more stored software programs, produce results, and typically include input devices, output devices, memory devices, arithmetic units, logic units and control units.

[0028]

[0034] "Software" or "application" can mean a set of rules for operating a computer. Examples of software or applications include code segments in one or more computer-readable languages, graphic and / or text instructions, applets, precompiled code, interpreted code, compiled code, and computer programs.

[0029]

[0035] Furthermore, by storing these computer program instructions, which can instruct a computer, other programmable data processing device, or other device to function in a specific manner, on a computer-readable medium, the instructions stored on the computer-readable medium can produce articles of manufacture containing instructions that perform functions / operations specified in one or more blocks of a flowchart and / or block diagram.

[0030]

[0036] Furthermore, process steps, method steps, or algorithms can be described in a specific order, but such processes, methods, and algorithms can also be configured to function in a different order. In other words, any order or sequence of steps that can be described does not necessarily indicate that these steps must be performed in that order. The steps of the processes described herein can be performed in any practical order. Moreover, several steps can be performed simultaneously.

[0031]

[0037] It will readily become clear that the various methods and algorithms described herein can be implemented, for example, by a properly programmed general-purpose computer and computer device. Typically, a processor (e.g., a microprocessor) executes a process defined by instructions by receiving and executing instructions from memory or similar devices. Furthermore, programs implementing such methods and algorithms can be stored and transmitted using a variety of known media.

[0032]

[0038] As used herein, the term “computer-readable medium” means any medium involved in providing data (e.g., instructions) that can be read by a computer, processor, or similar device. Such mediums can take many forms, including, but are not limited to, non-volatile, volatile, and transmitting media. Non-volatile media include, for example, optical or magnetic disks and other persistent memory. Volatile media include dynamic random-access memory (DRAM), which typically constitutes main memory. Transmitting media include coaxial cables, copper wires, and optical fibers, including wires that include a system bus coupled to a processor. Transmitting media include, or can transmit, sound waves, light waves, and electromagnetic radiation, such as those generated during radio frequency (RF) and infrared (IR) data communications. Examples of common forms of computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs, any other optical media, punch cards, paper tapes, any other physical media having a hole pattern, RAM, PROMs, EPROMs, FLASHEEPROMs, any other memory chips or cartridges, carrier waves as described below, or any other media that can be read by a computer.

[0033]

[0039] Various forms of computer-readable media can be involved in transporting a series of instructions to the processor. For example, a series of instructions can be (i) supplied to the processor from RAM, (ii) transported via a wireless transmission medium, and / or (iii) formatted according to numerous formats, standards, or protocols such as Bluetooth, TDMA, CDMA, 3G, 4G, and 5G.

[0034]

[0040] Embodiments of the present invention may include devices that perform the operations disclosed herein. These devices may be specifically configured for a desired purpose, or they may include general-purpose devices that are selectively operated or reconfigured by internally stored programs.

[0035]

[0041] Unless otherwise stated, and as may become apparent from the following description and claims, throughout this specification, any use of terms such as “process,” “calculate,” “calculate,” or “determine” should be understood to mean the operation and / or process of a computer, computer system, or similar electronic computer device that manipulates data represented as physical quantities, such as electron quantities in the registers and / or memory of a computer system, and / or transforms it into other data similarly represented as physical quantities in the memory, registers, or other such information storage, transmission, or display device of a computer system.

[0036]

[0042] Similarly, the term “processor” can mean any device or part of a device that processes electronic data from registers and / or memory and converts this electronic data into other electronic data that can be stored in registers and / or memory or transmitted to an external device, so as to cause a physical change or operation of an external device.

[0037]

[0043] The terms “agent,” “intelligent agent,” “artificial agent,” or “artificial intelligence agent” are intended to mean any artificial entity that chooses actions in response to observation. “Agent” can mean, without limitation, a robot, a simulated robot, a software agent or “bot,” an adaptive agent, an internet or webbot.

[0038]

[0044] The term "robot" can mean any system that is directly or indirectly controlled by a computer or computer system that issues actions or commands in response to sensing or observation. The term can also mean "bots" without limitation, such as conventional physical robots with physical sensors such as cameras, touch sensors and distance sensors, simulated robots existing in virtual simulations, or game bots existing as software in a network.

[0039]

[0045] The term “observation” means any information about the agent’s environment or the agent itself that the agent receives by any means. In some embodiments, this information may be perceptual information or signals received through perceptual devices such as cameras, touch sensors, distance sensors, temperature sensors, wavelength sensors, acoustic or voice sensors, position sensors, pressure or force sensors, velocity or acceleration or other motion sensors, or location sensors (e.g., GPS). In other embodiments, this information may also include edited, abstract or situational information created from a combination of a group of perceptual devices and stored information. In non-limiting examples, the agent may receive abstract information about the location or characteristics of the agent itself or other objects as an observation.

[0040]

[0046] The term "behavior" refers to any means of the agent that can control, influence, or affect the agent's environment, physical or simulated self, or internal functions, ultimately controlling or influencing the agent's future behavior, behavioral choices, or behavioral preferences. In many embodiments, these behaviors can directly control physical or simulated servos or actuators. In some embodiments, these behaviors can be expressions of preferences or sets of preferences ultimately intended to influence the agent's choices. In some embodiments, information about the agent's (single or plural) behaviors may include, but are not limited to, probability distributions across the agent's (single or plural) behaviors and / or outgoing information intended to influence the agent's final behavioral choices.

[0041]

[0047] The terms “state” or “state information” mean any set of information relating to the state of the environment or the agent, which may include, but is not limited to, information relating to the agent’s current and / or past observations.

[0042]

[0048] The term “policy” means any function or mapping from any complete or partial state information to any action information. A policy may be hardcoded or may be modified, adapted or trained using any appropriate learning or teaching method, including, but not limited to, any reinforcement learning method or control optimization method. A policy may be an explicit or implicit mapping, such as one that may result from optimizing a particular measure, value, or function. A policy may include, but not limited to, additional relevant information, features, or characteristics, such as start conditions (or probabilities) that reflect the conditions under which the policy may be started or continued, or end conditions (or probabilities) that reflect the conditions under which the policy may be terminated.

[0043]

[0049] Generally, embodiments of the present invention provide a method for training an agent by mixing general and highly specific race scenarios. Aspects of the method may include one or more of the following: (1) distributing one or more cars on a track to train an agent to perform exceptionally well in time trials; (2) running the agent in various race scenarios involving a variable number of opponents starting in different configurations around a track; (3) varying opponents using agents provided in a game, previous versions of agents trained according to aspects of the present invention, agents trained for different behaviors, or agents controlled by a controller that follow a specific racing line; (4) setting up specific short scenarios involving opponents in various race situations involving specific success criteria; and (5) having a dynamic curriculum based on how the agent performs in various evaluation scenarios. The training method may, for example, use quantile regression soft actor-critic model-free-off-policy deep reinforcement learning techniques.

[0044]

[0050] Embodiments of the present invention provide a technique called "mixed-scenario training," as shown in Figures 2 and 3, as a solution to the above-mentioned problems. Mixed-scenario training is designed for more complex areas such as autonomous driving, but requires the ability to configure an environment 200 to initiate a scenario (e.g., via a simulator). Instead of allowing the environment 200 to extract the agent's initial state, critical training conditions are encoded in a scenario configuration 202, which consists of the information shown in Figure 2.

[0045]

[0051] Typically, a scenario configuration 202 may include the following: (1) Activation conditions 204 that specify the start of the scenario. These conditions themselves can be randomized. For example, in an autonomous driving task, a "one-to-one" scenario may define the distribution of possible positions and speeds for the learning agent and one other car. (2) Opponent distribution 206 that specifies the form and probability of other agents in the scenario. For example, in an autonomous driving scenario, potential opponents may include line followers, pre-built AI controllers, and even pre-trained RL policies. (3) Replication count 208 that indicates the number of parallel agent scenarios to run in the environment. A scenario can be replicated many times in a particular environment. For example, in an environment that can accommodate 20 cars, 10 different one-to-one scenarios can be activated, as long as sufficient spacing is maintained. (4) Stop conditions 210 for when this scenario ends. These can be based on time, distance, or other conditions. (5) Scenario weighting 214 that determines the proportion of this scenario in the overall task sampling. For example, if the weighting of a "one-to-one" scenario is 0.1, it can indicate that 10% of all sampled tasks come from a one-to-one scenario. (6) Experience table mapping 212 can store data from a particular scenario in a specific partition called a "table" to ensure representation in the training batch and prevent data imbalance between longer or shorter scenarios within the same training set.

[0046]

[0052] Figure 3 illustrates an embodiment of a mixed-scenario training process for a deep reinforcement learning agent using parallel rollout workers. In this version, N workers are conceivable, representing N potential environments where scenarios can be executed in parallel. Data from these workers can be streamed back to an experience replay buffer on a centralized trainer, which can perform computations to update the training agent's policy and other models, as is standard in deep reinforcement learning architectures. Once a worker completes its rollout task, a new scenario can be extracted from the set of candidate scenarios, and parameters such as trigger conditions and opponent types can be further sampled. Multiple instances of a scenario (corresponding to the number of replications) can be executed in a single worker environment, and data can be streamed back to an experience table in a replay buffer specified by the scenario. Training continues periodically, and the policy can be updated with batches of data covering various tables in pre-defined proportions.

[0047]

[0053] In some embodiments, the set of candidate scenarios can be dynamically expanded based on events that occur during training or policy evaluation. For example, if two cars collide, the scenario immediately preceding the collision can be added to the set so that the agent can learn how to avoid the impending collision. This event detection feature is shown in the dashed box in Figure 3.

[0048]

[0054] In some embodiments, the curriculum can be used to adjust the proportion of scenarios based on some measure of progress. Examples of these progress measures include the number of training iterations or metrics that measure the performance of the current policy. For example, an autonomous driving curriculum could start by sampling only scenarios in which the learning agent exists alone, and then introduce "one-to-one" scenarios later when the agent demonstrates driving ability. This curriculum monitoring function is shown in the dashed box in Figure 3.

[0049]

[0055] The mixed-scenario training procedure addresses several problems with conventional methods, as described above. For example, scenario trigger conditions relieve the agent of the burden of reaching or planning to reach important scenarios. Also, by explicitly setting the trigger configuration and sampling a set of opponents, the need for "cooperative" opponents is reduced. Furthermore, by using scenario-to-table mapping and reweighting of data based on table ratios in each training batch, important data from shorter or harder-to-reach situations is not ignored during training. Finally, with the potentially rich encoding of trigger, opponent behavior, and stop conditions, mixed-scenario training can provide a vehicle for encoding prior knowledge by domain experts.

[0050]

[0056] Figure 3 shows one embodiment of a deep reinforcement learning architecture 300 that utilizes mixed-scenario training. The two main modules of the architecture are a trainer 302 and a set of rollout workers 304. The trainer 302 can refine the model 306 and policy 308 used to determine actions in the environment. Various representations of these models 306 and policy 308 are possible, including deep neural networks. Policy refinement can be performed by sampling a "batch" 310 of data from an experience replay buffer 312 populated with data from various scenarios previously run. This replay buffer 312 may include various tables 314 that partition the data, for example, separating data from solo driving experiences from data related to heavy traffic driving. Each time a batch is built for policy refinement, a pre-specified set of table weights 316 can be used to specify the proportion of data from each table to be used in the batch. Since the system links specific scenarios to tables (though not necessarily in a one-to-one (bijective) relationship), this table sampling ensures that a certain percentage of the data from each scenario is used for batch construction.

[0051]

[0057] The trainer 302 also functions as a "task manager" 318, determining which scenarios should be started for idle workers. Whenever a worker indicates that it has completed a previous scenario, the trainer 318 can perform a random selection from a set of potential scenarios based on the scenario weights. In other embodiments, this randomization can be replaced by a cyclic queue or other deterministic sampling process. In either case, the selected scenarios are instantiated by performing a random selection based on their startup parameters (such as car placement) and the set of opponents associated with this scenario. Again, weighted random selection or more complex history-based sampling can be performed to ensure a balance of the instantiated tasks.

[0052]

[0058] The instantiated scenario can be sent to the rollout worker 304 along with the current control policy. The rollout worker 304 can read this specification and instantiate the requested number of replicas of the scenario (from the scenario parameters) in the configurable environment 320. For example, this process could include a driving simulator setup with multiple spaced "one-to-one" scenarios. The rollout worker 304 can then execute the scenario using the transmitted policy and select the actions of all learning agents at each step. Data from the environment 320 (specifically, the state, actions, and rewards of all agents) is recorded (322) and sent to a table 314 in the experience replay buffer 312 specified by the scenario. This process continues until the scenario's termination condition is reached. This termination condition can alternatively be time-based (execute the scenario for X seconds), distance-based (drive the autonomous vehicle for X miles), or triggered by other events (execute for 1 hour or until it crashes into something).

[0053]

[0059] The architecture can be extended in various ways, including two modifications described below. First, a modification is possible that allows the set of potential scenarios to be dynamically increased based on events that occur during the execution of a rollout task. In the instantiation example, an event such as a collision between vehicles can be specified. If such an event occurs during rollout collection or policy evaluation, the state before the event (e.g., 10 seconds before the collision) is sent back to the trainer, and a new scenario can be dynamically constructed and added to the set of possible scenarios. This scenario helps to teach the agent to avoid undesirable events (in this case, collisions). Such "replay" scenarios can also be specified with a limit on the number of times they can be executed, so that they expire within the set of potential scenarios, potentially leaving room for other replay scenarios.

[0054]

[0060] When using a curriculum to modify a set of potential scenarios or their proportional weights, another potential variation arises. In this variation, you can specify a curriculum in which the set of potential scenarios or their weights are updated based on the curriculum phase. The curriculum phase is triggered by a learning event. A learning event may be as simple as the number of policy updates performed, or it may be a more complex criterion that includes metrics recorded by the rollout worker across various scenarios (such as the policy's driving aptitude). For example, an autonomous driving curriculum might start by only sampling scenarios in which the learning agent exists alone, and then later introduce "one-to-one" scenarios when the agent demonstrates driving ability.

[0055]

[0061] While some techniques in reinforcement learning literature share terminology with mixed-scenario training, there are significant gaps between them and the architectures, data structures, and procedures described above. For example, target-search algorithms can find specific states where the agent feels it needs more experience. However, these algorithms are insufficient in complex domains because they have no way to address the aforementioned problems. In particular, trigger-search algorithms may fail to reach the desired state because they cannot even control or sample the opponent's behavior. In contrast, mixed-scenario training according to aspects of the present invention directly configures the environment with appropriate trigger conditions and allows for setting or sampling the opponent's policies as needed.

[0056]

[0062] Curriculum learning in reinforcement learning may include modules that "generate" new environments, which can be considered a form of scenario generation. However, this generation is used to replace previous environments that the agent has sequentially acquired. In contrast, mixed-scenario training according to aspects of the present invention can address the problem of balancing many different scenarios at once in complex domains where multiple source tasks are required to maintain different skills.

[0057]

[0063] While many RL applications are trained in multiple environments to facilitate generalization, the goal of such agents is to enable them to run in many different small environments. In contrast, mixed-scenario training according to aspects of the present invention can focus on areas where training in specific scenarios builds skills that facilitate good behavior in full-fledged complex scenarios, even in very large, monolithic environments such as autonomous driving.

[0058]

[0064] The reinforcement learning option framework encodes many different smaller policies for different domains within a larger domain. However, this approach is fundamentally different from mixed-scenario training according to aspects of the present invention. Mixed-scenario training attempts to learn a single generalized policy for a larger domain using experience with target scenarios.

[0059]

[0065] In summary, mixed-scenario training of reinforcement learning agents can be used in complex domains. Here, scenarios focusing on specific skills can be sampled, divided into training data, and policies that excel at all desired behaviors can be created. Scenarios can be designed using launch configurations that can be randomized themselves, but that allow them to "spawn" in situations that help the learning agent learn a particular skill. Scenarios can be drawn from either a set type or distribution of opponents. These opponents are typically well-behaved within the scenario (e.g., vehicles following a line), allowing for learning and exploration of scenarios that would otherwise be impossible. Stopping conditions can be based on time, distance, or other criteria, so scenarios can be short and focused on specific skills, or long and potentially open-ended and focused on more general techniques. Mapping each scenario to a "table" partition in the experience playback buffer allows sufficient data from each scenario to be provided to the learning agent, regardless of differences in duration or sampling rate. Aspects of the present invention provide the ability to run replicas of numerous scenarios simultaneously in a large environment, for example, by collecting data from many "single" agents positioned sufficiently spaced apart in a driving environment. Aspects of the present invention further provide the ability to run many different scenarios in parallel among many rollout workers managed by a task manager in a training module. Aspects of the present invention provide the ability to add new scenarios encountered during a rollout activity to a set of potential scenarios via event triggers. Finally, curriculum logic can be used to modify the distribution of sampled scenarios over time or based on performance metrics collected from rollouts or other policy evaluations.

[0060]

[0066] A method according to an embodiment of the present invention can, for the first time, solve a simulated racing challenge using model-free deep reinforcement learning. An embodiment of the present invention provides a novel reinforcement learning algorithm that enhances the learning process with mixed-scenario training to encourage the agent to incorporate racing tactics into an integrated control policy. Furthermore, the method of the present invention constructs a reward function that enables the agent to adhere to the rules of sporting racing etiquette. A specific example described below demonstrates the capabilities of the artificial agent by achieving a 3 out of 3 victory against four of the world's best racing game drivers. This demonstrates that the agent training method according to an embodiment of the present invention can be successfully used to train championship-level automated racing car drivers, and such methods can be further applied to other complex dynamic systems and real-world applications.

[0061]

[0067] Game agents can be trained using deep reinforcement learning algorithms, including but not limited to quantile regression soft actor-critic (QR-SAC). This method learns a policy (actor) that selects actions based on the agent's observations, and a value function (critic) that estimates the future reward for each possible action. QR-SAC extends the soft actor-critic method by replacing the expected values ​​of future rewards with a representation of the probability distribution of those rewards and modifying it to handle N steps of return. Both the actor and the critic can be represented by a neural network containing four layers of 2048 nodes each. QR-SAC allows the neural network to be trained asynchronously; that is, QR-SAC samples data from an experience-replay buffer (ERB) while the actor practices driving using the latest policy, continuously filling the buffer with new experiences.

[0062]

[0068] Figure 4 shows the system. Trainer 10 distributes training scenarios to rollout workers 26. Each rollout worker 26 controls one game console running an instance of the game. Agent 28 within a rollout worker 26 controls up to 20 cars on track 30 by running one copy of the latest policy 22(π). Agent 28 sends an action(a) to the game for each car it controls. Asynchronously, the game computes the next frame and sends each new state(s) to agent 28. When the game reports that an action has been registered, agent 28 sends a tuple 24 of state, action, and reward.<s、a、r> The trainer reports this to the trainer 10, which stores it in the ERB 14. The trainer 10 samples the ERB 14 and updates the policy 20(π) and Q-function (18) network via the QR-SAC 16. Figure 4 shows a system including four rollout workers 26 currently assigned task 12 to perform 1:0, 1:1, 1:3, and 1:7 scenarios.

[0063]

[0069] The agent's core actions can be mapped to two continuous dimensions: changing speed (acceleration or braking) and steering (left or right). The effects of the actions can be enforced by the game to match the physical properties of the environment. For example, the agent can be prevented from braking harder than a human, but it can learn more precisely when to brake. The agent can interact with the game at 10Hz, which is within the range of interaction between professional players and video games.

[0064]

[0070] The agent can access position, speed, acceleration, and other relevant state information for itself and all opponents. The agent may also have a map of the track as a list of points defining the left and right edges and the centerline. The agent may not have other information available in the visual image, such as the size and shape of the curbs or the type of road surface outside the edges of the track.

[0065] Race Tactics

[0071] To learn race tactics using deep reinforcement learning (RL), the agent needs to represent observations of other cars in a way that can be interpreted by a neural network. The agent can maintain two lists of opponent state features: one for cars in front of the agent and another for cars behind it. Both lists can be sorted from nearest to furthest and can be restricted to a maximum range.

[0066]

[0072] In some embodiments, progress rewards alone may not be sufficient to motivate the agent to win the race. If the opponent is fast enough, the agent will learn to overtake the opponent and accumulate large rewards without taking the risk of a potentially catastrophic collision. Adding a specific reward for overtaking can help the agent learn to overtake other cars. Overtaking rewards can be used in proportion to the distance the agent has improved its position relative to each opponent in its local area. The rewards can be symmetrical, so if an opponent catches up to the agent, the agent receives a proportionally negative reward.

[0067]

[0073] Another complex issue that arises when learning tactics is that behavior can be heavily influenced by the opponent. If the opponent is overly aggressive or overly submissive, the agent may learn degenerate overtaking behavior or perform aggressive maneuvers. To avoid such instability, the agent can practice against curated policies from previous experiments selected to avoid unsportsmanlike behavior, rather than practicing against copies of itself. Mixed-scenario training supports the use of such populations through opponent distribution construction, allowing each scenario to potentially have its own unique mixture of opponent types and their distribution in these training scenarios.

[0068]

[0074] Finally, an agent may have few opportunities to learn a particular skill. This is called the exposure problem, where certain states of the world are inaccessible to the agent without the cooperation of an opponent. For example, to perform a "slingshot pass," a car needs to be in the opponent's slipstream on a long straight, a state that may occur naturally a few times throughout the race, or never at all. If the opponent always drives only on the right, the agent may learn to overtake only on the left, which is easily prevented by a human who chooses to drive on the left. Mixed scenario training addresses this problem. A few race situations that are likely to be critical on each track can be identified. Then, scenarios can be constructed that present the agent with noisy variations of those critical situations. In some scenarios, a simple PID controller can be used to make the opponent follow a specific trajectory, such as driving on the left, which is desirable for the agent to be prepared to encounter. This technique has allowed the agent to learn more robust skills.

[0069] Training scenario

[0075] To learn racing, agents need to acquire various skills, such as navigating crowded starts, tactically overtaking on open roads, and driving accurately on the track alone. To encourage basic racing skills, agents are placed in scenarios with 0, 1, 3, or 7 opponents launched nearby (1v0, 1v1, 1v2, 1v3, and 1v7, respectively). To create variety, track positioning, starting speed, spacing between cars, and opponent strategies can be randomized. Taking advantage of the fact that the game supports 20 cars at a time, the utilization of the game console can be maximized by launching multiple groups on the track. All basic scenarios were run for 150 seconds. Furthermore, scenarios with time or distance limitations can be utilized on specific course sections to expose agents to situations where they can learn skills highlighted by expert advisors. Exemplary skill scenarios include an 8-car grid start, one-on-one slipstream passing, navigating congested narrow chicanes in various possible traffic conditions, practicing overtaking opportunities in specific turn sequences offering multiple race lines and overtaking opportunities, practicing defensive maneuvers in the same corners while racing ahead, and combinations of aggressive and defensive driving when between other cars. Figures 5a–5f illustrate these scenarios on three race tracks. Playback tasks from the aforementioned extensions can be incorporated, particularly to learn how to avoid catastrophic consequences on high-speed tracks.

[0070]

[0076] Unlike curriculum training, where early skills are replaced by later skills or skills are built up hierarchically, the training scenarios used in aspects of the present invention are complementary and trained to a single control policy for racing in traffic situations. During training, the trainer assigned each rollout worker a new scenario by selecting from a set configured for that track based on manually adjusted ratios designed to provide sufficient skill coverage. However, even with this relative balance of execution, variations in random sampling from the buffer can cause skills to be forgotten between consecutive training epochs. Therefore, by implementing multi-table stratified sampling based on the aforementioned table sampling weights, the proportion of each scenario in each training mini-batch can be explicitly enforced, significantly stabilizing skill retention. Figure 7 shows an example of an experience retrieval buffer with data split into multiple tables.

[0071]

[0077] The importance of various aspects of mixed-scenario training is illustrated in Figures 6a and 6b, which show ablation studies of several components. The results, showing each agent's score against a common opponent in a 4v4 race, highlight the importance of controlling the scenario set ("no PID opponent"), using specific scenarios to train skills ("no slipstream"), and using multiple tables to organize skill-specific data in the experience replay buffer. The line graph shows the ability of different agents to overtake other agents in a "slipstream" evaluation scenario. The solid line represents the performance of one seed under each condition, and the dotted line represents the average of all seeds across all epochs. Baseline skills may fluctuate, but their impact is more generalized across other conditions.

[0072] experiment Gaming environment

[0078] Since its debut in 1997, the Gran Turismo® series has sold over 80 million copies. The latest release, GT Sport, is known for its precise vehicle dynamics simulation and racing realism, and has been FIA-approved and selected as the platform for the first Virtual Olympics. GT runs exclusively on PlayStation®4, and the dynamics simulation cycle is 60Hz. Up to 20 cars can participate in a race.

[0073]

[0079] The agent runs asynchronously on a separate computer and communicates with the game via HTTP over a wired Ethernet connection.

[0074]

[0080] The agent requests the latest observation at 60Hz, but makes decisions and takes action every 100ms (10Hz). Action frequencies from 5Hz to 60Hz were tested, but acting more frequently than 10Hz did not significantly improve performance. The agent must be robust against network latency, which is rare but does occur. Agent actions are treated similarly to human game controller input, but the GT API currently only supports a subset of action functions. For example, the API cannot allow the agent to control gear shifts, traction control systems, or brake balance, all of which can be adjusted by the player in-game.

[0075] Computing environment

[0081] Each experiment used a single trainer on a compute node containing either one NVIDIA V100 or half of an NVIDIA A100, coupled with approximately eight vCPUs and 55GB of memory. Some of these trainers ran in the data center, while others ran on AWS EC2 using p3.2xlarge instances.

[0076]

[0082] Each experiment also utilized several rollout workers. Each rollout worker consisted of a game console and a computing node (see Figure 4). In this setup, the game console ran the game, while the computing node managed the rollout by performing tasks such as calculating actions, sending actions to the game, sending experience streams to the trainer, and retrieving updated policies from the trainer (see Figure 4). The computing node used approximately two vCPUs and 3.3GB of memory. Typically, one worker primarily evaluated intermediate policies rather than generating new training data. Approximately 21 rollout workers were used to train the race policy over 14 days.

[0077] action

[0083] The GT API allows control of three independent sequential actions: throttle, brake, and steering. Since throttle and brake rarely operate simultaneously in practice, the agent was presented with throttle and brake control as a single sequential action dimension. Both throttle / brake and steering were scaled to the range [-1,1]. The policy network selects an action by outputting a squashed normal distribution containing mean and diagonal covariance matrices learned across these two dimensions.

[0078] Features

[0084] State features input to the neural network are either obtained directly from the game state or processed into a more convenient format and concatenated before being input to the model for training. Features include, but are not limited to, the following: The "Time Trial" feature set captured the car's 3D speed, 3D angular velocity, 3D g-force, load on each tire, tire slip angle, sine and cosine components, local course surface slope, car orientation relative to the course centerline, and a set of course points indicating upcoming track sections. The agent also received indicators when it hit a fixed barrier, when the game deemed it to be off-course, and when it received real values ​​from the game view of the car's latest steering angle, throttle intensity, and brake intensity.

[0079]

[0085] When training the agent to race against other cars, the list of features also included a car contact flag for detecting collisions and a slipstream scalar indicating whether a car was experiencing a slipstream effect from the car in front. To represent nearby cars, the agent used fixed forward and backward distance boundaries to determine which cars should be encoded. Cars were ordered by their relative distance from the agent and represented using relative position, velocity, and g-force.

[0080]

[0086] When training a neural network, the input can be standardized based on knowledge of the range of each feature scalar to keep the features described here within an appropriate numerical range. It can be assumed that the samples are drawn from a uniform distribution with a given range, and that the expected mean and standard deviation have been calculated. These were used to calculate the Z-score for each scalar before being input into the model.

[0081] Reward

[0087] The reward function used to train the agent was a linear combination of several individual components that were manually adjusted. These components included rewards for progress based on the agent's movement along the centerline of the course, penalties for going off course, hitting walls, or skidding, bonuses for overtaking other cars, and various penalties for crashing into other cars. Particularly severe collisions clearly caused by the agent, such as crashing into another car from behind, were given greater penalties. The weighting of these components differed for certain tracks and cars, and some were removed. This allowed the agent to utilize the track and car to their physical limits within the bounds of racing etiquette.

[0082] result

[0088] To evaluate the agent, it competed against top human drivers in two events. In the first event, three of the world's top drivers were asked to beat the agent's lap times on three tracks. The human drivers were allowed to see the agent's "ghost" while attempting to beat the agent's lap times. In these races, the agent won all three matches. Notably, there is evidence that at least one driver learned from or was inspired by the agent, and improved their own time trial performance after the event. This human driver recorded the fastest human time in Figure 7 (indicated by the circled number 1).

[0083]

[0089] The second event featured the world's best game drivers. This event consisted of one race per track. The top four game drivers formed teams and competed against four instances of agents. Points were awarded to each driver based on their final ranking (10 points for 1st place, 8 points for 2nd place, 6 points for 3rd place, and 5, 4, 3, 2, and 1 for the remaining positions), as shown in parentheses in Figures 8a and 8c. The longest and most difficult races awarded double points to the teams. Because the agents recorded the fastest times in the seeding laps before the races, the agents started from odd-numbered positions in all three races. Humans started from even-numbered positions and selected their position within their teams.

[0084]

[0090] The agent finished first in all three races and recorded the fastest lap times in all three races. The agent finished first in all three races, placing 1st, 2nd, 4th, and 5th in both of the first two races, and 1st, 2nd, 5th, and 6th in the most complex third race. Figures 8a-8c show the placement of the agent's and human cars in each race. From these results, the agent achieved an overall score of 104 to 52 against the humans. The results clearly demonstrate that the agent outperformed the world's best human drivers. The agent combined incredible speed with genuine racing skills to successfully overtake top drivers on both straightaways and curves.

[0085]

[0091] Those skilled in the art can make numerous changes and modifications without departing from the spirit and scope of the present invention. Therefore, the illustrated embodiments are merely examples and should not be interpreted as limiting the present invention as defined by the following claims. For example, while the elements of the claims are shown below in specific combinations, it should be clearly understood that the present invention includes other combinations of elements, fewer, more, or different from those disclosed.

[0086]

[0092] The words used herein to describe the present invention and its various embodiments should be understood to include not only their generally defined meanings but also, by their specific definitions herein, comprehensive structures, materials, or actions that represent a single kind.

[0087]

[0093] Therefore, in this specification, the definitions of words or elements in the following claims are not limited to combinations of elements explicitly stated in the text. In this sense, it is conceivable that one of any of the elements in the following claims may be replaced by two or more equivalent substitutes of elements, or a single element may be replaced by two or more elements of the claims. While elements are described above as functioning in specific combinations, and claims may initially be made in this manner, it should be clearly understood that, in some cases, one or more elements resulting from the claimed combinations may be removed from these combinations, and the claimed combinations may be directed towards lower combinations or variations of lower combinations.

[0088]

[0094] It is explicitly assumed that any minor changes from the subject matter, as seen by those skilled in the art, whether currently known or later devised, are equally included in the claims. Therefore, obvious substitutions, both currently known and hereafter known to those skilled in the art, are also defined as being within the scope of the specified elements.

[0089]

[0095] Therefore, the claims should be understood to include those specifically illustrated and described above, those that are conceptually equivalent, those that are clearly substituted, and those that incorporate the fundamental ideas of the present invention. [Explanation of Symbols]

[0090] 10 Trainers 12 tasks 14 ERB 16 QR-SAC 18 Q function 20. Strategy (π) 22 Latest Strategy (π) 24. Tuples of state, action, and reward<s、a、r> 26 Rollout Workers 28 Agents 100 Reinforcement Learning (RL) Agents 102 Environment 104 Observation 106 Action 108 Models 110 measures 200 Environment 202 Scenario Structure 204 Start condition 206 Opponent Distribution 208 replication count 210 Stop conditions 212 Experience Table Mapping 214 Scenario Weighting 300 Deep Reinforcement Learning Architectures 302 Trainer 304 Rollout Worker 306 Model 308 Measures 310 batch samples 312 Experience Replay Buffer 314 Table 316 Table Weights 318 Task Manager 320 Environment 322 data collection

Claims

1. A method for training a reinforcement learning agent in mixed scenarios using a trainer and a rollout worker, The steps include providing a rollout worker to an environment configured to have one or more predetermined pieces of information in a scenario that includes one or more predetermined pieces of information that constitute an environment in which the reinforcement learning agent takes action, The steps include: operating the rollout worker in the environment while focusing on one or more specific skills that the reinforcement learning agent acquires during training; A step of providing a reward when one or more of the above-mentioned specific skills are successfully achieved, The steps include: The rollout worker creating a strategy to optimize the reward; The step includes streaming data from the rollout worker to an experience response buffer, wherein the data in the experience response buffer is divided into one or more tables. The method further comprises the step of the trainer reweighting the data in the experience response buffer based on table ratios to specify the proportion of the data in the experience response buffer to be used in a batch sample.

2. The method according to claim 1, wherein one or more pieces of information in the scenario include one or more of the following: a start condition specifying the start of the scenario; an opponent distribution option for specifying other agents in the scenario; a replication number indicating the number of scenarios to be executed in parallel by multiple rollout workers; a stop condition for when the scenario ends; an experience table mapping for storing data from a particular scenario in the experience response buffer; and a scenario weighting on which the trainer relies when determining a scenario from a set of potential scenarios.

3. The method according to claim 1, further comprising the step of activating additional rollout workers in an additional environment configured to have predetermined information.

4. The method according to claim 3, wherein the predetermined information is randomly selected by the trainer randomly selecting a scenario from a set of potential scenarios when the trainer decides which scenario to start the additional rollout worker.

5. The method according to claim 3, wherein when the trainer determines which scenario to start for the additional rollout worker, the trainer extracts scenarios from a set of potential scenarios based on scenario weighting, so that the predetermined information is selected based on scenario weighting.

6. The method according to claim 3, wherein when the trainer determines which scenario to start for the additional rollout worker, the trainer extracts a scenario from a set of potential scenarios based on a cyclic queue sampling process, and the predetermined information is automatically generated from events that occurred from previous rollout workers in a previous environment.

7. The method according to claim 1, further comprising the step of configuring the environment to have information on the distribution of opponents that falls within the scope of racing etiquette within that environment.

8. The method according to claim 1, characterized in that the one or more predetermined pieces of information include a replication count that defines the number of parallel rollout workers to operate in the environment.

9. The method according to claim 1, characterized in that the one or more predetermined pieces of information include a stop condition.

10. The method according to claim 9, characterized in that the stop conditions are defined to be short in terms of time, distance, or other conditions, thereby creating an environment focused on achieving specific skills for the rollout worker.

11. The method according to claim 9, characterized in that by leaving time, distance, or other conditions open-ended in the stop conditions, an environment is created in which the rollout worker is focused on achieving general techniques.

12. A deep reinforcement learning system for training a reinforcement learning agent in mixed scenarios using a set of trainers and rollout workers, The set of the aforementioned rollout workers, The aforementioned trainer, Each rollout worker comprises one or more predetermined pieces of information that constitute the environment in which the reinforcement learning agent acts, Includes, The trainer improves the models and strategies used to determine the behavior of the rollout workers in the environment. The rollout worker operates within the environment based on a predetermined start condition which is extracted from one or more predetermined pieces of information and specifies the start of a scenario. Data is collected from the rollout worker operating within the environment including the predetermined startup conditions and stored in the trainer's experience playback buffer. A deep reinforcement learning system characterized by the following:

13. The deep reinforcement learning system according to claim 12, characterized in that the trainer performs policy improvement by sampling a batch of data from the experience replay buffer, which is populated with data from the operation of the set of rollout workers in the environment, which includes various activation conditions.

14. The deep reinforcement learning system according to claim 13, characterized in that the experience playback buffer includes a table for dividing the data.

15. The deep reinforcement learning system according to claim 14, characterized in that the batch of data includes data from multiple tables among the tables, and each table is provided with predetermined table weights.

16. The deep reinforcement learning system according to claim 12, characterized in that the trainer includes a task manager module for determining which information should be used by idle rollout workers from the set of rollout workers.

17. The deep reinforcement learning system according to claim 12, characterized in that the data in the experience playback buffer includes the state, behavior, and reward of each rollout worker.

18. A method for training an agent using deep reinforcement learning to interact in a racing video game, The steps include learning a policy for selecting an action based on observations by the agent and a value function that estimates the future reward for each possible action, A step of mapping the core actions of the agent to either a changing velocity dimension or a steering dimension, wherein both the changing velocity dimension and the steering dimension are continuous-value dimensions. A step of training the agent in an environment configured to have predetermined information, wherein the predetermined information includes a start condition specifying the start of a scenario, an opponent distribution option for specifying other agents in the scenario, a replication number indicating the number of scenarios to be run in parallel on multiple rollout workers, a stop condition for when the scenario ends, an experience table mapping for storing data from a particular scenario in the experience response buffer, and a scenario weighting on which the trainer relies when determining a scenario from a set of potential scenarios. A method characterized by including the following.

Images