Method for training artificial intelligence agents using curricula and skills - Patents.com

Abstract

The method for training an agent uses a mix of scenarios designed to teach specific skills useful in a larger domain, such as a mix of general racing and very specific tactical racing scenarios. Aspects of the method can include one or more of: (1) training an agent to be very good at time trials with one or more cars distributed around a track, (2) running the agent in various racing scenarios with a variable number of opponents starting in different configurations around the track, (3) varying the opponents using agents provided in the game, agents trained according to aspects of the invention, or agents controlled to follow specific driving lines, (4) setting up specific short scenarios involving opponents in various racing situations with specific success criteria, and (5) having a dynamic curriculum based on how the agent performs in various evaluation scenarios.

Description

[Technical field]

[0001]

[0001] Embodiments of the present invention relate generally to training artificial intelligence agents. More specifically, the present invention relates to methods for training game agents by both general gameplay and placing agents in specific scenarios. More specifically, aspects of the present invention can use mixed-scenario training for reinforcement learning in configurable environments, such as racing game agents. [Background technology]

[0002]

[0002] The following background information may present examples of certain aspects of the prior art (e.g., but not limited to, techniques, facts or conventional wisdom) that are expected to serve to further inform the reader of additional aspects of the prior art, but should not be construed as limiting the invention or any embodiment thereof to anything mentioned or implied therein or inferred therefrom.

[0003]

[0003] Referring to Figure 1, a reinforcement learning (RL) agent 100 is a form of artificial intelligence that is trained (also called "learning") through interactions with an environment 102. At each time step of the agent's training, the agent is provided with observations 104 of its current state. The agent 100 then executes actions 106, which transition the agent to a new state and generate a reward term. Various existing RL algorithms and models 108 provide routines for ultimately finding an optimal policy 110 (a mapping from states to actions) that maximizes some function of the reward term (such as an expected sum).

[0004]

[0004] In simple domains, RL agents are expected to be able to experience all possible states based on their actions. However, in complex problems such as learning to drive or racing an autonomous vehicle in traffic, a learning agent that needs data from useful learning scenarios, such as driving between two cars, faces many challenges. For example, if the environment is large enough, random or target search by the learning agent will have too many domains to explore and will likely miss important scenarios.

[0005]

[0005] Furthermore, since a reinforcement learning agent almost always has a finite horizon (or a finite effective horizon) in its planning, even if it identifies a scenario it wants to visit, it may not be practical to execute a plan to get there. If an agent needs data between two cars, but the cars are far apart, the agent may not have a reliable way to reach the cars.

[0006]

[0006] Also, when other agents share the environment, they may not perform actions that lead to the experience the learning agent needs. For example, if the autonomous driving system is a racing simulator, the two cars do not slow down to allow the learning agent to get between them. Even if the learning agent manages to experience a scenario in which it drives between the two cars, the amount of experience needed to reach that scenario may be much greater than the experience in the scenario, and the impact on training is minimal.

[0007]

[0007] Furthermore, complex environments often come with strong prior knowledge from human experience about what scenarios are useful for learning. For example, a driving instructor may be able to prescribe many scenarios that are useful for learning an agent. However, basic reinforcement learning formulations do not have an encoding of these known learning scenarios available. Summary of the Invention [Problem to be solved by the invention]

[0008]

[0008] These problems need to be solved in order to effectively learn in such complex domains without simply running the agent for a huge number of steps.

[0009] An example of a complex environment where an agent needs multiple skills to succeed is a simulated auto race. To control a race car effectively, the driver needs a detailed understanding of the vehicle's dynamics and the characteristics of the track on which he or she races (idiosyncrasies). The driver then builds on this foundation with the tactical skills necessary to overtake or defend against opponents, executing precise maneuvers at high speed with little room for error. [Means for solving the problem]

[0010]

[0009] An embodiment of the present invention provides a method for training a reinforcement learning agent in mixed-scenario training, the method comprising the steps of providing a rollout worker to an environment having one or more predetermined scenario characteristics, operating the rollout worker within the environment while focusing on one or more particular skills, providing a reward for successfully achieving the one or more particular skills, and creating a strategy for the rollout worker to optimize the reward.

[0011]

[0010] An embodiment of the present invention further provides a deep reinforcement learning architecture using mixed scenario training, comprising a set of rollout workers, a trainer, and a set of scenario characteristics, wherein the trainer refines models and strategies used to determine behavior of the rollout workers within an environment, the rollout workers operating within the environment based on predefined launch conditions derived from the scenario characteristics, and data from the rollout workers operating in the environment including the predefined launch conditions is collected and stored in an experience replay buffer of the trainer.

[0012]

[0011] An embodiment of the present invention also provides a method of training an agent using deep reinforcement learning to interact in a racing video game, the method comprising the steps of: learning a strategy for selecting actions based on observations by the agent and based on a value function that estimates future rewards for each possible action; mapping core actions of the agent to one of a changing velocity dimension and a steering dimension, both of which are continuous-valued dimensions; and training the agent in an environment including predetermined scenario characteristics, the predetermined scenario characteristics including launch conditions, opponent distribution options, replication number, stopping conditions, experience table mapping, and scenario weighting.

[0013]

[0012] In some embodiments, the method further includes providing the agent with position, velocity, and acceleration state information for itself and each opponent, and providing the agent with a map of the track as a list of points defining the left and right edges and a centerline.

[0014]

[0013] In some embodiments, the method further includes training the agent in a racing scenario involving a variable number of opponents starting in different configurations around the track, and training the agent against opponents selected from artificial agents provided in the game, other agents trained with various reward functions, and agents controlled by a controller following a particular driving line.

[0015]

[0014] These and other features, aspects and advantages of the present invention will become better understood with reference to the following drawings, description and claims.

[0016]

[0015] Some embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference symbols can indicate similar elements and in which: [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 shows a schematic diagram of a reinforcement learning agent interacting with its environment. [Diagram 2] FIG. 1 illustrates a diagram of scenario components and their consumers in a single environment for mixed-scenario training, in accordance with an exemplary embodiment of the present invention. [Diagram 3] FIG. 1 illustrates a schematic diagram of a complete mixed-scenario training in accordance with an exemplary embodiment of the present invention. [Figure 4] FIG. 1 illustrates an example system configuration in which a trainer distributes training scenarios to rollout workers, with each rollout worker controlling one game console running an instance of a racing game. [Figure 5a] FIG. 1 illustrates a scenario configuration on three different racing tracks used to train an agent using mixed-scenario training. [Figure 5b] FIG. 1 illustrates a scenario configuration on three different racing tracks used to train an agent using mixed-scenario training. [Figure 5c] FIG. 1 illustrates a scenario configuration on three different racing tracks used to train an agent using mixed-scenario training. [Figure 5d] FIG. 1 illustrates a scenario configuration on three different racing tracks used to train an agent using mixed-scenario training. [Figure 5e] FIG. 1 illustrates a scenario configuration on three different racing tracks used to train an agent using mixed-scenario training. [Figure 5f] FIG. 1 illustrates a scenario configuration on three different racing tracks used to train an agent using mixed-scenario training. [Figure 6a] Figure 1 illustrates an ablation study of various aspects of the scenarios used to train racing agents. When scenarios are altered or removed, the agent's ability to learn and retain important skills is diminished. [Figure 6b] Figure 1 illustrates an ablation study of various aspects of the scenarios used to train racing agents. When scenarios are altered or removed, the agent's ability to learn and retain important skills is diminished. [Figure 7] FIG. 13 illustrates the contents of an experience playback buffer that uses multiple tables to organize data from different scenarios. [Figure 8a] Figure 1 shows the results of a race in a simulated racing game in which an agent trained in a mixed scenario competed against the four best drivers. [Figure 8b] Figure 1 shows the results of a race in a simulated racing game in which an agent trained in a mixed scenario competed against the four best drivers. [Figure 8c] Figure 1 shows the results of a race in a simulated racing game in which an agent trained in a mixed scenario competed against the four best drivers. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018]

[0024] Unless otherwise indicated, the figures in the drawings are not necessarily drawn to scale.

[0019]

[0025] A better understanding of the invention and its various embodiments can be obtained by reference to the following detailed description which describes the illustrated embodiments, with the express understanding that the illustrated embodiments are presented by way of example and are not intended to limit the invention as ultimately defined by the claims.

[0020]

[0026] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting of the present invention. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. As used herein, the singular forms "a," "an," and "the" are intended to include the plural in addition to the singular, unless the context clearly indicates otherwise. Furthermore, it should be understood that the terms "comprises" and / or "comprising," as used herein, indicate the presence of stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0021]

[0027] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by those skilled in the art to which the present invention belongs. Furthermore, terms as defined in commonly used dictionaries should be interpreted to have a meaning in accordance with these meanings in the context of the relevant art and this disclosure, and should not be interpreted in an ideal or overly formal sense unless expressly defined herein.

[0022]

[0028] In the description of the present invention, it will be understood that multiple techniques and steps are disclosed. Each of these has its own advantages and can also be used with one or more, or in some cases all, of the other techniques disclosed. Thus, for the sake of clarity, the description will not unnecessarily repeat all possible combinations of the individual steps. However, the specification and claims should be read with the understanding that such combinations are fully within the scope of the present invention and the claims.

[0023]

[0029] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without these specific details.

[0024]

[0030] The present disclosure should be considered as an exemplification of the invention and is not intended to limit the invention to the specific embodiments shown in the following figures or description.

[0025]

[0031] Devices or system modules that are in at least general communication with each other need not be in continuous communication with each other, unless otherwise specified, and devices or system modules that are in at least general communication with each other may communicate directly or indirectly through one or more intermediary devices.

[0026]

[0032] A description of an embodiment including multiple components in communication with each other does not imply that all such components are required, but rather a variety of optional components are described to illustrate the wide range of possible embodiments of the present invention.

[0027]

[0033] A "computer" or "computing device" may mean one or more devices and / or one or more systems that can accept structured input, process the structured input according to prescribed rules, and produce the results of the processing as output. Examples of computers or computing devices may include computers, fixed and / or portable computers, computers with a single processor, multiple processors, or multi-core processors capable of operating in parallel and / or non-parallel, supercomputers, mainframes, superminicomputers, minicomputers, workstations, microcomputers, servers, clients, interactive televisions, web appliances, communications devices with Internet access, hybrid combinations of computers and interactive televisions, portable computers, tablet personal computers (PCs), personal digital assistants (PDAs), mobile phones, application specific hardware that emulates a computer and / or software, such as digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific instruction set processors (ASIPs), chips, multiple chips, systems on a chip or chipsets, data acquisition devices, optical computers, quantum computers, biocomputers, and generally devices that accept data, process the data in accordance with one or more stored software programs, and generate results, and which may typically include input devices, output devices, storage devices, arithmetic units, logic units, and control units.

[0028]

[0034] "Software" or "application" can mean prescribed rules for operating a computer. Examples of software or applications can include code segments, graphical and / or textual instructions, applets, precompiled code, interpreted code, compiled code, and computer programs in one or more computer readable languages.

[0029]

[0035] These computer program instructions, which can direct a computer, other programmable data processing apparatus, or other device to function in a particular manner, can also be stored on a computer readable medium such that the instructions stored on the computer readable medium produce an article of manufacture that includes instructions that perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0030]

[0036] Additionally, although process steps, method steps, algorithms, or the like may be described in a certain order, such processes, methods, and algorithms may be configured to work in other orders. In other words, any order or sequence of steps that may be described does not necessarily indicate that the steps must be performed in this order. Steps of processes described herein may be performed in any order that is practical. Additionally, some steps may be performed simultaneously.

[0031]

[0037] It will be readily apparent that the various methods and algorithms described herein can be implemented, for example, by appropriately programmed general purpose computers and computing devices. Typically, a processor (e.g., a microprocessor) receives instructions from a memory or similar device and executes those instructions to perform the process defined by those instructions. Furthermore, programs implementing such methods and algorithms can be stored and transmitted using a variety of known media.

[0032]

[0038] The term "computer-readable medium" as used herein means any medium that participates in providing data (e.g., instructions) that may be read by a computer, a processor, or a similar device. Such media may take many forms, including, but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical or magnetic disks and other persistent memory. Volatile media include dynamic random access memory (DRAM), which typically constitutes the main memory. Transmission media include coaxial cables, copper wire, and fiber optics, including the wires that comprise a system bus coupled to the processor. Transmission media may include or convey acoustic waves, light waves, and electromagnetic radiation, such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer readable media include, for example, a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, a DVD, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM, EPROM, FLASHEEPROM, any other memory chip or cartridge, a carrier wave as described below, or any other medium from which a computer can read.

[0033]

[0039] Various forms of computer readable media may be involved in carrying the set of instructions to the processor, for example, the set of instructions may (i) be provided to the processor from a RAM, (ii) be carried over a wireless transmission medium, and / or (iii) be formatted according to a number of formats, standards, or protocols, such as Bluetooth, TDMA, CDMA, 3G, 4G, 5G, etc.

[0034]

[0040] Embodiments of the present invention may include apparatuses for performing the operations disclosed herein, which may be specially constructed for the desired purposes, or which may comprise general-purpose devices selectively activated or reconfigured by programs stored in the devices.

[0035]

[0041] Unless otherwise stated, and as will become apparent from the following description and claims, throughout this specification, descriptions using terms such as "processing," "calculating," "computing," or "determining" should be understood to mean operations and / or processes of a computer, computer system, or similar electronic computing device that manipulates data represented as physical quantities, such as electronic quantities in a computer system's registers and / or memory, and / or transforms data into other data that is similarly represented as physical quantities in a computer system's memory, registers, or other such information storage, transmission, or display device.

[0036]

[0042] Similarly, the term "processor" may refer to any device or part of a device that processes electronic data from registers and / or memory and converts this electronic data into other electronic data that can be stored in registers and / or memory or transmitted to an external device so as to cause a physical change or action in an external device.

[0037]

[0043] The term "agent" or "intelligent agent" or "artificial agent" or "artificial intelligence agent" is intended to mean any artificial entity that selects an action in response to an observation. "Agent" can mean, without limitation, a robot, a simulated robot, a software agent or "bot", an adaptive agent, an internet or web bot.

[0038]

[0044] The term "robot" can mean any system controlled directly or indirectly by a computer or computer system that issues actions or commands in response to sensing or observing. The term can mean, without limitation, a traditional physical robot having physical sensors such as cameras, touch sensors, and distance sensors, or a simulated robot that exists in a virtual simulation, or a "bot" such as a gamebot that exists as software in a network.

[0039]

[0045] The term "observation" refers to any information received by an agent by any means about the agent's environment or the agent itself. In some embodiments, this information can be sensory information or signals received through sensory devices, such as, but not limited to, cameras, touch sensors, distance sensors, temperature sensors, wavelength sensors, acoustic or voice sensors, position sensors, pressure or force sensors, velocity or acceleration or other motion sensors, location sensors (e.g., GPS), etc. In other embodiments, this information can include, but is not limited to, compiled, abstract, or situational information created from a combination of a set of sensory devices and stored information. In a non-limiting example, an agent can receive abstract information about the location or characteristics of itself or other objects as an observation.

[0040]

[0046] The term "behavior" refers to any means by which an agent can control, affect, or influence the agent's environment, the agent's physical or simulated self, or the agent's internal functions, ultimately controlling or influencing the agent's future actions, action selections, or behavioral preferences. In many embodiments, these actions can directly control physical or simulated servos or actuators. In some embodiments, these actions can be expressions of preferences or sets of preferences ultimately intended to influence the agent's choices. In some embodiments, information regarding the agent's action(s) can include, but is not limited to, probability distributions over the agent's action(s) and / or outgoing information intended to influence the agent's ultimate action selection.

[0041]

[0047] The terms "state" or "state information" refer to any collection of information regarding the state of the environment or an agent, which may include, but is not limited to, information regarding the current and / or past observations of the agent.

[0042]

[0048] The term "policy" refers to any function or mapping from any complete or partial state information to any action information. A policy can be hard coded or modified, adapted or trained using any suitable learning or teaching method, including but not limited to any reinforcement learning or control optimization method. A policy can be an explicit or implicit mapping, such as but not limited to, one that may result from optimizing a particular measure, value or function. A policy can include associated further information, features or characteristics, such as but not limited to, starting conditions (or probabilities) reflecting under what conditions the policy can be started or continued, and ending conditions (or probabilities) reflecting under what conditions the policy can be terminated.

[0043]

[0049] In general, embodiments of the invention provide a method for training an agent in a mix of general and very specific racing scenarios. Aspects of the method can include one or more of the following: (1) distributing one or more cars on a track and training the agent to be very good at time trials; (2) running the agent in various racing scenarios with a variable number of opponents starting in different configurations around the track; (3) varying the opponents using agents provided in the game, previous versions of the agent trained according to aspects of the invention, agents trained for different behaviors, or agents controlled by a controller that follows a specific driving line; (4) setting up specific short scenarios with opponents in various racing situations with specific success criteria; and (5) having a dynamic curriculum based on how the agent performs in various evaluation scenarios. The training method can use, for example, quantile regression soft actor-critic model-free off-policy deep reinforcement learning techniques.

[0044]

[0050] As a solution to the above problems, an embodiment of the present invention provides a technique called "mixed scenario training" as shown in Figures 2 and 3. Mixed scenario training is designed for more complex domains such as autonomous driving, but requires the ability to configure the environment 200 (e.g., via a simulator) to launch scenarios. Instead of allowing the environment 200 to extract the initial state of the agent, important training situations are encoded into a scenario configuration 202 that is composed of information as shown in Figure 2.

[0045]

[0051] Typically, a scenario configuration 202 can include: (1) launch conditions 204 that specify the start of a scenario. These conditions themselves can be randomized. For example, in an autonomous driving task, a "one-vs-one" scenario can specify the distribution of possible positions and velocities of the learning agent and one other car. (2) opponent distributions 206 that specify the form and probability of the other agents in the scenario. For example, in an autonomous driving scenario, potential opponents can include line followers, pre-built AI controllers, or even pre-trained RL policies. (3) replication number 208 that indicates the number of parallel agent scenarios to run in the environment. A scenario can be replicated many times in a particular environment. For example, in an environment that can accommodate 20 cars, 10 different one-vs-one scenarios can be launched as long as sufficient spacing is maintained. (4) stopping conditions 210 when this scenario ends. They can be based on time, distance, or conditions. (5) scenario weighting 214 that determines the proportion of this scenario in the overall task sampling. For example, a weighting of 0.1 for the "1 vs. 1" scenario may indicate that 10% of all sampled tasks are from the 1 vs. 1 scenario. (6) Experience table mapping 212 may store data from specific scenarios in specific partitions called "tables" to ensure representation in training batches and prevent data imbalance for longer or shorter scenarios in the same training set.

[0046]

[0052] FIG. 3 illustrates an embodiment of a mixed-scenario training process for a deep reinforcement learning agent using parallel rollout workers. In this version, N workers are considered, representing N potential environments in which scenarios can be run in parallel. Data from these workers can be streamed back to an experience replay buffer on a centralized trainer that can perform calculations to update the learning agent's policy and other models, as is standard in deep reinforcement learning architectures. As workers complete rollout tasks, they can extract new scenarios from the set of candidate scenarios and further sample parameters such as launch conditions and opponent type. Multiple instances of a scenario (corresponding to the number of replications) can be run in a single worker environment, and data can be streamed back to experience tables in the replay buffer specified by the scenario. Training continues periodically, and the policy can be updated with batches of data that cover a pre-set percentage of the various tables.

[0047]

[0053] In some embodiments, the set of candidate scenarios can be dynamically expanded based on events that occur during training or policy evaluation. For example, if two cars collide, a scenario immediately before the collision can be added to the set to allow the agent to learn how to avoid the impending collision. This event detection functionality is indicated by the dashed box in Figure 3.

[0048]

[0054] In some embodiments, the curriculum can be used to adjust the proportion of scenarios based on some progress measure. Examples of these progress measures can include the number of training iterations, or a metric that measures the performance of the current strategy. For example, an autonomous driving curriculum can start out by only sampling scenarios where the learning agent is alone, and later introduce "one-on-one" scenarios when the agent demonstrates driving ability. This curriculum monitoring function is illustrated by the dashed box in FIG. 3.

[0049]

[0055] The mixed-scenario training procedure addresses several issues of previous methods, as discussed above. For example, scenario launch conditions relieve the agent from the burden of reaching or planning to reach important scenarios. Also, explicitly setting launch configurations and sampling the population of opponents mitigates the need for "cooperative" opponents. Furthermore, the use of scenario-to-table mapping and data reweighting based on table proportions in each training batch ensures that important data from shorter or hard-to-reach situations is not ignored in training. Finally, with a potentially rich encoding of launches, opponent behaviors, and stopping conditions, mixed-scenario training can provide a vehicle for the encoding of prior knowledge by domain experts.

[0050]

[0056] FIG. 3 illustrates an embodiment of a deep reinforcement learning architecture 300 that utilizes mixed scenario training. The two main modules of the architecture are a trainer 302 and a set of rollout workers 304. The trainer 302 can refine models 306 and policies 308 used to determine behavior in the environment. Various representations of these models 306 and policies 308 are possible, including deep neural networks. Policy refinement can be performed by sampling "batches" 310 of data from an experience replay buffer 312 that has been populated with data from various previously executed scenarios. This replay buffer 312 can include various tables 314 that partition the data, for example, separating data from solo driving experiences from data related to driving in heavy traffic. Each time a batch is built for policy refinement, a set of pre-specified table weights 316 can be used to specify the proportion of data from each table to be used in the batch. Because the system links specific scenarios to tables (although not necessarily in a bijective relationship), this table sampling ensures that a consistent percentage of data from each scenario is used in the batch construction.

[0051]

[0057] The trainer 302 also acts as a "task manager" 318, determining which scenarios should be started with idle workers. Each time a worker indicates completion of a previous scenario, the trainer 318 can perform a random selection from the set of potential scenarios based on the weights of the scenarios. In other embodiments, this randomization can be replaced by a circular queue or other deterministic sampling process. In either case, the selected scenario is instantiated by performing a random selection based on its launch parameters (such as car placement) and the population of opponents associated with this scenario. Again, weighted random selection or more complex history-based sampling can be done to ensure balance of the instantiated tasks.

[0052]

[0058] The instantiated scenario can be sent to the rollout worker 304 along with the current control policy. The rollout worker 304 can read this specification and instantiate the requested number of replicas of the scenario (from the scenario parameters) in the configurable environment 320. For example, this process can include setting up a driving simulator with multiple spaced “one-to-one” scenarios. The rollout worker 304 can then run the scenario using the communicated policy and select behaviors for all learning agents at each step. Data from the environment 320 (specifically, the states, actions, and rewards of all agents) is recorded 322 and sent to a table 314 in the experience playback buffer 312 specified by the scenario. This process continues until an end condition for the scenario is reached. This end condition can alternatively be time-based (run the scenario for X seconds), distance-based (drive the autonomous vehicle for X miles), or conditional on other events (run for an hour or until it hits something).

[0053]

[0059] Various extensions of the architecture are possible, including two variants described below. First, a variant is possible where the set of potential scenarios can be dynamically increased based on events that occur during the execution of the rollout task. In an example of instantiation, an event such as a collision between vehicles can be specified. When such an event occurs during rollout collection or policy evaluation, the state before the event occurs (e.g., 10 seconds before the collision) can be sent back to the trainer and a new scenario can be dynamically constructed and added to the set of potential scenarios. This scenario helps teach the agent to avoid undesirable events (in this case, a collision). Such "playback" scenarios can also be specified with a limit on the number of times they are executed, which allows them to expire in the set of potential scenarios, potentially making room for other playback scenarios.

[0054]

[0060] Another potential variant occurs when using a curriculum to change the set of potential scenarios or their proportional weights. In this variant, a curriculum can be specified where the set of potential scenarios or their weights are updated based on the curriculum phase. A curriculum phase is triggered by a learning event. A learning event can be as simple as the number of policy updates performed, or it can be a more complex criterion that includes metrics (such as the driving suitability of the policy) recorded by the rollout worker in various scenarios. For example, an autonomous driving curriculum can start out by only sampling scenarios where the learning agent is alone, and later introduce "one-to-one" scenarios when the agent has demonstrated driving ability.

[0055]

[0061] Although some techniques in the reinforcement learning literature share terminology with mixed-scenario training, there are significant gaps between the architectures, data structures, and procedures described above. For example, target search algorithms can seek out specific states where an agent feels it needs more experience. However, these algorithms fall short in complex domains because they lack ways to address the problems mentioned above. In particular, trigger search algorithms may not be able to reach the states they want to explore because they cannot control or even sample the behavior of the opponent. In contrast, mixed-scenario training according to aspects of the present invention can directly configure the environment with appropriate launch conditions and set or sample the opponent's policy as needed.

[0056]

[0062] Curriculum learning in reinforcement learning may include modules that "generate" new environments, which can be considered a form of scenario generation. However, this generation is used to replace previous environments that the agent has mastered in sequence. In contrast, mixed-scenario training according to aspects of the present invention can address the problem of balancing many different scenarios at once in complex domains where multiple source tasks are required to maintain different skills.

[0057]

[0063] While there are many RL applications where an agent is trained in multiple environments to promote generalization, the goal of such an agent is to be able to run in many different small environments. In contrast, mixed-scenario training according to aspects of the present invention can focus on very large, monolithic environments, such as autonomous driving, but where training in specific scenarios builds skills that promote good behavior in the full complex scenario.

[0058]

[0064] Reinforcement learning frameworks encode many different small policies for different domains within a large domain. However, this approach is fundamentally different from mixed-scenario training according to an aspect of the present invention. Mixed-scenario training can attempt to learn a single generalized policy for a large domain using experience with a target scenario.

[0059]

[0065] In summary, mixed-scenario training of reinforcement learning agents can be used in complex domains, where scenarios each focused on a specific skill can be sampled and split into training data to create a good policy for all desired behaviors. Scenarios can be designed with launch configurations that can themselves be randomized, but allow the learning agent to "spawn" in situations that help it learn a specific skill. Scenarios can be drawn from either a set opponent type or distribution. These opponents are typically well-behaved in the scenario (e.g., a vehicle following a line), allowing for the learning and exploration of scenarios that would not be possible otherwise. Stopping conditions can be based on time, distance, or other criteria, so that scenarios can be short and focused on a specific skill, or long and potentially open-ended, focusing on a more general technique. Mapping each scenario to a "table" partition in the experience replay buffer allows the learning agent to be provided with sufficient data from each scenario, regardless of differences in duration or sampling rate. Aspects of the invention provide the ability to run replicas of many scenarios at once in a large environment, e.g., collecting data from many "single" agents spaced well apart in an operational environment. Aspects of the invention further provide the ability to run many different scenarios in parallel across many rollout workers managed by a task manager in a training module. Aspects of the invention provide for adding new scenarios encountered during rollout activities to the set of potential scenarios via event triggers. Finally, curriculum logic can be used to change the distribution of sampled scenarios over time or based on performance metrics collected from rollout or other policy evaluations.

[0060]

[0066] Methods according to embodiments of the present invention can solve simulated racing problems for the first time using model-free deep reinforcement learning. Aspects of the present invention provide a new reinforcement learning algorithm and enhance the learning process with mixed-scenario training to encourage the agent to incorporate racing tactics into the integrated control strategy. Furthermore, the method of the present invention constructed a reward function that allows the agent to adhere to the racing etiquette rules of the sport. A specific example described below demonstrates the capabilities of the artificial agent by winning three out of three races against four of the world's best racing game drivers. This shows that agent training methods according to aspects of the present invention can be successfully used to train championship-level autonomous race car drivers, and such methods can be further applied to other complex dynamical systems and real-world applications.

[0061]

[0067] Game agents can be trained using deep reinforcement learning algorithms, including but not limited to quantile regression soft actor-critic (QR-SAC). This technique learns a policy (actor) that selects an action based on the agent's observations, and a value function (critic) that estimates the future reward of each possible action. QR-SAC extends the soft actor-critic technique by replacing the expected value of future rewards with a representation of the probability distribution of those rewards, and modifying it to handle N-step returns. Both the actor and the critic can be represented by neural networks containing four layers of 2048 nodes each. QR-SAC can train the neural network asynchronously; that is, it samples data from an experience replay buffer (ERB) while the actor practices driving using the latest policy, continuously filling the buffer with new experiences.

[0062]

[0068] The system is shown in Figure 4. A trainer 10 distributes training scenarios to rollout workers 26. Each rollout worker 26 controls one game console running an instance of the game. An agent 28 in a rollout worker 26 runs one copy of the latest policy 22 (π) to control up to 20 cars on a track 30. The agent 28 sends an action (a) to the game for each car it controls. Asynchronously, the game calculates the next frame and sends each new state (s) to the agent 28. When the game reports that an action has been registered, the agent 28 writes a state, action, reward tuple 24<s、a、r> to the trainer 10, which stores it in the ERB 14. The trainer 10 samples the ERB 14 to update the policy 20 (π) and the Q-function (18) network via the QR-SAC 16. Figure 4 shows a system with four rollout workers 26 currently assigned tasks 12 to perform 1 vs. 0, 1 vs. 1, 1 vs. 3, and 1 vs. 7 scenarios.

[0063]

[0069] The agent's core actions can be mapped to two continuous-valued dimensions: changing speed (accelerate or brake) and steering (left or right). The effects of the actions can be enforced by the game to match the physics of the environment. For example, the agent can be prevented from braking harder than a human would, but the agent can learn more accurately when to brake. The agent can interact with the game at 10 Hz, which is within the range of how professional players interact with video games.

[0064]

[0070] An agent has access to position, velocity, acceleration, and other relevant state information for itself and all opponents. An agent may also have a map of the track as a list of points that define the left and right edges and the centerline. An agent may not have other information available in the visual image, such as the size and shape of the curbs and the type of surface outside the edges of the track.

[0065] Race Tactics

[0071] To learn racing tactics with deep reinforcement learning (RL), an agent needs to represent its observations of other cars in a way that can be interpreted by a neural network. An agent can keep two lists of opponent state features: one for the cars ahead of the agent, and one for the cars behind the agent. Both lists can be ordered from nearest to farthest and bounded by a maximum range.

[0066]

[0072] In some embodiments, progress rewards alone may not be enough to motivate an agent to win a race. If the opponents are fast enough, the agent will learn to overtake them and accumulate a large reward without risking a potentially catastrophic collision. Adding a reward specifically for overtaking can help the agent learn to overtake other cars. Overtaking rewards can be used proportional to the distance the agent has improved its position relative to each opponent in the local area. The rewards can be symmetric, so that if an opponent catches up with the agent, the agent receives a proportionally negative reward.

[0067]

[0073] Another complication that arises when learning tactics is that behavior can be highly dependent on the opponent. If the opponent is overly aggressive or overly submissive, the agent may learn degenerate overtaking behavior or engage in aggressive maneuvers. To avoid such instability, rather than practicing against a copy of itself, the agent can practice against curated policies from previous experiments that have been selected to not exhibit unsportsmanlike behavior. Mixed-scenario training supports the use of such ensembles through opponent distribution construction, allowing each scenario to potentially have its own unique mix of opponent types and distributions of their use in these training scenarios.

[0068]

[0074] Finally, opportunities for an agent to learn certain skills may be rare. This is called the exposure problem: certain states of the world are inaccessible to the agent without the cooperation of the opponent. For example, executing a "slingshot pass" requires a car to be in the opponent's slipstream on a long straightaway, but this state may occur naturally several times throughout the race, or never at all. If the opponent always drives only on the right, the agent may learn to overtake only from the left, easily thwarted by a human choosing to drive on the left. Mixed-scenario training addresses this problem. A small number of racing situations that may be crucial on each track can be identified. Scenarios can then be constructed that present the agent with noisy variants of those crucial situations. In some scenarios, a simple PID controller can be used to make the opponent follow a particular trajectory, such as driving on the left, which is something the agent is hopefully prepared to encounter. This technique has led the agent to learn more robust skills.

[0069] Training Scenarios

[0075] Learning to race requires mastering a variety of skills, such as navigating a crowded start, overtaking tactically on an open road, and driving solo and accurately around a track. To encourage basic racing skills, agents were placed in scenarios where 0, 1, 3, or 7 nearby opponents were launched (1 vs. 0, 1 vs. 1, 1 vs. 2, 1 vs. 3, and 1 vs. 7, respectively). To create variety, the track location, starting speed, spacing between cars, and opponents' strategies can be randomized. Taking advantage of the fact that the game supports 20 cars at a time, the utilization of the game console can be maximized by launching multiple groups on the track. All basic scenarios were run for 150 seconds. Additionally, scenarios with time or distance restrictions on certain course sections can be utilized to expose agents to situations that allow them to learn the skills highlighted by the expert advisor. Exemplary skill scenarios include an eight-car grid start, one-to-one slipstream passing, driving through a congested narrow chicane in each of the possible traffic conditions, practicing overtaking opportunities in a particular turn sequence that offers multiple race lines and overtaking opportunities, practicing defensive maneuvers in the same corner while racing in front, and combining aggressive and defensive driving while between other cars. Figures 5a-f show illustrations of these scenarios on three race tracks. Replay tasks from the previously mentioned extensions can be incorporated to learn how to avoid catastrophic outcomes, especially on high-speed tracks.

[0070]

[0076] Unlike curriculum training, where earlier skills are replaced by later skills or skills are built hierarchically, the training scenarios used in the present invention are complementary and trained on a single control strategy for racing in traffic conditions. During training, the trainer assigned each rollout worker a new scenario by selecting from a set constructed for that track based on a manually tuned ratio designed to provide sufficient skill coverage. However, even with this relative execution balance, skills may be forgotten between successive training epochs due to fluctuations in random sampling from the buffer. Therefore, multi-table stratified sampling based on the table sampling weights mentioned above can be implemented to explicitly enforce the proportion of each scenario in each training mini-batch and greatly stabilize skill retention. Figure 7 shows an example of an experience replay buffer with data split into multiple tables.

[0071]

[0077] The importance of various aspects of mixed-scenario training is illustrated in Figures 6a and 6b, which show an ablation study of several components. Results showing the scores of each agent against a common opponent in a 4v4 race demonstrate the importance of controlling the population of scenarios ("no PID opponent"), using specific scenarios to train skills ("no slipstream"), and using multiple tables to organize skill-specific data in the experience replay buffer. Line graphs show the ability of various agents to overtake another agent in the "slipstream" evaluation scenario. The solid lines represent the performance of one seed in each condition, while the dotted lines represent the average of all seeds over all epochs. Although baseline skills can vary, the effects are more prevalent in other conditions.

[0072] experiment Game Environment

[0078] Since its debut in 1997, the Gran Turismo® series has sold more than 80 million copies. The latest release, GT Sport, is known for its precise vehicle dynamics simulation and racing realism, and has earned the honor of being FIA-sanctioned and chosen as the platform for the first Virtual Olympics. GT runs exclusively on PlayStation®4, with a dynamics simulation cycle of 60Hz. Up to 20 cars can participate in a race.

[0073]

[0079] The agent runs asynchronously on a separate computer and communicates with the game via HTTP over wired Ethernet.

[0074]

[0080] Agents request up-to-date observations at 60Hz, but make decisions and issue actions every 100ms (10Hz). Action frequencies from 5Hz to 60Hz were tested, but acting more frequently than 10Hz did not significantly improve performance. Agents must be robust to rare but real-world network delays. Agent actions are treated similarly to human game controller input, although the GT API currently supports only a subset of behavioral features. For example, the API does not allow agents to control gear shifting, traction control systems, or brake balance, all of which can be adjusted by the player in-game.

[0075] Computing Environment

[0081] Each experiment used a single trainer on a compute node containing either one NVIDIA V100 or half an NVIDIA A100 coupled with approximately 8 vCPUs and 55GB of memory. Some of these trainers were run in our datacenter and some were run on AWS EC2 using p3.2xlarge instances.

[0076]

[0082] Each experiment also used several rollout workers. Each rollout worker consisted of a game console and a computing node (see Figure 4). In this setup, the game console ran the game, and the computing node managed the rollout by performing tasks such as computing actions, sending actions to the game, sending an experience stream to the trainer, and getting updated policies from the trainer (see Figure 4). The computing node used approximately 2 vCPUs and 3.3 GB of memory. Typically, one worker was primarily evaluating intermediate policies rather than generating new training data. 21 rollout workers were used to train the race policies for approximately 14 days.

[0077] action

[0083] The GT API allows the control of three independent continuous actions: throttle, brakes, and steering. Since in practice throttle and brakes are rarely actuated simultaneously, throttle and brake control were presented to the agent as a single continuous action dimension. Both throttle / brake and steering were scaled to the range [-1,1]. The policy network selects an action by outputting a squashed normal distribution containing the mean and diagonal covariance matrix learned over these two dimensions.

[0078] Features

[0084] The state features input to the neural network are either taken directly from the game state or processed and concatenated into a more convenient form before being input to the model for training. Features include, but are not limited to, the following: The "Time Trial" feature set captured the car's 3D speed, 3D angular velocity, 3D g-forces, the load on each tire, tire slip angle, sine and cosine components, local course surface slope, the car's orientation relative to the course centerline, and a set of course points indicating upcoming sections of the track. The agent also received indicators if it had contacted a fixed barrier, if it had been deemed off-course by the game, and received the game's view of the real values ​​of the car's most recent steering angle, throttle strength, and braking strength.

[0079]

[0085] When training the agent to race other cars, the list of features also included a car contact flag to detect collisions, and a slipstream scalar to indicate if the car was experiencing a slipstream effect from the car in front. To represent nearby cars, the agent used fixed front and rear distance bounds to determine which cars to encode. Cars were ordered by their relative distance to the agent and represented using their relative position, velocity, and g-forces.

[0080]

[0086] When training a neural network, to keep the features described here in their proper numerical ranges, the inputs can be standardized based on knowledge of the range of each feature scalar. Samples can be assumed to be drawn from a uniform distribution given the range, and the expected mean and standard deviation calculated. These were used to calculate a Z-score for each scalar before it was input to the model.

[0081] Rewards

[0087] The reward function used to train the agent was a manually tuned linear combination of several separate components. These included rewards for progress as the agent moved along the centerline of the track, penalties for going off course, hitting a wall, or experiencing wheel slippage, bonuses for overtaking other cars, and various penalties for hitting other cars, with larger penalties given for particularly egregious collisions that the agent apparently caused, such as hitting another car from behind. For certain tracks and cars, these components were weighted differently and some were removed. This allowed the agent to utilize the tracks and cars to their physical limits, within the bounds of racing etiquette.

[0082] result

[0088] To evaluate the agent, the agent was pitted against top human drivers in two events. In the first event, three of the world's top drivers were asked to beat the agent's lap times on three tracks. The human drivers were allowed to see a "ghost" of the agent as they attempted to beat the agent's lap times. In these races, the agent won all three matches. Notably, there is evidence that at least one driver learned from or was inspired by the agent, improving their own time trial performance since the event. This human driver records the fastest human time in Figure 7 (denoted by the circled number 1).

[0083]

[0089] The second event featured the world's best game drivers. This event consisted of one race at each track. The top four game drivers teamed up to compete against four instances of the agent. Points were awarded to teams based on final placement (10 points for 1st place, 8 points for 2nd place, 6 points for 3rd place, and 5, 4, 3, 2, 1 for remaining placements) as shown in brackets in Figure 8a-8c for each driver, with double points awarded for the longest and most difficult race. Agents started in odd-numbered positions in all three races because they had recorded the fastest times on the pre-race seeding laps. Humans started in even-numbered positions and chose their placement within their teams.

[0084]

[0090] The agent placed first in all three races and recorded the fastest lap times in all three races. The agent placed first in all three races, finishing 1st, 2nd, 4th, and 5th in both the first two races, and 1st, 2nd, 5th, and 6th in the third race, which was the most complex. Figures 8a-8c show the placement of the agent and human cars in each race. From these results, the agent received an overall score of 104 vs. 52 against the human. The results clearly show that the agent outperformed the world's best human drivers. The agent combined incredible speed with real racing skills to successfully outrun the top drivers on the straights and in the curves.

[0085]

[0091] Those skilled in the art may make many changes and modifications without departing from the spirit and scope of the present invention. It is therefore to be understood that the illustrated embodiments are presented by way of example only and should not be construed as limiting the invention as defined by the following claims. For example, although the elements of the claims are presented below in specific combinations, it should be expressly understood that the invention includes other combinations of elements that are less, more, or different than those disclosed.

[0086]

[0092] The words used herein to describe the invention and its various embodiments are to be understood to include not only their generally defined meaning, but also the generic structures, materials or acts for which they represent a single species by specific definitions herein.

[0087]

[0093] Therefore, the definition of the words or elements in the following claims is set forth herein to include more than just the combination of elements expressly stated in the text. In this sense, it is therefore considered that the equivalent substitute of two or more elements may be used in place of any one of the elements in the following claims, or a single element may be used in place of two or more elements in the claims. Although elements may be described above as functioning in a particular combination and may be initially claimed in this manner, it is to be clearly understood that in some cases, one or more elements resulting from the claimed combination may be deleted from these combinations, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0088]

[0094] Slight variations from the subject matter as viewed by one of ordinary skill in the art, whether now known or later devised, are expressly contemplated to be equally within the scope of the claims. Thus, obvious substitutions now known to those of ordinary skill in the art and those that will become known in the future are also defined as being within the scope of the specified elements.

[0089]

[0095] The claims are therefore to be understood to include what is specifically illustrated and described above, what is conceptually equivalent, what is obviously substituted, and what incorporates the basic idea of ​​the invention. [Explanation of symbols]

[0090] 10 Trainer 12 Tasks 14 ERB 16QR-SAC 18 Q-Function 20 Strategy (π) 22 Latest Strategies (π) 24 State, action, reward tuples<s、a、r> 26 Rollout Workers 28 Agent 100 Reinforcement Learning (RL) Agents 102 Environment 104 Observations 106 Action 108 Models 110 Measures 200 Environment 202 Scenario Composition 204 Start condition 206 Opponent Distribution 208 Replications 210 Stop conditions 212 Experience Table Mapping 214 Scenario Weighting 300 Deep Reinforcement Learning Architectures 302 Trainer 304 Rollout Worker 306 Model 308 Measures 310 Batch Samples 312 Experience Replay Buffer 314 Table 316 Table Weight 318 Task Manager 320 Environment 322 Data Collection

Claims

1. 1. A method for training a reinforcement learning agent with mixed scenario training, comprising: providing a rollout worker to an environment having one or more predefined scenario characteristics; operating the rollout worker within the environment while focusing on one or more specific skills; providing a reward upon successful achievement of the one or more specified skills; creating a strategy for the rollout worker to optimize the reward; The method according to claim 1, further comprising:

2. 2. The method of claim 1, further comprising streaming data from the rollout worker to an experience response buffer, wherein the data in the experience response buffer is partitioned into one or more tables.

3. 3. The method of claim 2 further comprising the step of reweighting data in the experiential response buffer based on a table ratio so that data from hard-to-reach situations is not ignored.

4. 2. The method of claim 1, wherein the scenario characteristics include one or more of: a start condition, an opponent distribution option, a replication count, a stop condition, an experience table mapping, and a scenario weighting.

5. 10. The method of claim 1, further comprising launching additional rollout workers in additional environments having a predetermined set of scenario characteristics.

6. The method of claim 5 , wherein the set of predetermined scenario characteristics is selected randomly.

7. The method of claim 5 , wherein the set of predetermined scenario characteristics is selected based on scenario weightings.

8. 6. The method of claim 5, wherein the set of predefined scenario characteristics is created automatically from events that occurred from a previous rollout worker in a previous environment.

9. 2. The method of claim 1, further comprising providing a scenario characteristic of a well-behaved opponent distribution within the environment.

10. The method of claim 1 , wherein the scenario characteristics include a replication count that defines a number of parallel rollout workers to operate in the environment.

11. The method of claim 1 , wherein the scenario characteristics include a stopping condition.

12. The method of claim 11 , wherein the stopping conditions are determined to create an environment focused on achieving a particular skill.

13. 12. The method of claim 11, wherein the stopping condition is open-ended and the rollout workers focus on achieving general technology.

14. 1. A deep reinforcement learning architecture using mixed scenario training, comprising: A set of rollout workers, Trainer and a set of scenario characteristics; Including, The trainer refines models and strategies used to determine the actions of rollout workers within the environment; the rollout worker operates within the environment based on predefined activation conditions derived from the scenario characteristics; data from the rollout workers operating in the environment including the predetermined launch conditions is collected and stored in a trainer experience playback buffer; A deep reinforcement learning architecture characterized by

15. 15. The deep reinforcement learning architecture of claim 14, wherein the trainer performs policy refinement by sampling batches of data from the experience playback buffer populated with data from operation of the set of rollout workers in the environment including a variety of launch conditions.

16. 16. The deep reinforcement learning architecture of claim 15, wherein the experience playback buffer includes a table for partitioning the data.

17. 17. The deep reinforcement learning architecture of claim 16, wherein the batch of data includes data from multiple ones of the tables, each table being provided with a predefined table weight.

18. 15. The deep reinforcement learning architecture of claim 14, wherein the trainer includes a task manager module for determining which scenario characteristics should be used by idle rollout workers in the set of rollout workers.

19. 15. The deep reinforcement learning architecture of claim 14, wherein the data in the experience playback buffer includes states, actions, and rewards for each rollout worker.

20. 1. A method of training an agent using deep reinforcement learning to interact in a racing video game, comprising: learning a policy for selecting actions based on observations by the agent and based on a value function that estimates future rewards for each possible action; mapping the agent's core actions to one of a changing velocity dimension and a steering dimension, both of which are continuous-valued dimensions; training the agent in an environment including predefined scenario characteristics, the predefined scenario characteristics including launch conditions, opponent distribution options, replication counts, stopping conditions, experience table mappings, and scenario weightings; The method according to claim 1, further comprising:

Images