Model-free reinforcement learning with regularized nash dynamics

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training policy neural networks for controlling agents. In particular, the policy neural network may be trained by model-free reinforcement learning with regularized Nash dynamics.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 356,009, filed on June 27, 2022. The disclosure of the prior application is considered a part of and incorporated by reference into the disclosure of the present application. Technical Field

[0003] This specification relates to processing data using machine learning models. Background Art

[0004] A machine learning model receives input and generates an output, such as a predicted output, based on the received input. Some machine learning models are parametric models and generate output based on the received input and the values ​​of the model's parameters.

[0005] Some machine learning models are deep models that employ multiple layers of models to generate outputs for received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, each of which applies a nonlinear transformation to the received input to generate an output. Summary of the invention

[0006] This specification generally describes a system implemented as a computer program on one or more computers at one or more locations that uses a policy neural network to control an agent that interacts with an environment to perform a task in the environment.

[0007] This specification also describes techniques for training policy neural networks so that the policy neural networks can be used to effectively control intelligent agents.

[0008] More specifically, this specification describes techniques for training policy neural networks over multiple training phases using a technique called "regularized Nash dynamics."

[0009] A computer-implemented method of training a policy neural network for controlling an agent that interacts with an environment, and optionally one or more other agents, to perform a task is described.

[0010] The method initializes a regularized control strategy, e.g., arbitrarily, for controlling an agent. The method involves multiple training phases. The regularized control strategy can define actions to be performed in response to observations deterministically or stochastically. The regularized control strategy can generally be any suitable control strategy for controlling an agent that assigns non-zero scores to at least some actions in a set and uses the scores to select actions to be performed by the agent. In at least some training phases from the multiple training phases, and as will be described below, the regularized control strategy depends on weights determined for a policy neural network in a previous training phase.

[0011] Each of the plurality of training phases can involve generating training data for a task, the training data comprising, for each of a plurality of steps during a trajectory (i.e., a sequence of actions) for performing the task, a tuple comprising data identifying (i) an observation, (ii) an action performed in response to the observation in the tuple, and (iii) a reward for the task. For actions performed by an agent, the action performed can be determined by processing the observation using a policy neural network to generate a policy output for selecting an action. In embodiments, other actions can be performed by one or more other agents.

[0012] The method involves generating modified training data for the modified task by generating a transformed reward for each tuple from the rewards in the tuple using a regularized control policy as of a (current) training phase. In some embodiments, a previous regularized control policy (from a previous training phase) can also be used. Generating the transformed reward can involve using the regularized control policy to calculate a modifier to apply to the reward to obtain the transformed reward. The modifier can be determined, for example, based on (the logarithm of) the ratio of the probability assigned to the action by the policy neural network to the probability assigned to the action by the regularized policy. The degree of modification of the reward by the modifier can be determined, for example, by multiplying by a regularization parameter (having a value greater than zero) of the modifier. For multiple agents, the modifier of the agent controlled by the policy neural network may have an opposite sign to that of the other agents.

[0013] The method generates a modified tuple that includes data identifying (i) an observation that identifies the tuple, (ii) an action identified in the tuple, and (iii) a transformed reward for the tuple, and the policy neural network is trained on the modified training data. For example, the policy neural network can be trained using a reinforcement learning technique based on a transformed reward, such as by using a reinforcement learning objective that depends on the transformed reward, using an optimizer such as Adam (a gradient descent optimization algorithm) to update parameters such as weights of the policy neural network. The method can perform multiple (interleaved) iterations of generating training data and / or modifying training data and training the policy neural network in a training phase before updating the regularized control policy, such as by controlling the agent to act in the environment after training the policy neural network to obtain more training data.

[0014] After training the policy neural network, the regularized control policy is updated to a policy that uses the policy neural network to select actions. That is, the control policy defined by the trained policy neural network (at the end of the training phase) defines the regularized control policy. For example, a regularized policy neural network corresponding to the policy neural network can be maintained, and its parameters (e.g., weights) can be updated to be the same as the updated weights of the policy neural network as of the completion of the training phase. In an embodiment, the regularized control policy is used to regularize the reward, and is also generated by the regularization of the reward.

[0015] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0016] By training the neural network as described in this specification, the system is able to train the neural network to perform well even on tasks with very large state spaces, very large action spaces, or both.

[0017] Furthermore, the system is able to train neural networks to perform well even in multi-agent scenarios with imperfect information (in addition to large state and / or action spaces), where observations convey both "private" and "public" information, and the "private" information of other agents is not available to the acting agent.

[0018] In particular, the system trains a policy neural network by repeatedly performing a training phase in which the system directly modifies the underlying learning dynamics by training the neural network on a modified task defined by a modified reward computed using a regularized policy.

[0019] By modifying the dynamics, the system modifies the task so that the policy neural network can effectively learn a high-quality control policy even in the presence of imperfect information, very large state spaces, very large action spaces, or some combination of the three. For example, the modified rewards can guide the reinforcement learning technique used to train the policy neural network towards a Nash equilibrium. By updating the regularized policy using the trained policy neural network and repeating the training process with the rewards modified using the updated regularized policy, the system can iteratively approach a Nash equilibrium, in particular without explicitly modeling the private states of other agents. Broadly speaking, the described techniques allow training to convergence even in astronomical state or action spaces characterized by imperfect information.

[0020] In particular, the system is able to train a policy neural network to learn a high-quality control policy without requiring any demonstration data, which may be difficult to obtain in sufficient quantities for complex real-world tasks or other tasks with large state spaces, or without requiring an environmental model, which may be difficult to generate or learn for complex environments.

[0021] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 An example action selection system is shown.

[0023] Figure 2 is a flow chart of an example process for training a policy neural network.

[0024] Figure 3 is a flow chart of an example process for performing the training phase.

[0025] Figure 4 is a flow chart of an example process for generating a transform reward.

[0026] Figure 5 An example architecture of a policy neural network is shown.

[0027] Fig. 6A An example training architecture for a training system is shown.

[0028] Figure 6B Another example training architecture for a training system is shown.

[0029] Like reference numbers and names in the various drawings indicate like elements. DETAILED DESCRIPTION

[0030] Figure 1 An example action selection system 100 is shown. Action selection system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below are implemented.

[0031] The action selection system 100 controls agents 104 interacting in the environment 106 to perform tasks by selecting actions 108 to be performed by the agents 104 and then causing the agents 104 to perform the actions 108 .

[0032] The agent can receive a corresponding reward 112 at each time step, where the reward is specified by a scalar value and represents, for example, the agent's progress towards completing an assigned task.

[0033] As a specific example, reward 112 can be a sparse binary reward that is zero unless the task is successfully completed due to the action being performed, e.g., the reward is non-zero, e.g., equal to one, only when the task is successfully completed due to the action being performed.

[0034] As another specific example, reward 112 can be a dense reward that measures the agent's progress toward completing the task as of a single observation received during an episode of attempting to perform the task, e.g., such that a non-zero reward can be received and is received frequently before successfully completing the task.

[0035] In particular, at each time step, system 100 receives observations 110 that characterize the state of the environment at that time step.

[0036] System 100 then processes inputs including observation 110 using policy neural network 122 .

[0037] Policy neural network 122 is configured to receive input including observations 110 characterizing a state of environment 106 and process the input to generate a policy output that defines an action 104 to be performed in response to the observations.

[0038] For example, the policy neural network 122 can generate a policy output that assigns a corresponding score to each action in a set of actions that can be performed by the agent 104. In this example, the system can, for example, select the action with the highest score or map the score to a probability distribution and then sample an action from the probability distribution.

[0039] As another example, to select action 104, policy neural network 122 can generate a policy output that includes parameters of a score distribution over a set of actions, e.g., parameters of a Gaussian distribution over a continuous space of possible actions. In this example, the system can, for example, select an action that represents the mean of the distribution or sample an action from the distribution.

[0040] As another example, the action set can include multiple subsets, for example, the actions in each subset are valid actions during the asynchronous portion required to perform the task.

[0041] In this example, the policy neural network 122 can include multiple “policy heads,” each corresponding to a different subset of actions and generating a corresponding output that either (i) assigns a corresponding score to each action in the corresponding subset or (ii) includes parameters defining a distribution of scores over the actions in the corresponding subset.

[0042] In this example, to select an action 104, the system can select one of the subsets, and then use the output of the head corresponding to the subset to select an action, such as described above. For example, each subset can be valid at a different set of time steps during an episode of performing a task, or can be valid at a different set of environmental states. The system can then select a subset and a corresponding head based on the current time step or the current environmental state at the current time step.

[0043] The policy neural network 122 can have any suitable architecture that allows the neural network 122 to map data in an input format to a policy output. For example, the policy neural network 122 can include one or more of a convolutional layer, a fully connected layer, a recurrent layer, an attention layer, etc.

[0044] References below Figure 5 An example architecture is described in more detail.

[0045] In some implementations, performing a task may require the agent 104 to interact with one, two, or more other agents 107 in the environment 106 that are also performing the same task or performing one or more other tasks.

[0046] The operation of other agents 107 may be controlled by computer-implemented methods, for example, similar to the methods for controlling agent 104, or some or all may be partially or entirely controlled by humans. That is, other agents may also be controlled using instances of policy neural network 122 (or different learning strategies) or by fixed computer-implemented strategies.

[0047] Typically, the tasks performed by agent 104 and other agents may have the same characteristics, for example, they may all be routing tasks, navigation tasks, assembly tasks, etc., or they may all be scheduling tasks, or they may all be manufacturing tasks.

[0048] The other tasks may be the same as or different from the task of agent 104 .

[0049] The tasks performed by an agent may contribute to achieving a common goal, or may be adversarial, i.e., tasks performed by other agents may hinder agent 104 from performing tasks.

[0050] Typically, the ability of agent 104 to perform a task in the environment is influenced by one or more other agents performing one or more other tasks in the environment.

[0051] In some embodiments where agent 104 interacts with other agents, different agents can perform actions at different time steps during a given round of performing a task. That is, for a proper subset (i.e., a subset less than all) of the time steps in a task round, the actions performed in the environment are the actions performed by agent 104 in response to observations 110 at the time steps. For each time step in the trajectory that is not in the proper subset, one of the one or more other agents performs the action.

[0052] As a specific example, when agent 104 interacts with a single other agent, agent 104 and the other agent can perform actions at alternating time steps.

[0053] Alternatively, different agents can perform actions at the same time step, ie two or more agents can perform corresponding actions at the same time step.

[0054] Additionally, in some embodiments where agent 104 interacts with other agents, the observations received by each action at a given time step can include different information.

[0055] That is, observation 110 can include (i) public information about the state of environment 106 available to agent 104 and one or more other agents 107, and (ii) private information about the state of environment 106 available only to an agent performing an action in response to observation 110.

[0056] For example, common information about the state of environment 106 includes information that can be measured by sensors of all agents in the environment at a given time step. Common information can also include static information about the environment that is provided to all agents, such as a map of the environment or other data that characterizes static properties of the environment.

[0057] Private information can include, for example, information that can only be measured by a single agent's sensors, such as proprioceptive sensor readings or sensor readings from sensors inside the agent. Private information can also include information that characterizes a task being performed by the agent, such as conditional inputs that instruct the agent which task to perform, such as a natural language sentence that characterizes the agent's goal or a target image that characterizes a target state of the environment.

[0058] In some embodiments, observation 110 includes a public information tensor including public information at a given time step and a private information tensor including private information at a given time step. Separating private information from public information in this manner enables neural network 122 to learn how to effectively combine these two different types of information.

[0059] Furthermore, in some embodiments, the public information includes information that is public about the agent 104 and can be observed by other agents 107. That is, the public information in the observation 110 can provide the agent 104 with knowledge about the public information available to other agents 107 about the state of the agent 104 at a given time step.

[0060] As will be described below, in some embodiments, observation 110 includes historical information from previous time steps, or policy neural network 122 is otherwise conditioned on information from previous time steps, for example, by including one or more recurrent neural network layers. In these embodiments, for any given agent acting at time step and controlled by the policy neural network, the system conditions policy neural network 122 only on the public historical information and the given agent's private historical information, and not on private historical information from any other agent.

[0061] System 100 also includes a training system 190 that trains policy neural network 122 so that policy neural network 122 can be used to effectively control agent 104.

[0062] In an embodiment, training system 190 trains policy neural network 122 via model-free reinforcement learning over multiple training phases using a technique called “regularized Nash dynamics”.

[0063] The training performed by training system 190 is referred to as “model-free” reinforcement learning because training system 190 uses reinforcement learning techniques that do not use an explicit model of the environment, i.e., the effects of actions performed by agent 104 and other agents 107 on the state of the environment are modeled.

[0064] By training the policy neural network 122 as described herein, the policy neural network 122 is able to effectively learn high-quality control policies even in the presence of imperfect information (i.e., when private information of other agents is not observable to the acting agent), very large state spaces, very large action spaces, or some combination of the three.

[0065] As will be described below, during training, one or more other agents 107 may also be controlled by the policy neural network 122 in order to generate training data for training.

[0066] After training, other agents 107 can be controlled by a different strategy, such as by a human-operated controller or a controller controlled by different software, or can continue to be controlled by an instance of the policy neural network 122.

[0067] References below Figures 2 to 4 Describe the training in more detail.

[0068] In some embodiments, the environment is a real-world environment, the agent is a mechanical agent that interacts with the real-world environment, such as a robot or a vehicle, such as an autonomous or semi-autonomous land, air, or sea vehicle that operates in or navigates through the environment, and the actions are actions taken by the mechanical agent to perform a task in the real-world environment. For example, the agent can be a robot that interacts with the environment to complete a specific task, such as to locate an object of interest in the environment or to move an object of interest to a specified location in the environment or to navigate to a specified destination in the environment.

[0069] As another example, the agents controlled by the policy neural network can be vehicles, e.g., with or without a human driver, and can be controlled using the policy neural network to control traffic signals that the driver must obey, such as traffic lights and lane control signals. If present, one or more other agents can also be mechanical agents, such as robots or human-driven or autonomous or semi-autonomous vehicles, e.g., forming a platoon of vehicles. Thus, some embodiments of the system can be used to control traffic, such as road traffic, or warehouse robots, or aircraft such as drones, to facilitate the smooth flow of traffic.

[0070] In these embodiments, observations may include, for example, one or more of the following: images, object position data, and sensor data to capture observations when the agent interacts with the environment, such as sensor data from images, distance or position sensors, or from actuators. For example, in the case of a robot, observations may include data characterizing the current state of the robot, such as one or more of the following: joint positions, joint velocities, joint forces, torques, or accelerations, such as gravity-compensated torque feedback, and the global or relative pose of an item held by the robot. In the case of a robot or other mechanical agent or vehicle, observations may similarly include one or more of the position, linear or angular velocity, force, torque, or acceleration, and global or relative pose of one or more components of the agent. Observations may be defined as 1, 2, or 3 dimensions, and may be absolute and / or relative observations. Observations may also include, for example, sensed electronic signals, such as motor currents or temperature signals; and / or image or video data, such as data from a camera or lidar sensor, such as data from a sensor of the agent or data from a sensor located separately from the agent in the environment. In certain traffic control applications, observations may include map data depicting static objects in the environment, such as roads, trees, buildings, etc., and / or real-time road condition data depicting non-static objects in the environment, such as pedestrians, other vehicles, the status of road signals - such as the color of a traffic light - etc.

[0071] In these embodiments, the action may be a control signal for controlling a robot or other mechanical agent, such as a torque of a robot joint or a higher-level control command, or a control signal for controlling an autonomous or semi-autonomous land, air, or sea vehicle, such as a torque of a control surface or other control element (e.g., a steering control element of a vehicle) or a higher-level control command. The control signal can include, for example, position, velocity, or force / torque / acceleration data of one or more joints of a robot or a component of another mechanical agent. The control signal may also or instead include electronic control data, such as motor control data, or more generally include data for controlling one or more electronic devices within an environment, whose control has an effect on the observed state of the environment. For example, in the case of an autonomous or semi-autonomous land, air, or sea vehicle, the control signal may define actions for controlling navigation (e.g., steering) and movement (e.g., braking and / or acceleration of the vehicle). In a traffic control setting, actions may also or instead include physical movements, such as moving forward, turning left or right, yielding, etc., and signaling actions, such as sending a message or signal to another agent to show the agent's intention. In such a setting, processing one or more historical observations as well as the current observation can be used to indicate historical relationships between agents, such as vehicles. The reward can indicate whether the agent has completed a task, such as navigating to a target location in the environment, or the agent's progress toward completing the task, or some other measurement, such as the speed of the agent's progress, the energy or fuel used, the estimated time to reach the destination, the relative order of reaching the destination compared to another vehicle, etc. When used, the hypothetical reward of another agent can correspond to one or more of these.

[0072] In some embodiments, the environment is a simulation of the real-world environment described above, and the agent is implemented as one or more computers that interact with the simulated environment. For example, the simulated environment can be a simulation of a robot or vehicle, and the reinforcement learning system can be trained on the simulation and then used in the real world once trained.

[0073] In some embodiments, the environment is a real-world manufacturing environment for manufacturing products such as chemical, biological or mechanical products or food. As used herein, "manufacturing" a product also includes refining starting materials to create a product, or processing starting materials, for example to remove pollutants, to produce a clean or recycled product. A manufacturing plant may include multiple manufacturing units, such as containers for chemical or biological substances, or machines for processing solids or other materials, such as robots. The manufacturing units are configured so that intermediate versions or components of the product can be moved between manufacturing units during product manufacturing, for example, via pipelines or mechanical transport. As used herein, the manufacture of a product also includes the manufacture of food by a kitchen robot.

[0074] The agent may include an electronic agent configured to control a manufacturing unit or a machine such as a robot that operates to manufacture a product. That is, the agent may include a control system configured to control the manufacture of a chemical, biological, or mechanical product. For example, the control system may be configured to control one or more of the manufacturing units or machines, or to control the movement of intermediate versions or components of a product between manufacturing units or machines.

[0075] As one example, the tasks performed by the agent may include the task of manufacturing a product or an intermediate version or component thereof. As another example, the tasks performed by the agent may include the task of controlling - e.g., minimizing - the use of resources, such as the task of controlling the consumption of electricity or water or the consumption of any material or consumable used in the manufacturing process.

[0076] Actions may include control actions to control the use of a machine or manufacturing cell for processing solid or liquid materials to manufacture a product or an intermediate or component thereof, or to control the movement of an intermediate version or component of a product within a manufacturing environment, for example, between manufacturing cells or machines. In general, an action may be any action that has an effect on an observed state of the environment, for example, an action configured to adjust any of the sensed parameters described below. These actions may include actions that adjust the physical or chemical conditions of a manufacturing cell, or actions that control the movement of mechanical parts of a machine or joints of a robot. Actions may include actions that impose operating conditions on a manufacturing cell or machine, or actions that cause a setting change to adjust, control, or turn on or off the operation of a manufacturing cell or machine.

[0077] The reward or reward may be related to a measure of the performance of the task. For example, in the case of a task to manufacture a product, the measure may include the number of products manufactured, the quality of the product, the speed at which the product is produced, or a measure of the physical cost of performing the manufacturing task, such as a measure of the amount of energy, material, or other resource used to perform the task. In the case of a task to control the use of resources, the measure may include any measure of resource use.

[0078] In general, observations of environmental states may include any electronic signals representing the operation of electronic and / or mechanical items of equipment. For example, representations of environmental states may be derived from observations made by sensors that sense the state of a manufacturing environment, such as sensors that sense the state or configuration of a manufacturing unit or machine, or sensors that sense the movement of materials between manufacturing units or machines. As some examples, such sensors may be configured to sense mechanical movement or force, pressure, temperature; electrical conditions such as current, voltage, frequency, impedance; quantity, level, flow / movement rate, or flow / movement path of one or more materials; physical or chemical conditions such as physical state, shape, or configuration or chemical state such as pH; configuration of a unit or machine, such as a mechanical configuration or valve configuration of a unit or machine; image or video sensors for capturing images or video observations of a manufacturing unit or machine or movement; or any other appropriate type of sensor. In the case of a machine such as a robot, observations from sensors may include observations of the position, linear or angular velocity, force, torque, acceleration, or posture of one or more components of the machine, such as data characterizing the current state of the machine or robot or items held or processed by the machine or robot. Observations may also include, for example, sensed electronic signals such as motor current or temperature signals, or image or video data such as from a camera or lidar sensor. Sensors such as these may be part of the agent in the environment or located separately from the agent.

[0079] In some embodiments, the environment is a real-world environment of a service facility that includes multiple items of electronic equipment, such as a server farm or data center, such as a telecommunications data center, or a computer data center for storing or processing data, or any service facility. The service facility may also include auxiliary control equipment that controls the operating environment of the equipment items, such as environmental control equipment such as temperature control, such as cooling equipment, or air flow control or air conditioning equipment. Tasks may include tasks that control - for example, minimize - resource usage, such as tasks that control power consumption or water consumption. Agents may include electronic agents that are configured to control the operation of equipment items, or to control the operation of auxiliary - for example, environmental - control equipment.

[0080] In general, an action may be any action that has an effect on the observed state of the environment, for example, an action configured to adjust any of the sensed parameters described below. These may include actions that control an item of equipment or assist in controlling equipment or impose operating conditions on an item of equipment or assist in controlling equipment, for example, an action that causes a setting to be changed to adjust, control, or turn on or off the operation of an item of equipment or assist in controlling equipment.

[0081] Typically, observations of environmental conditions may include any electronic signal representing the operation of a facility or equipment in the facility. For example, representations of environmental conditions may be derived from observations made by any sensor sensing the state of the physical environment of the facility, or by any sensor sensing the state of one or more items of equipment or one or more items of auxiliary control equipment. These include sensors configured to sense electrical conditions such as current, voltage, power, or energy; temperature of the facility; fluid flow, temperature, or pressure within the facility or within the cooling system of the facility; or physical facility configurations, such as whether a vent is open.

[0082] The reward or reward may be related to a measure of the performance of the task. For example, in the case of a task to control - e.g., minimize - resource usage, such as a task to control electricity or water usage, the measure may include any measure of resource usage.

[0083] In some embodiments, the environment is a real-world environment of a power generation facility, such as a renewable power generation facility such as a solar farm or a wind farm. Tasks may include control tasks to control the power generated by the facility, such as to control the delivery of power to the distribution network, such as to meet demand or reduce the risk of mismatch between grid elements, or to maximize the power generated by the facility. The agent may include an electronic agent configured to control the power generation of the facility or to couple the generated power into the grid. When there are multiple agents, each agent may control the power generation by a different corresponding power generation entity, such as a facility or a part of a facility. The action may include an action to control the electrical or mechanical configuration of the power generator, such as the electrical or mechanical configuration of one or more renewable power generation elements, such as to control the configuration of a wind turbine or one or more solar panels or mirrors, or to control the electrical or mechanical configuration of a rotating generator. Mechanical control actions may, for example, include actions to control the conversion of energy input to electrical energy output, such as conversion efficiency or the degree of coupling of energy input to electrical energy output. Electrical control actions may, for example, include actions to control one or more of the voltage, current, frequency, or phase of the generated power.

[0084] The reward or reward may be related to a measure of the performance of the task. For example, in the case of a task to control the delivery of power to a distribution network, the measure may be related to a measure of the power delivered, or to a measure of an electrical mismatch such as a voltage, current, frequency, or phase mismatch between the generating facility and the grid, or to a measure of power or energy lost in the generating facility. In the case of a task to maximize the delivery of power to a distribution network, the measure may be related to a measure of power or energy delivered to the grid, or to a measure of power or energy lost in the generating facility.

[0085] In general, observations of environmental conditions may include any electronic signal representing the electrical or mechanical function of power generation equipment in a power generation facility. For example, representations of environmental conditions may be derived from observations made by any sensor that senses the physical or electrical state of equipment in a power generation facility that is generating electricity, or the physical environment of such equipment, or the condition of auxiliary equipment that supports the power generation equipment. Such sensors may include sensors configured to sense electrical conditions of equipment, such as current, voltage, power, or energy; temperature or cooling of the physical environment; fluid flow; or the physical configuration of the equipment; and observations of electrical conditions of the power grid, such as from local or remote sensors. Observations of environmental conditions may also include one or more predictions of future conditions regarding the operation of power generation equipment, such as predictions of future wind levels or solar irradiance or predictions of future electrical conditions of the power grid.

[0086] As another example, the environment can be a chemical synthesis or protein folding environment, such that each state is a corresponding state of a protein chain or one or more intermediates or precursor chemicals, and the agent is a computer system for determining how to fold a protein chain or synthesize a chemical. In this example, the actions are possible folding actions for folding a protein chain or actions for assembling precursor chemicals / intermediates, and the results to be achieved can include, for example, folding a protein to stabilize the protein and enable it to achieve a specific biological function or provide an efficient synthetic pathway for a chemical. As another example, the agent can be a mechanical agent that automatically performs or controls a protein folding action or chemical synthesis step selected by the system without human intervention. Observations can include direct or indirect observations of the state of a protein or chemical / intermediate / precursor and / or can be derived from a simulation.

[0087] In a similar manner, the environment can be a drug design environment, such that each state is a corresponding state of a potential pharmaceutically active compound, and the agent is a computer system for determining elements of the pharmaceutically active compound and / or a synthetic pathway for the pharmaceutically active compound. The drug / synthesis can be designed based on a reward derived from a drug goal, such as in a simulation. As another example, the agent can be a mechanical agent that performs or controls the synthesis of the drug.

[0088] As further examples, an action may include presenting an advertisement, an observation may include an ad impression or a click count or click rate, and a reward may characterize an item or content previously selected by one or more users.

[0089] In some cases, observations may include textual or verbal instructions provided to the agent by a third party (e.g., an operator of the agent). For example, the agent may be an autonomous vehicle, and the user of the autonomous vehicle may provide textual or verbal instructions to the agent (e.g., to navigate to a specific location).

[0090] As another example, the environment can be an electrical, mechanical, or electromechanical design environment, for example, an environment that simulates the design of an electrical, mechanical, or electromechanical entity. The simulation environment can be a simulation of a real-world environment in which the entity is expected to work. The task can be to design an entity. The observations can include observations that characterize the entity, i.e., observations of the mechanical shape or electrical, mechanical, or electromechanical configuration of the entity, or observations of parameters or properties of the entity. The actions can include actions that modify the entity, such as modifying one or more observations. The reward or reward can include one or more performance metrics of the design of the entity. For example, the reward or reward can relate to one or more physical properties of the entity, such as weight or strength, or to one or more electrical properties of the entity, such as a measure of the efficiency of performing the specific function for which the entity is designed. The design process can include outputting a manufacturing design, such as in the form of computer executable instructions for manufacturing the entity. The process can include manufacturing the entity according to the design. Thus, the design of the entity can be optimized by, for example, reinforcement learning, and then the optimized design for manufacturing the entity can be output, such as as computer executable instructions; the entity with the optimized design can then be manufactured. As another specific example, the agent may be a routing agent configured to route interconnects of an integrated circuit, the observations may be component locations and interconnects, the actions may be routing actions, such as interconnect or component selection and / or placement actions, and the rewards may depend on routing metrics, such as interconnect length, resistance, capacitance, impedance, loss, speed, or propagation delay, and / or physical line parameters, such as width, thickness, or geometry, and design rules. After routing, the routed interconnects may be used to manufacture the integrated circuit.

[0091] In some applications, one or more agents are software agents and the environment is a real-world computing environment. In one example, the agent manages the distribution of tasks across computing resources, such as on mobile devices and / or in data centers. In these applications, observations may include observations of computing resources such as computing and / or memory capacity or Internet accessible resources; and actions may include assigning tasks to specific computing resources. Rewards may be configured to maximize or minimize one or more of: utilization of computing resources, power, bandwidth, and computing speed.

[0092] In another example, a software agent manages the processing of a queue of jobs that arrive continuously, for example, by one or more real-world servers. Observations may include observations of the departure times of consecutive jobs, or the time intervals between the departures of consecutive jobs, or the time it takes a server to process each job, such as the start and end of a series of times, or the arrival times of consecutive jobs, or the time intervals between arrivals, or data characterizing the type of job. Actions may include actions to assign specific jobs to specific computing resources; rewards may be configured to minimize the overall queuing or processing time or the queuing or processing time of one or more individual jobs, or generally to optimize any metric based on observations.

[0093] In some applications, the environment is a real-world computing environment, and the software agent manages the distribution of tasks / jobs across computing resources, such as on mobile devices and / or in data centers. In these embodiments, observations may include observations related to the operation of computing resources in processing tasks / jobs, actions may include assigning tasks / jobs to specific computing resources, and rewards may be related to one or more metrics of using computing resources to process tasks / jobs, such as a usage metric of computing resources, bandwidth, or power, or a processing time metric, or a numerical accuracy metric, or one or more metrics related to a desired load balance between computing resources.

[0094] In some applications, the environment is a data packet communication network environment, and one or more agents are part of one or more routers for routing data packets through the communication network. The actions may include data packet routing actions, and the observations may include, for example, observations of a routing table including routing metrics, such as routing path length, bandwidth, load, number of hops, path cost, delay, maximum transmission unit (MTU), and reliability. The reward may be defined in relation to one or more routing metrics, i.e., configured to maximize the one or more routing metrics.

[0095] As previously mentioned, the environment can be a simulated environment. Typically in the case of a simulated environment, observations can include simulated versions of one or more previously described observations or observation types, and actions can include simulated versions of one or more previously described actions or action types. For example, the simulated environment can be a motion simulation environment, such as a driving simulation or a flight simulation, and the agent can be a simulated vehicle navigating through the motion simulation. In these embodiments, the action can be a control input to control a simulated user or a simulated vehicle. Typically, the agent can be implemented as one or more computers that interact with the simulated environment.

[0096] The simulation environment can be a simulation of a specific real-world environment and agent. For example, the system can be used to select actions in the simulation environment during training or evaluation of the system, and after training or evaluation or both are completed, it can be deployed to control the real-world agent in the specific real-world environment that is the subject of the simulation. This can avoid unnecessary wear and damage to the real-world environment or the real-world agent, and can allow the control neural network to be trained and evaluated in situations that rarely occur or are difficult or unsafe to reproduce in the real-world environment. For example, the system can be partially trained using a simulation of a mechanical agent in a simulation of a specific real-world environment, and later deployed to control a real mechanical agent in a specific real-world environment. Therefore, in such a case, the observations of the simulation environment are related to the real-world environment, and the selected actions in the simulation environment are related to the actions to be performed by the mechanical agent in the real-world environment.

[0097] Optionally, in any of the above embodiments, observations at any given step may include data from previous steps that may be useful in characterizing the environment, such as actions performed at previous steps, rewards received at previous steps, or both.

[0098] In some embodiments, performing a task may require the agent to interact with one, two, or more other agents in the environment that are also performing the same task or performing one or more other tasks. The operations of the other agents may be controlled by computer-implemented methods, e.g., similar to the methods for controlling the first agent, or some or all may be partially or fully controlled by humans.

[0099] Typically, the tasks may have the same characteristics, for example, they may all be routing tasks, navigation tasks, assembly tasks, etc., or they may all be scheduling tasks, or they may all be manufacturing tasks. The other tasks may be the same as or different from the task of the first agent. The tasks performed by the agent may contribute to achieving a common goal, or may be adversarial, i.e., the tasks performed by other agents may prevent the agent from performing the task. In a typical embodiment, the ability of the first agent to perform a task in the environment is affected by one or more other agents performing one or more other tasks in the environment.

[0100] Figure 2 is a flow chart of an example process 200 for training a policy neural network. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, an action selection system appropriately programmed according to the present specification, such as Figure 1 The action selection system 100 can perform the process 200.

[0101] In particular, the system initializes a regularization strategy (step 202). In general, a regularization strategy is a strategy that assigns a corresponding (non-zero) likelihood to each action in a set of actions.

[0102] The regularization strategy can be initialized to any arbitrary strategy that assigns non-zero likelihood to some or all actions in the action set. For example, the regularization strategy can be a uniform strategy that assigns the same score to each action, or a random strategy that assigns a random score to each action.

[0103] After initializing the regularization strategy, the system performs training across multiple training phases (also referred to as “stages”) (step 204 ).

[0104] In each training phase, the system trains the policy neural network using the regularization strategy up to that training phase to update the weights of the policy neural network (step 206).

[0105] For example, before the system trains the policy neural network in a first training phase, the system can initialize the weights of the policy neural network using appropriate machine learning initialization techniques.

[0106] For each subsequent training phase, the system is able to train the policy neural network starting from the weights determined at the previous training phase, i.e., without resetting the values ​​of the weights between training phases.

[0107] The training phase using the regularization strategy will be referenced below. Figure 3 Give a description.

[0108] At the completion of each training phase, the system updates the regularization strategy (step 208).

[0109] In particular, the system updates the regularized policy to be equal to the policy that controls the agent using the policy neural network as of the completion of the training phase, i.e., the policy neural network with updated weights that have been determined as of the completion of the training phase or the policy neural network with target weights determined as of the completion of the training phase. "Target weight" refers to a weight that is calculated as a moving average of the weights as they are updated during the training phase.

[0110] The strategy for controlling an agent using a policy neural network can be, for example, a strategy that samples actions according to scores generated by the policy neural network, adds noise to scores generated by the policy neural network, and then samples actions using the noisy scores, etc.

[0111] Figure 33 is a flow chart of an example process 300 for performing a training phase. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, an action selection system appropriately programmed according to the present specification, such as Figure 1 The action selection system 100 can perform the process 200.

[0112] In each training phase, the system generates training data for the task, which reflects the results of executing one or more action trajectories when performing the task (step 302).

[0113] An action trajectory can include data for some or all steps in an episode of performing a task.

[0114] An episode is a period of interaction in which the agent attempts to perform a task, starting from the initial state of the task and lasting until the agent successfully completes the task, interacts with the environment for a threshold number of steps, or meets different termination criteria.

[0115] For example, each trajectory can include data for all steps in an episode, or can include up to a maximum number of steps from an episode.

[0116] Thus, for each of a plurality of steps in a trajectory for performing a task, the training data includes a tuple including data identifying (i) an observation, (ii) an action performed in response to the observation in the tuple, and (iii) a reward for the task.

[0117] For example, the system is able to generate each trajectory by controlling the agent using a policy neural network during a task episode.

[0118] When the task requires multi-agent interaction, the system is also able to use policy neural networks to control other agents interacting in the environment.

[0119] Thus, in these cases, for a proper subset of steps in the trajectory, the action is the action performed by the agent in response to the observation, and for each of the multiple steps in the trajectory that are not in the proper subset, the action is the action performed by one of the one or more other agents in response to the observation in the tuple.

[0120] Therefore, actions in different tuples at different steps in the trajectory can be actions performed by different agents.

[0121] In addition, for each tuple, the reward for the task identified in the tuple is the reward of the agent that performs the action in response to the observation. In some cases, the tuple also includes data identifying the corresponding reward or hypothetical reward (the value of) for each agent other than the agent that performs the action in response to the tuple. For example, when two agents are performing an adversarial task, the reward for the agent that does not act at a given step can be the negative of the reward of the agent that acts at the given step, i.e., equal in magnitude but opposite in sign.

[0122] The system generates modified training data for the modified task from the training data (step 304 ).

[0123] More specifically, for each tuple, the system generates a transformed reward from the reward in the tuple using the regularized control policy as of the training phase, and generates a modified tuple that includes data identifying (i) the observation identified in the tuple, (ii) the action identified in the tuple, and (iii) the transformed reward for the tuple.

[0124] When there are multiple agents and multiple rewards per tuple, the system is able to generate a corresponding transformed reward for each reward in the tuple - that is, for each agent at each step in the trajectory.

[0125] Thus, the system modifies the "dynamics" of the task to generate a modified task, i.e., a task in which acting in the environment results in a transformed reward being received (rather than the actual reward in the original training data).

[0126] Modifying the rewards using a regularized control strategy will be referenced below. Figure 4 Described in more detail in .

[0127] Generating modified training data can be referred to as the "reward transformation" stage of the training phase.

[0128] The system then trains the policy neural network via reinforcement learning on the modified training data, e.g., on an objective that measures the expected reward of the transition (step 306).

[0129] For example, the goal can be to maximize the expected reward, where the reward is calculated as the sum or time-discounted sum of expected future transformation rewards. When there are multiple agents, the goal can be to maximize the expected reward for each agent, where the reward is calculated as the sum or time-discounted sum of expected future transformation rewards for that agent.

[0130] Training the policy neural network (and optionally the evaluator neural network) can be referred to as the dynamics stage of the training phase.

[0131] In general, the system can train the policy neural network on the modified training data using any suitable reinforcement learning technique.

[0132] As a specific example, the system can train a policy neural network in conjunction with a critic neural network using an actor-critic reinforcement learning technique.

[0133] An evaluator neural network is a neural network configured to receive an observation and process the observation to generate a predicted value for the observation, where the predicted value is a prediction of the reward (in terms of a transition reward) that would be received starting from the environment being in the state represented by the observation.

[0134] For example, the evaluator neural network can be implemented as a separate output head of the policy neural network that generates value predictions given input observations of the policy neural network.

[0135] Specifically, in this example, the system computes, for each trajectory, a corresponding value estimate for each step in the trajectory, which is an estimate of a combination—e.g., a sum or a time-discounted sum—of future transition rewards received after the environment is in the state represented by the tuple in the step, and then trains the policy neural network and the critic neural network using the transition rewards and the corresponding value estimates for the steps in each trajectory.

[0136] When there are multiple agents, the system computes, for each trajectory, a corresponding value estimate for each agent at each step in the trajectory, which is an estimate of the combination of future transition rewards received for the agent after the environment is in the state represented by the tuple in that step, and then trains the policy neural network and the critic neural network using the transition rewards and the corresponding value estimates of the agent at each step in the trajectory.

[0137] In some embodiments, the system computes the corresponding value estimate for each agent, e.g., recursively, starting from the last step of the trajectory and continuing to the first step of the trajectory, using the transformed rewards and outputs generated by the policy neural network and the critic neural network. Computing the value estimates in this manner can reduce bias. For example, the system can compute the corresponding value estimate for each agent using a V-trace estimator, e.g., as described in Espeholt et al. arXiv:1802.01561.

[0138] In some embodiments, to further reduce bias, the system can use data from a full episode to compute value estimates even if the trajectory is less than the length of a full episode, i.e., without any guidance. An example technique for computing estimates across a full episode when the trajectory is shorter than a full episode is described below.

[0139] In some embodiments, when calculating value estimates using the V-trace estimator, the system uses target values ​​for the weights of the policy neural network and the evaluator neural network. For example, during the training phase, the target value can be maintained as a moving average of the current value of the weight.

[0140] For example, the system can train an critic neural network using corresponding value estimates and train a policy neural network using a loss that depends on the output generated by the critic neural network, e.g., on the value estimate, the Q-function estimate, or both, generated using the critic neural network.

[0141] As a specific example, but not necessarily, the system can train the policy neural network by a neural replicator dynamics (NeuRD) loss calculated based on the output generated by the evaluator neural network. NeuRD loss is described in more detail in D. Hennes et al., Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS) (2020). For example, the NeuRD loss can be a loss of the sum of possible actions with respect to the gradient of the policy neural network weights that depends on the following metrics: (i) the difference between the observed policy output and the target value for steps in the trajectory, e.g., the average of the actions of the observed policy outputs, and (ii) an advantage estimate of the action, which is calculated using a value estimate and measures the relative advantage of executing the action in response to the observation relative to executing a different action from the set (in terms of the estimated return calculated from the transformed reward).

[0142] When there are multiple agents, for each step, the system can only calculate the loss of the agent acting at that step when training the policy neural network on NeuRD loss. For example, the overall loss can be the average NeuRD loss over all time steps in the trajectory.

[0143] To train the critic neural network, the system can train the neural network on a regression loss that measures, for each step in each trajectory, the error between the corresponding value estimate for that step and the predicted value generated by processing the observations in the tuple for that step using the critic neural network.

[0144] When there are multiple agents, for each step, the system can use only the value estimates of the agent acting at that step when training the critic neural network on a regression loss. For example, the regression loss can be the mean squared error over all time steps in the trajectory.

[0145] Typically, within a given training phase, the system is able to interleave training and generation of training data, i.e., such that at each of a plurality of training steps, the neural network is trained using some or all of the training data that has been collected up to that training step.

[0146] Therefore, by interleaving the iterations of steps 302 to 306 during the training phase, different tuples will be generated according to different weights of the policy neural network.

[0147] Alternatively, the system can first generate all the training data for the training phase, and then train the policy neural network on the training data.

[0148] After completing this phase, the system updates the regularization policy using the policy neural network as described above.

[0149] By repeatedly performing these training phases in which the system directly modifies the underlying learning dynamics by training the neural network on the modified tasks, the system is able to train the neural network to perform well even on tasks with very large state spaces, very large action spaces, or both. Furthermore, the system is able to train the neural network to perform well even in multi-agent scenarios with imperfect information (in addition to large state and / or action spaces), where observations convey both "private" and "public" information, and the "private" information of other agents is not available to the agent that is acting.

[0150] The training technique is therefore called "regularized Nash dynamics" because at each training level the system learns a policy that converges (approximately) to a fixed point on the modified task defined by the modified training data generated using the regularized policy.

[0151] In some implementations, after performing process 300 for each of a plurality of training phases, the system fine-tunes the policy neural network using the modified training technique.

[0152] For example, the system can fine-tune the policy neural network by applying a different projection to the log odds generated by the policy neural network when calculating the value estimates used in training, i.e., the action scores. For example, the system can fine-tune the training using a different projection that thresholds and discretizes the action probabilities using a specified threshold level and probability quanta. The threshold level and probability quanta can be hyperparameters of the training process. Performing this fine-tuning can help reduce low-probability errors caused by the policy neural network due to the softmax projection assigning non-zero probabilities to each action in the space.

[0153] Figure 4 is a flow chart of an example process 400 for modifying the reward for a given step in a trajectory using a regularization strategy. For convenience, process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, an action selection system appropriately programmed according to the present specification, such as Figure 1 The action selection system 100 is capable of performing the process 400.

[0154] The system generates a first transformed reward from the reward at a given step using the regularized control policy up to the training phase (step 402).

[0155] To generate the first transition reward, the system calculates the ratio of (i) the probability assigned by the policy neural network to the action performed by the agent at that step (based on the weights when the actions were selected) to (ii) the probability assigned by the regularized policy to the action performed by the agent at that step.

[0156] The system then calculates a modifier based on this ratio.

[0157] As an example, the system can calculate the modifier based on the product of the regularization parameter and the logarithm of the ratio.

[0158] The regularization parameter can be a hyperparameter of training and controls how strongly the regularization policy affects the actual reward. For example, a larger regularization parameter can produce more stable and faster convergence during training of the policy neural network in a single training epoch, but can provide a bias towards the regularization control policy, which may mean slower overall training to convergence over multiple training epochs.

[0159] The system then applies the modifier to the agent's reward to generate a first transformed reward. For example, when the agent receiving the reward is the agent that acted at the step, the system can subtract the modifier from the reward to generate the first transformed reward. When the agent receiving the reward is not the agent that acted at the step, the system can add the modifier to the reward to generate the first transformed reward.

[0160] In some cases, the system sets the transformation reward equal to the first transformation reward.

[0161] In some other cases, the system smooths the transition from one regularization policy to another between training phases by incorporating the regularization policy from the previous training phase into the calculation of the modified reward. For the first training phase, i.e., without a previous training phase, the system can set the regularization policy from the previous training phase equal to the regularization policy of the first training phase.

[0162] In particular, in these cases, the system generates a second transformation reward using a previous regularized control policy from a previous training phase (step 404). That is, the system can calculate the second transformation reward in the same manner as the first transformation reward, but using the previous regularized control policy from the previous training phase instead of the regularized control policy of the current training phase.

[0163] Then, the system generates a transformed reward by combining the first transformed reward and the second transformed reward (step 406). For example, the system can calculate a weighted sum of the first transformed reward and the second transformed reward to generate the transformed reward.

[0164] In some implementations where multiple training iterations are performed during a training phase, the weights in the weighted sum can depend on the iteration, with iterations occurring later during the training of the policy neural network assigning higher weights to the first transformed reward. Typically, each training iteration corresponds to an update to the policy neural network weights during the training phase. For example, each training iteration can correspond to training the neural network on a batch of training data (or a set of multiple batches of training data in parallel, such as in a SIMD architecture described below) during the training iteration.

[0165] In some of these embodiments, after a threshold training iteration, the system no longer uses the second transform reward and sets the transform reward equal to the first transform reward.

[0166] For example, the weight assigned to the first transformation reward can be equal to the minimum of 1 and a constant a, such as two, three, or four, multiplied by the ratio of the index of the current iteration to the total number of iterations. The weight assigned to the second transformation reward can be equal to 1 minus the weight assigned to the first transformation reward. Thus, in this example, once more than 1 / a has been trained during a given training phase, the system stops using the second transformation reward and sets the transformation reward equal to the first transformation reward.

[0167] As some specific examples, in some embodiments, it is possible to add or subtract items from a reward. to generate the transformation reward, where η is the regularization parameter, π(a|o) is the action selection policy defined by the policy neural network (e.g., the observed action probability α or the observed history o), and π reg (α|o) is the regularized control policy. By way of example only, η can be in the range of 0 to 1, such as 0.2. In some embodiments, the term added to or subtracted from the reward can be ηγ(α|o), where where π reg,curr is the current regularized control policy, and π reg，prev is the previous regularized control strategy, and a is interpolated between them, e.g., for Δ m The iteration n in the total iterations is a=min(1,2n / Δ m ).

[0168] Figure 5 An example architecture of policy neural network 122 is shown.

[0169] exist Figure 5In the example of , the policy neural network 122 includes a neural network trunk 520 and three policy neural network heads 540.

[0170] In addition, Figure 5 In the example of , the evaluator neural network is implemented as a value head 550 that receives the output of the neural network trunk 520.

[0171] like Figure 5 As shown, trunk 520 processes observation 510 to generate an embedding of the environment state as output.

[0172] Each of the policy neural network head 540 and the value head 550 is configured to receive an input including an embedding of a state of the environment, and process the input to generate a corresponding output, such as a corresponding policy output or a value estimate.

[0173] As described above, when policy neural network 122 includes multiple policy heads 540, each policy head corresponds to a different subset of actions in the set.

[0174] In this example, to select an action, the system can select one of the subsets and then select an action using the output of the head corresponding to that subset, such as described above. For example, each subset can be valid at a different "stage" of the task, such as a different set of time steps during an episode of performing the task, or can be valid at a different set of environmental states.

[0175] Optionally, the system can also receive phase-specific observations 520, which are also provided as input to heads 540 and 550. Phase-specific observations 520 can include different types of data for different phases of the task. For example, the data can be at different levels of granularity for different phases, e.g., having a higher level of granularity at the beginning of the task and a lower level of granularity at the end of the task.

[0176] exist Figure 5 In the example of , the policy neural network 122 is a convolutional neural network. More specifically, in Figure 5 In the example of , the torso 520 and heads 540 and 550 each include a corresponding pyramid module 560, which includes a "pyramid" of convolutional neural network layers, for example, having convolutional residual blocks ("convresblocks") followed by deconvolutional residual blocks ("deconvresblocks").

[0177] More generally, the policy neural network 122 can have different types of convolutional architectures or architectures with different types of neural network layers, such as recurrent layers, attention layers, etc.

[0178] Fig. 6AAn example training architecture of a training system for performing the training phase is shown.

[0179] like Fig. 6A As shown, during the training phase, multiple actors 602 independently and asynchronously generate training data ("experience") using the policy neural network 122 as described above, and store the generated training data in the replay buffer memory 604. For example, the actor can use "self-play", in which all agents in the environment are controlled by the policy neural network 122. A single learner 606 repeatedly samples training data from the replay buffer memory 604 and uses the training data to train the policy neural network 122 as described above. The learner 606 then periodically sends the updated weights ("parameters") generated as a result of the training to the actor 602 so that the actor 602 can use the updated weights to generate additional training data.

[0180] The actors 602 and learners 606 can be implemented in a variety of configurations. For example, each actor 602 can be implemented on a respective machine, and the learner 606 can be implemented on a different machine.

[0181] As another example, all of the actors 602 and learners 606 can be implemented on the same machine. In this example, the actors 602 can be implemented as software using primarily CPU resources, the buffers 604 can use primarily RAM or other device memory, and the learners 606 use primarily resources provided by the machine's machine learning hardware accelerator.

[0182] Optionally, the training system 190 can also include one or more evaluators 608 that evaluate the performance of the policy neural network 122. Fig. 6A As shown in the example of , the evaluator 608 is able to evaluate the performance of the policy neural network 122 by controlling other agents using different control strategies ("bots").

[0183] Figure 6B Another example training architecture for a training system that performs a training stage is shown.

[0184] like Figure 6B As shown, during the training phase, the system utilizes multiple learner machines 660 to scale the training pipeline using a single instruction multiple data (SIMD) model.

[0185] In this example, several learner machines 660 participate in training, and each machine 660 implements multiple actors 602, buffers 604, and learners 606. The actors 602 within each machine 660 generate training data asynchronously, as described above. Additionally, the learners 606 within each machine 660 perform learning steps synchronously with the learners 606 of other machines 660: each learner reads a different batch of training data from the corresponding buffer 604 and calculates the network parameter gradients; then the system 190 calculates the average of the gradients across all machines 660; and finally each learner 606 applies those average gradients as updates to the network weights.

[0186] As described above, in some embodiments, to further reduce bias, the system is able to use data from a full episode to compute value estimates even if the trajectory is less than the length of a full episode, i.e., without any guidance. An example technique for computing these estimates in configuration 600 or 650 when the episode is of variable length will now be described.

[0187] Computing exact rewards for variable-length trajectories requires storing complete episodes (instead of fixed-length sequences of time steps) in the replay buffer. When the learner 606 samples batches of episodes, shorter episodes are padded to the length of the longest episode in the batch. The batch is cut into a variable number of fixed-length chunks along the time axis. The learner 606 then makes two passes over the list of chunks: a) a forward pass to recompute the exact policy and trajectory statistics for each time step; b) a backward pass, where each chunk is reprocessed again, but the chunk is given the exact network state and complete trajectory statistics, such as exact rewards, at the beginning of the chunk. Note that (a) occurs outside the gradient computation and is used to avoid bootstrapping or any kind of approximation for both the rewards and the network state. Whereas (b) occurs within the loss and is used to compute the gradients of the network weights.

[0188] The gradients computed for each chunk are accumulated during (b) and applied to the network weights at the end of the (b) pass using the optimizer.

[0189] This specification uses the term "configured" in connection with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof, which, in operation, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a particular operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0190] The embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more computer program instruction modules encoded on a tangible non-transitory storage medium, for execution by a data processing device or for controlling the operation of the data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0191] The term "data processing apparatus" refers to data processing hardware and covers all kinds of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or a plurality of processors or computers. The apparatus can also be or further include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus can optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0192] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, a single file dedicated to the program in question, or in multiple coordinated files, such as files storing one or more modules, subroutines or code portions. A computer program can be deployed and executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0193] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.

[0194] The processes and logic flows described in this specification can be performed by one or more programmable computers implementing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit such as an FPGA or ASIC, or by a combination of a dedicated logic circuit and one or more programmed computers.

[0195] The computer suitable for executing computer programs can be based on a general or special microprocessor or both, or any other kind of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic element of a computer is a central processing unit for executing or implementing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented or incorporated therein by a special logic circuit. Typically, the computer will also include or be operably coupled to receive data from one or more large-capacity storage devices such as magnetic disks, magneto-optical disks or optical disks for storing data or to transmit data thereto or both. However, the computer does not need to have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0196] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0197] To provide interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and a pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and the input from the user can be received in any form including acoustic, voice, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. In addition, the computer can interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving a response message from the user.

[0198] The data processing apparatus for implementing the machine learning model can also include, for example, dedicated hardware accelerator units for processing the common and computationally intensive part of machine learning training or production, namely the inference workload.

[0199] Ability to implement and deploy machine learning models using machine learning frameworks such as the Tensor Flow framework, Microsoft Cognitive Toolkit framework, Apache Singa framework, or Apache MXNet framework.

[0200] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer with a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0201] A computing system can include a client and a server. The client and the server are usually remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. In some embodiments, the server transmits data such as HTML pages to the user device, for example, to display data to a user interacting with the device as a client and to receive user input from the user. Data generated at the user device, such as the result of the user interaction, can be received from the device at the server.

[0202] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or the scope that can be claimed, but should be interpreted as descriptions of features that can be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment can also be implemented separately or in any suitable sub-combination in multiple embodiments. In addition, although features can be described above as working in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination can be deleted from the claimed combination, and the claimed combination can be directed to a sub-combination or a variant of the sub-combination.

[0203] Similarly, although operations are depicted in the figures and described in the claims in a particular order, it should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or grouped into multiple software products.

[0204] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing can be advantageous.

Claims

1. A method for training a policy neural network for controlling an agent that interacts with an environment to perform a task, the method comprising: Initializing a regularized control strategy for controlling the agent; as well as In each of the plurality of training stages, an operation is performed, the operation comprising: generating training data for the task, the training data comprising, for each of a plurality of steps during a trajectory for performing the task, a tuple comprising data identifying (i) an observation, (ii) an action performed in response to the observation in the tuple, and (iii) a reward for the task; Generating modified training data of the modified task according to the training data, comprising, for each tuple: Generate a transformed reward from the rewards in this tuple using the regularized control policy up to this training stage, and generating a modified tuple, the modified tuple comprising data identifying (i) the observation identified in the tuple, (ii) the action identified in the tuple, and (iii) the transformed reward for the tuple; and training the policy neural network on the modified training data; and After training the policy neural network, the regularized control policy is updated to a policy that selects actions using the policy neural network.

2. The method according to claim 1, wherein: The environment includes one or more other agents, and performing the task requires interaction with the one or more other agents.

3. The method according to claim 2, wherein: For a proper subset of steps in the trajectory, the action is the action performed by the agent in response to the observation, and For each step of the plurality of steps in the trajectory that is not in the proper subset, the action is an action performed by one of the one or more other agents in response to the observation in the tuple.

4. The method according to claim 3, wherein: The observation at each step represents (i) public information about the state of the environment available to the agent and the one or more other agents, and (ii) private information about the state of the environment available only to the agent that performs the action in response to the observation.

5. The method according to any one of claims 2 to 4, wherein: For each tuple, the reward for the task identified in the tuple is a reward for the agent that performs the action in response to the observation, and wherein the tuple further includes data identifying a corresponding reward for each agent other than the agent that performs the action in response to the tuple.

6. The method according to any one of claims 2 to 5, wherein: Generating the training data includes controlling the agent to perform episodes of the task using the policy neural network.

7. A method according to claim 6 when dependent on claim 3, wherein: Generating the training data includes controlling the agent to perform episodes of the task using the policy neural network while also controlling the one or more other agents using the policy neural network.

8. A method according to any preceding claim, wherein: Training the policy neural network on the modified task includes training the policy neural network through reinforcement learning on an objective that measures an expected transformed reward.

9. The method according to claim 8, wherein: Training the policy neural network includes jointly training the policy neural network with an evaluator neural network, the evaluator neural network being configured to receive an observation and process the observation to generate a predicted value for the observation, and wherein the training includes: for each trajectory, computing a corresponding value estimate for each step in the trajectory, the value estimate being an estimate of a combination of future transition rewards received after the environment is in the state represented by the tuple in that step; and The policy neural network and the critic neural network are trained using the transformed rewards and corresponding value estimates for each step in each trajectory of the trajectory.

10. The method according to claim 9, wherein: Training the critic neural network includes training the critic neural network using a regression loss that measures, for each step in each trajectory, the error between a corresponding value estimate for the step and a predicted value generated by processing the observations in the tuple for the step using the critic neural network.

11. The method according to claim 9 or claim 10, wherein: Training the policy neural network and the critic neural network using the transformed rewards and corresponding value estimates for each step in each trajectory of the trajectory includes training the policy neural network via a neural replicator dynamics (NeuRD) loss.

12. A method according to any preceding claim, wherein: Generating a transformed reward from the rewards in the tuple using the regularized control strategy comprises: A first transformation reward is generated using the regularized control policy.

13. The method according to claim 12, wherein: Generating a transformed reward from the rewards in the tuple using the regularized control strategy includes: generating a second transformation reward using a prior regularized control policy from a prior training phase; and The transformed reward is generated by combining the first transformed reward and the second transformed reward.

14. The method according to claim 13, wherein: Generating the transformed reward by combining the first transformed reward and the second transformed reward comprises: A weighted sum of the first transformed reward and the second transformed reward is calculated.

15. The method of any one of claims 12 to 14, wherein Generating a first transformation reward using the regularized control strategy includes: calculating a modifier based on a ratio of (i) a probability assigned by the policy neural network to the action performed by the agent at the step and (ii) a probability assigned by the regularized policy to the action performed by the agent at the step; and Applying the modifier to the reward of the agent.

16. The method according to claim 15, wherein: Applying the modifier to the reward of the agent includes: The modifier is subtracted from the reward.

17. The method according to claim 15 or claim 16, wherein: The modifier is based on the product of a regularization parameter and the logarithm of the ratio.

18. A method according to any preceding claim when dependent on claim 9, wherein: Computing the corresponding value estimate for each step in the trajectory includes: The corresponding value estimate is recursively calculated starting from the last step in the trajectory and continuing until the first step in the trajectory.

19. A method according to any preceding claim, wherein: The policy neural network is configured to receive an input comprising an observation characterizing a state of the environment and to generate a policy output defining an action from a set of actions to be performed in response to the observation.

20. A method according to any preceding claim, wherein: The agent is a mechanical agent and the environment is a real-world environment.

21. The method according to claim 20, wherein: The agent is a robot.

22. A method according to any preceding claim, wherein: The environment is a real-world environment of a service facility including a plurality of items of electronic equipment, and the agent is an electronic agent configured to control the operation of the service facility.

23. A method according to any preceding claim, wherein: The environment is a real-world manufacturing environment for manufacturing a product, and the agents include electronic agents configured to control a manufacturing cell or a machine that operates to manufacture the product.

24. The method according to any one of claims 1 to 19, wherein: The environment is a simulation of a real-world environment, and the agent is a simulation of a real-world agent, and wherein the method further comprises: After the training, the policy neural network is provided for controlling the real-world agent in the real-world environment, in particular by processing observations of the environment using the policy neural network to generate policy outputs to control actions of the agent in the environment.

25. A method according to any one of claims 1 to 19, wherein The environment is a simulation of a real-world environment, and the agent is a simulation of a real-world agent, and wherein the method further comprises: After the training, the policy neural network is used to control the real-world agent in the real-world environment, in particular by processing observations of the environment using the policy neural network to generate policy outputs to control actions of the agent in the environment.

26. The method according to claim 24 or 25, wherein: The agent is a mechanical agent.

27. The method according to claim 26, wherein: The agent is a robot.

28. The method according to claim 24 or 25, wherein: The real-world environment is a real-world environment of a service facility including a plurality of electronic equipment items, and the real-world agent is an electronic agent configured to control the operation of the service facility.

29. The method according to claim 24 or 25, wherein: The real-world environment is a real-world manufacturing environment for manufacturing a product, and the real-world agent includes an electronic agent configured to control a manufacturing unit or a machine operating to manufacture the product.

30. A system comprising: one or more computers; as well as One or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the corresponding method according to any one of claims 1 to 29.

31. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1 to 29.

Images