Training an Action Selection System Using Relative Entropy Q-Learning

Abstract

To train an action selection system using reinforcement learning techniques.SOLUTION: A method comprises at each of multiple iterations: a step of obtaining a batch of experience, each experience tuple comprising a first observation, an action, a second observation, and a reward; a step, for each experience tuple, of determining a state value for the second observation, the step comprising: processing the first observation using a policy neural network to generate an action score for each action in a set of possible actions; sampling multiple actions from the set of possible actions in accordance with the action scores; processing the second observation using a Q neural network to generate a Q value for each sampled action; and determining the state value for the second observation; and a step of determining an update to current values of the Q neural network parameters using the state values.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of the filing date of U.S. Provisional Patent Application No. 63 / 057,826, filed July 28, 2020, which is incorporated herein by reference in its entirety.

[0002] This specification relates to processing data using machine learning models. [Background technology]

[0003] A machine learning model receives input and generates an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models, generating an output based on the received input and on the values ​​of the parameters of the model.

[0004] Some machine learning models are deep models that utilize multiple layers of the model to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a nonlinear transformation to the received input to generate an output. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Rae Jeong et al., "Learning Dexterous Manipulation from Suboptimal Experts," arXiv:2010.08587v2, January 5, 2021 Summary of the Invention [Means for solving the problem]

[0006] This specification generally describes a system, implemented as a computer program on one or more computers at one or more locations, for training an action selection system used to control an agent that interacts with an environment to perform a task using reinforcement learning techniques. The reinforcement learning techniques described herein may be referred to as relative entropy Q-learning. The action selection system may comprise a Q-neural network and a policy neural network, as described in more detail below.

[0007] According to a first aspect, there is provided a method executed by one or more data processing devices for training an action selection system used to select actions to be performed by an agent interacting with an environment to perform a task. The action selection system comprises a Q-neural network and a policy neural network. The method includes, at each of a plurality of iterations, retrieving a batch of experience tuples from a replay buffer, the batch characterizing previous interactions of the agent with the environment. Each experience tuple includes (i) a first observation characterizing a state of the environment, (ii) an action performed by the agent in response to the first observation, (iii) a second observation characterizing a state of the environment after the agent performs the action in response to the first observation, and (iv) a reward received as a result of the agent performing the action in response to the first observation. The method includes, for each experience tuple, determining, in each of the plurality of iterations, a state value for a second observation in the experience tuple, the step including processing the first observation in the experience tuple with a policy neural network to generate a respective action score for each action in a set of possible actions that can be performed by the agent, sampling a plurality of actions from the set of possible actions according to the action scores, processing the second observation with a Q-neural network to generate a respective Q-value for each sampled action, and determining a state value for the second observation using the Q-neural network parameters. The method further includes, in each of the plurality of iterations, using the state value for the second observation in the experience tuple to determine an update to current values ​​of a set of Q-neural network parameters of the Q-neural network.

[0008] In some implementations, the action selection system is used to select an action to be performed by a machine agent, e.g., a robot, that interacts with a real-world environment to perform a task. For example, the action selection system may be used to process observations regarding the real-world environment, and the selected action may relate to an action to be performed by the machine agent. Thus, the method may further include using the action selection neural network to control the machine agent, e.g., a robot, to perform a task while interacting with the real-world environment by obtaining observations from one or more sensor devices that sense the real-world environment, e.g., sensor data from image, distance, or position sensors of the machine agent, or from actuators, and processing the observations using the action selection system to select an action for controlling the machine agent to perform the task.

[0009] In some implementations, for each experience tuple, determining a state value for the second observation using the Q values ​​for the sampled actions includes determining the state value for the second observation as a linear combination of the Q values ​​for the sampled actions.

[0010] In some implementations, determining the state value for the second observation as a linear combination of the Q values ​​for the sampled actions includes determining a temperature coefficient based on the Q values ​​for the sampled actions, determining a respective modified Q value for each sampled action as a ratio of (i) the Q value for the sampled action and (ii) the temperature coefficient, applying a softmax function to the modified Q values ​​to determine a weighting factor for each sampled action, and determining the state value for the second observation as a linear combination of the Q values ​​for the sampled actions, wherein the Q value for each sampled action is scaled by the weighting factor for the sampled action.

[0011] In some implementations, the state value for the second observation is:

number

number

[0012] In some implementations, determining the temperature coefficient based on the Q value for the sampled action includes determining, in each of one or more optimization iterations, a gradient of a dual function with respect to the temperature coefficient, where the dual function depends on (i) the temperature coefficient and (ii) the Q value for the sampled action, and adjusting a current value of the temperature coefficient using the gradient of the dual function with respect to the temperature coefficient.

[0013] In some implementations, the dual function is:

number

[0014] In some implementations, determining an update to the current values ​​of the set of Q-neural network parameters of the Q-neural network using the state value for the second observation in the experience tuple includes, for each experience tuple, processing the first observation in the experience tuple using the Q-neural network to generate a Q-value for the action in the experience tuple, using the state value for the second observation in the experience tuple to determine a target Q-value for the action in the experience tuple, and, for each experience tuple, determining a gradient of a Q-objective function that measures the error between (i) the Q-value for the action in the experience tuple and (ii) the target Q-value for the action in the experience tuple, and using the gradient to determine an update to the current values ​​of the set of Q-neural network parameters.

[0015] In some implementations, determining a target Q-value for an action in the experience tuple using a state value for a second observation in the experience tuple includes determining the target Q-value as the sum of the product of (i) the reward in the experience tuple and (ii) a discount factor and the state value for the second observation in the experience tuple.

[0016] In some implementations, the error between (i) the Q value for the action in the experience tuple and (ii) the target Q value for the action in the experience tuple comprises a squared error between (i) the Q value for the action in the experience tuple and (ii) the target Q value for the action in the experience tuple.

[0017] In some implementations, the Q objective function is:

number

[0018] In some implementations, the method further includes, at each of a plurality of iterations, determining updates to current values ​​of a set of policy neural network parameters of the policy neural network, the steps including, for each experience tuple, processing a first observation in the experience tuple using a Q neural network to generate a Q value for the action in the experience tuple, determining a state value for the first observation in the experience tuple, and determining an advantage value for the experience tuple as the difference between (i) the Q value for the action in the experience tuple and (ii) the state value for the first observation in the experience tuple; and determining updates to current values ​​of the set of policy neural network parameters of the policy neural network based only on experience tuples having non-negative advantage values.

[0019] In some implementations, determining updates to current values ​​of a set of policy neural network parameters of a policy neural network based only on experience tuples with non-negative advantage values ​​includes determining a gradient of a policy objective function that depends only on experience tuples with non-negative advantage values, and using the gradient to determine updates to current values ​​of the set of policy neural network parameters.

[0020] In some implementations, for each experience tuple with a non-negative advantage value, the policy objective function depends on the action score for the action in the experience tuple that is generated by processing the first observation in the experience tuple using the policy neural network.

[0021] In some implementations, the policy objective function is:

number

[0022] In some implementations, the method further includes, in each of one or more of the plurality of iterations, generating a plurality of new experience tuples using an action selection system, an expert action selection strategy, or both, and adding the new experience tuples to the replay buffer.

[0023] In some implementations, generating a plurality of new experience tuples includes, at each of one or more time steps, receiving a current observation for the time step, selecting an action to be performed by the agent at the time step using an action selection system or expert action selection strategy, receiving a next observation and a reward resulting from the agent performing the selected action, and generating a new experience tuple including the current observation, the selected action, the next observation, and the reward.

[0024] In some implementations, selecting an action to be performed by an agent at a time step using an action selection system or expert action selection strategy includes probabilistically choosing between using the action selection system or expert action selection strategy to select an action to be performed by an agent at a time step.

[0025] In some implementations, selecting an action to be performed by an agent at a time step using an action selection system includes: processing current observations for the time step using a policy neural network to generate a respective action score for each action in a set of possible actions; processing current observations for the time step using a Q neural network to generate a respective Q value for each action in the set of possible actions; determining a final action score for each action based on (i) the action scores for the action and (ii) the Q values ​​for the action; and selecting an action to be performed by the agent according to the final action scores.

[0026] In some implementations, the final action score for an action is:

number

[0027] In some implementations, the agent is a robotic agent that interacts with a real-world environment, and the expert action selection strategy is generated by orchestrating a waypoint tracking controller.

[0028] According to another aspect, a system is provided that includes one or more computers and one or more storage devices communicatively coupled to the one or more computers, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the techniques described herein.

[0029] According to another aspect, one or more non-transitory computer storage media are provided that store instructions that, when executed by one or more computers, cause the one or more computers to perform operations of the techniques described herein.

[0030] Particular embodiments of the present subject matter described herein can be implemented to realize one or more of the following advantages.

[0031] The training system described herein can train an action selection system used to control an agent interacting with an environment using both policy-on and policy-off experience tuples that characterize the agent's previous interactions with the environment. In particular, the training system can train the action selection system using mixed training data including, for example, policy-on "exploration" experience tuples generated by the action selection system that select actions relevant to performing a task and policy-off "expert" experience tuples generated by an expert action selection policy. A particular advantage of the techniques described herein is that they can benefit from suboptimal expert action selection policies, i.e., those that select actions that are highly off-policy and relevant to only a portion of the task, and can effectively combine them with policy-on "exploration" data that characterizes the agent's interactions with the environment, obtained by using the method. In the context of a robotic agent interacting with a real-world environment, an expert action selection policy can be generated by orchestrating the (suboptimal) expert action selection policies of a waypoint tracking controller. By being trained on mixed training data (e.g., rather than policy-on training data alone), the action selection system can be trained more quickly (e.g., over fewer training iterations) and achieve better performance (e.g., by enabling the agent to perform the task more effectively). By training the action selection system more quickly, the training system can consume fewer computational resources (e.g., memory and computing power) during training than some conventional training systems.

[0032] The training system described herein can train a Q-neural network with experience tuples representing interactions between an agent and an environment, using state values ​​determined by importance sampling using a policy neural network. For example, to determine a state value for an observation, the training system can use the policy neural network to process prior observations to generate a score distribution over a set of possible actions, sample multiple actions according to the score distribution, and determine the state value by combining the Q-values ​​for the sampled actions. Training the Q-neural network with state values ​​determined by importance sampling using a policy neural network can regularize and accelerate the training of the Q-network, thus reducing the consumption of computational resources by the training system.

[0033] The training system described herein can train a policy neural network with experience tuples that represent previous interactions between the agent and the environment, where the agent performed carefully selected actions, e.g., actions associated with non-negative advantage values. The advantage value for a given action can characterize the difference between the return (e.g., a cumulative measure of reward) received by performing the given action and the return received by performing an average (e.g., randomly selected) action. Training a policy neural network with experience tuples that represent effective agent interactions with the environment can accelerate the training of the action selection system (thus reducing the consumption of computational resources during training) and improve the performance of the trained action selection system.

[0034] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0035] [Figure 1] FIG. 1 is a block diagram of an exemplary action selection system. [Figure 2] FIG. 1 is a block diagram of an exemplary training system. [Figure 3] 1 is a flow diagram of an exemplary process for training an action selection system using relative entropy Q-learning. [Figure 4] 1 is a flow diagram of an example process for determining state values ​​of observations. [Figure 5] 1 is a flow diagram of an exemplary process for updating the current values ​​of Q network parameters. [Figure 6] 1 is a flow diagram of an example process for updating current values ​​of strategy network parameters. [Figure 7] 1 is a flow diagram of an exemplary process for using an action selection system to select an action to be performed by an agent to interact with an environment. DETAILED DESCRIPTION OF THE INVENTION

[0036] Like reference numbers and designations in the various drawings indicate like elements.

[0037] 1 illustrates an exemplary action selection system 100. Action selection system 100 is an example of a system, implemented as a computer program on one or more computers at one or more locations, in which the systems, components, and techniques described below are implemented.

[0038] The system 100 selects an action 112 to be performed by an agent 114 interacting with an environment 116 at each of a plurality of time steps to accomplish a goal. At each time step, the system 100 receives data characterizing the current state of the environment 116, e.g., an image of the environment 116, and selects an action 112 to be performed by the agent 114 in response to the received data. The data characterizing the state of the environment 116 is referred to herein as observations 120. At each time step, the state of the environment 116 at the time step (characterized by the observations 120) depends on the state of the environment 116 at the previous time step and the action 112 performed by the agent 114 at the previous time step.

[0039] At each time step, the system 100 may receive a reward 118 based on the current state of the environment 116 and the actions 112 of the agent 114 at that time step. Generally, the reward 118 may be expressed as a numerical value. The reward 118 may be based on any event or aspect of the environment 116. For example, the reward 118 may indicate whether the agent 114 accomplished a goal (e.g., navigated to a goal location in the environment 116 or completed a task) or the progress of the agent 114 toward accomplishing a goal, e.g., a task.

[0040] In some implementations, the environment is a real-world environment and the agent is a machine agent that interacts with the real-world environment, e.g., moves within (by translating and / or rotating within the environment and / or changing its configuration) and / or modifies the real-world environment. For example, the agent may be a robot that interacts with the environment, e.g., to locate an object of interest within the environment, to move an object of interest to a specified location within the environment, to physically manipulate an object of interest within the environment, or to navigate to a specified destination within the environment, or the agent may be an autonomous or semi-autonomous land, air, or sea vehicle that navigates through the environment toward a specified destination within the environment.

[0041] In these implementations, the observations may include, for example, one or more of images, object position data, and sensor data, e.g., from image, distance, or position sensors, or from actuators, to capture the observations as the agent interacts with the environment.

[0042] For example, in the case of a robot, the observations may include data characterizing the current state of the robot, such as one or more of joint positions, joint velocities, joint forces, torques or accelerations, e.g., gravity compensation torque feedback, and global or relative poses of members assumed by the robot.

[0043] In the case of a robot or other mechanical agent or vehicle, the observations may similarly include one or more of position, linear or angular velocity, force, torque or acceleration, and global or relative pose of one or more parts of the agent. The observations may be defined in one, two, or three dimensions and may be absolute and / or relative observations.

[0044] The observations may include, for example, data acquired by one or more sensor devices that sense the real-world environment, e.g., sensed electronic signals such as motor current or temperature signals, and / or image or video data, e.g., from a camera or LIDAR sensor, e.g., data from sensors on the agent or from sensors located separately from the agent in the environment.

[0045] In the case of an electronic agent, the observations may include data from one or more sensors monitoring portions of the facility or service equipment, such as current, voltage, power, temperature, and other sensors and / or electronic signals representing that the electronic and / or mechanical components of the equipment are functioning.

[0046] An action may be a control input or advanced control command to control a robot, e.g., torque on a joint of the robot, or a torque or advanced control command to an autonomous or semi-autonomous land, air, or vehicle, e.g., a control surface or other control element of the vehicle.

[0047] In other words, an action may include, for example, position, velocity, or force / torque / acceleration data for one or more joints of a robot or part of another mechanical agent. Actions may additionally or alternatively include electronic control data, such as motor control data, or more generally, data for controlling one or more electronic devices in the environment, the control of which has an effect on the observed state of the environment. For example, in the case of an autonomous or semi-autonomous land, air, or sea vehicle, actions may include actions for controlling navigation, e.g., steering, and movement, e.g., braking and / or acceleration, of the vehicle.

[0048] In some implementations, the environment is a simulated environment and the agent is implemented as one or more computers that interact with the simulated environment.

[0049] For example, the simulated environment may be a simulation of a robot or vehicle, and the action selection network may be trained on the simulation. For example, the simulated environment may be a motion simulation environment, such as a driving simulation or a flight simulation, and the agent may be a simulated vehicle that navigates through the motion simulation. In these implementations, the actions may be control inputs for controlling the simulated user or the simulated vehicle.

[0050] In another example, the simulated environment may be a video game and the agent may be a simulated user playing the video game.

[0051] In a further example, the environment may be a protein-folding environment, whereby each state is a respective state of a protein chain, and the agent is a computer system for determining how to fold the protein chain. In this example, the actions are possible folding actions for folding the protein chain, and the results to be achieved may include, for example, folding the protein so that it is stable and performs a particular biological function. As another example, the agent may be a machine agent that performs or controls protein-folding actions selected by the system automatically without human interaction. The observations may include direct or indirect observations of protein states and / or may be derived from simulations.

[0052] Generally, in the case of a simulated environment, the observations may include simulated versions of one or more of the previously described observations or observation types, and the actions may include simulated versions of one or more of the previously described actions or action types.

[0053] Training an agent in a simulated environment may enable the agent to learn from large amounts of simulated training data while avoiding the hazards associated with training an agent in a real-world environment, such as harm to the agent from performing an incorrectly chosen action. An agent trained in a simulated environment may then be deployed in a real-world environment. That is, action selection system 100 may be trained with experience tuples representing the agent's interactions with the simulated environment. After being trained with experience tuples representing the agent's interactions with the simulated environment, action selection system 100 may be used to control a real-world agent interacting with the real-world environment.

[0054] In some other applications, agents can control actions in a real-world environment that includes pieces of equipment, e.g., in a data center, in a grid-drawn power / or water distribution system, or in a manufacturing or service facility. In that case, observations can relate to the operation of the facility. For example, observations can include observations of power or water usage by the equipment, or observations of power generation or distribution control, or observations of resource usage or waste production. The agents can control actions in the environment to increase efficiency, e.g., by reducing resource usage, and / or to reduce the environmental impact of operations in the environment, e.g., by reducing waste. Actions can include actions that control or impose conditions on operating conditions on pieces of equipment of the facility / equipment and / or actions that result in changes to the operating settings of the facility / equipment, e.g., to adjust or turn on / off components of the facility / equipment.

[0055] Optionally, in any of the above implementations, the observations at any given time step may include data from previous time steps that may be useful in characterizing the environment, for example, actions performed in the previous time steps.

[0056] The action selection system 100 selects an action 112 to be taken by an agent 114 in an environment 116 at each time step by processing the current observations 120 for the time step using a Q neural network 104 and a policy neural network 106.

[0057] The Q neural network 104 processes inputs including the current observation 120 to generate a respective Q value 108 for each action in a set of possible actions that can be performed by the agent. The Q value for a given action is an estimate of the cumulative measure of reward (e.g., the time-discounted sum of rewards) that would be received over a sequence of time steps if the agent started in the state represented by the current observation and performed the given action in response to the current observation.

[0058] The policy network 106 processes the current observations 120 to generate a set of action scores 110. The policy network 106 can generate a respective action score for each action in the set of possible actions.

[0059] The action selection system 100 selects an action to be performed by an agent at a time step according to the Q-values ​​108 and the action scores 110. For example, the action selection system 100 may combine the Q-values ​​108 and the action scores 110 to generate a respective "final" action score for each possible action and then use the final action scores to select an action to be performed by the agent. An exemplary process for selecting an action to be performed by an agent using the Q-neural network 104 and the policy neural network 106 is described in more detail with reference to FIG. 7.

[0060] The action selection system 100 may be trained, for example, by a training system 200. The training system 200 may train the action selection system 100 using a replay buffer 102 that stores training data. The training data stored in the replay buffer 102 may be, for example, experience tuples that characterize an interaction between an agent and an environment. Each experience tuple in the replay buffer 102 may include a first observation that characterizes an initial state of the environment, an action (e.g., action 112) taken by the agent to interact with the environment, a second observation (e.g., observation 120) that characterizes the state of the environment after the action is taken by the agent, and a corresponding reward (e.g., reward 118).

[0061] The training system 200 can train the action selection system 100 by updating the network parameter values ​​of the Q-neural network 104, and optionally, the policy neural network 106, in each of a number of iterations. The training system 200 can update the network parameter values ​​of the Q-neural network and the policy neural network in each iteration by sampling batches of experience tuples from the replay buffer 102 and training the Q-neural network and the policy neural network with the sampled experience tuples. An exemplary process for training an action selection system is described in more detail with reference to FIG. 2.

[0062] 2 illustrates an exemplary training system 200. Training system 200 is an example of a system in which the systems, components, and techniques described below are implemented as computer programs on one or more computers at one or more locations.

[0063] The training system 200 trains the Q-network 106 and, optionally, the policy neural network 104 used to control an agent that interacts in an environment to perform a task. The training system 200 trains the Q-network 106 and the policy neural network 104 of the action selection system by updating the current values ​​of the Q-network parameters and the policy neural network parameters in each series of iterations.

[0064] At each iteration, the training system 200 samples a batch (set) of experience tuples 204 from the replay buffer 102. Each experience tuple includes a first observation of the environment, an action taken by the agent in response to the first observation of the environment, a second observation of the environment after the action is taken, and a reward for taking the action. For example, the system may randomly sample a batch of experience tuples from the replay buffer 102.

[0065] At each iteration, and for each experience tuple in the batch of experience tuples 204, the training system 200 uses the policy network 106 to process the first observation from the experience tuple to generate a respective set of action scores 110 for each action in the set of possible actions that can be taken by the agent.

[0066] At each iteration, and for each experience tuple in the batch of experience tuples 204, training system 200 samples multiple actions from the set of possible actions according to the action score. Training system 200 may use sampling engine 212 to sample M actions according to the action score for each first observation, where M>2 is a positive integer. For example, sampling engine 212 may process the action scores (e.g., using a softmax function) to obtain a probability distribution over the set of possible actions, and then individually sample M actions from the probability distribution over the set of possible actions.

[0067] At each iteration, and for each experience tuple in the batch of experience tuples 204, the training system 200 processes the second observation using the Q-network 106 to generate a respective Q-value for each sampled action, e.g., Q-value 220. For example, the Q-network 106 can process the second observation from the experience tuple to generate a respective Q-value for each action in the set of possible actions, and the Q-value corresponding to the sampled action can be determined from the generated Q-values.

[0068] At each iteration, and for each experience tuple in the batch of experience tuples 204, training system 200 uses state value engine 222 to generate a respective state value for the second observation. The state value for a state may represent, for example, an estimate of a cumulative measure of reward (e.g., a time-discounted sum of rewards) that would be received by the agent over successive steps if the agent were to start in that state. State value engine 222 may process each Q-value to determine a respective state value for the second observation. For example, state value engine 222 may determine the state value as a linear combination of the Q-values, as discussed in further detail below with reference to FIG. 3.

[0069] At each iteration, the training system 200 uses an update engine 226 to update the current values ​​of the Q-network parameters 228 of the Q-network 106. The update engine 226 processes the state values ​​224 to generate updates to the current values ​​of the Q-network parameters 228, for example, using the gradient of a Q-objective function that depends on the state values ​​224. An exemplary process for updating the current values ​​of the Q-network parameters is described in more detail with reference to FIG.

[0070] Optionally, at each iteration, the training system 200 can update the current values ​​of the policy neural network. For example, the training system can update the policy neural network parameter values ​​using the gradient of the policy objective function, e.g., as described in Figure 6. Updating the values ​​of the policy neural network parameters can facilitate and regularize the training of the Q-neural network.

[0071] 3 is a flow diagram of an exemplary process for training an action selection system using relative entropy Q-learning. For convenience, process 300 is described as being performed by a system of one or more computers at one or more locations. For example, a training system suitably programmed in accordance with this specification, such as training system 200 of FIG. 2, can perform process 300.

[0072] Prior to training, the system may initialize the network parameter values ​​of the policy neural network and the Q neural network in any suitable manner. For example, the system may randomly initialize the network parameter values ​​of the policy neural network and the Q neural network.

[0073] At each iteration (i.e., of multiple training iterations), the system retrieves a batch of experience tuples from the replay buffer, each experience tuple including (1) a first observation, (2) an action performed by the agent, (3) a second observation obtained, and (4) a corresponding reward (302). For example, the system can randomly sample the batch of experience tuples to provide a representative sampling of the replay buffer across multiple iterations. Each experience tuple in the replay buffer can represent a previous interaction of the agent with the environment when the agent was controlled by an action selection system or when the agent was controlled by an expert action selection strategy, e.g., as described with reference to FIG. 7.

[0074] At each iteration and for each experience tuple, the system generates action scores by processing the first observation in the experience tuple using the policy neural network (304). The system generates a respective action score for each action in the possible set of actions given by the first observation s. In some implementations, the system can maintain a set of target policy neural network parameters and a set of current policy network parameters. The system can generate these action scores using the target policy neural network values. The system can update the current policy neural network parameter values ​​at each iteration, as discussed in more detail with reference to FIG. 5. The system can update the target policy neural network parameters to the current policy neural network parameter values ​​every U iterations, where U is a positive integer. Maintaining a separate set of target policy network parameters whose values ​​are periodically updated to the current policy neural network values ​​can stabilize and regularize training.

[0075] The policy neural network may have any suitable neural network architecture that enables it to perform the described function, i.e., process observations of the environment and generate a respective action score for each action in the set of possible actions. In particular, the policy neural network may include any suitable type of neural network layers (e.g., fully connected layers, attention layers, convolutional layers, etc.) in any suitable number (e.g., 1 layer, 5 layers, or 25 layers) and connected in any suitable configuration (e.g., as a linear sequence of layers). In a particular example, the policy neural network may include a sequence of convolutional neural network layers followed by a fully connected layer, the fully connected layer including a respective neuron corresponding to each action in the set of possible actions.

[0076] In each iteration, and for each experience tuple, the system samples multiple actions according to the action score (306). The system may sample M actions for the experience tuple according to the action score, where M>2 is a positive integer. For example, the system may process the action scores (e.g., with a softmax function) to obtain a probability distribution over the set of possible actions, and then individually sample M actions from the probability distribution over the set of possible actions.

[0077] At each iteration and for each experience tuple, the system generates a respective Q-value for each sampled action by processing a second observation of the state of the environment (308). The system can generate each Q-value using a Q-neural network. The system can process the second observation of the state to generate respective action scores for a set of possible actions and match the sampled action with the appropriate action score generated using the Q-network. In some implementations, the system can maintain a set of target Q-neural network parameters and a set of current Q-neural network parameters. The system can generate these Q-values ​​using the target Q-neural network values. The system can update the current Q-neural network parameter values ​​at each iteration, as discussed in more detail with reference to FIG. 4. The system can update the target Q-neural network parameter values ​​to the current Q-neural network parameter values ​​every V iterations, where V is a positive integer. Maintaining a separate set of target Q-neural network parameters whose values ​​are periodically updated to the current Q-neural network values ​​can stabilize and regularize training.

[0078] The Q-neural network may have any suitable neural network architecture that enables it to perform the described function, i.e., process observations of the state of the environment to generate a respective Q-value for each action in the set of possible actions. In particular, the Q-neural network may include any suitable type of neural network layer (e.g., fully connected layers, attention layers, convolutional layers, etc.) in any suitable number (e.g., 1 layer, 5 layers, or 25 layers) and connected in any suitable configuration (e.g., as a linear sequence of layers). In a particular example, the Q-neural network may include a sequence of convolutional neural network layers followed by a fully connected layer, the fully connected layer including a respective neuron corresponding to each action in the set of possible actions.

[0079] At each iteration and for each experience tuple, the system determines 310 state values ​​for the second observations using the corresponding Q values. The system can determine each state value for the second observations using a state value function. For example, the state value function can determine each state value for the second observations as a linear combination of the corresponding Q values, as discussed in more detail with reference to FIG. 4.

[0080] At each iteration, the system uses the state values ​​to determine 312 updates to the current Q-network parameter values. The system can generate updates to the current Q-network parameter values ​​by determining the gradient of a Q-objective function. For example, for each experience tuple, the Q-objective function can measure the error between the Q-values ​​of the actions in the experience tuple and the target Q-values ​​based on the state values, as discussed in more detail in FIG. 5.

[0081] Optionally, at each iteration, the system can determine updates to the current policy neural network parameter values ​​(314). Updating the policy neural network parameters may allow the system to better train and regularize the Q neural network parameter values. For example, the system can determine the updates using the gradient of the policy objective function, as discussed in more detail in FIG. 6.

[0082] At each iteration, the system determines whether the termination criteria for the training are met (316). If the termination criteria are not met, the training loops back to step 302. For example, the termination criteria may include the system performing a predefined number of iterations. Once the system has performed the predefined number of iterations, the system can terminate the training.

[0083] If the system determines that the termination criteria are met, the system terminates the training loop (318).

[0084] 4 is a flow diagram of an exemplary process for determining a state value of a second observation in an experience tuple. For convenience, process 400 is described as being performed by a system of one or more computers at one or more locations. For example, a state value engine suitably programmed in accordance with this specification, such as state value engine 222 of FIG. 2, may perform process 400.

[0085] The system performs process 400 during each training iteration, for example, during step 310 of the training process described in FIG.

[0086] The system receives 402 a Q-value for each sampled action corresponding to the particular experience tuple that includes the second observation. The Q-values ​​may be generated using, for example, a Q-network, as described in step 308 of FIG. 3.

[0087] The system generates a temperature factor based on the Q value (404). The temperature factor may be initially initialized and then updated using the gradient of the (Lagrangian) dual function of the temperature factor. For example, the temperature factor may be

number

number

[0088] The system generates a respective weighting factor for each Q-factor based on the Q-factor and the temperature coefficient (406). Each weighting factor may be a function of the Q-factor and the respective temperature coefficient. For example, for a sampled action a j The weighting coefficients for

number

[0089] The system determines the state value as a linear combination of the Q values ​​using the respective weighting coefficients (408). For example, determining the state value may be

number

[0090] 5 is a flow diagram of an exemplary process for updating current values ​​of Q-network parameters. For convenience, process 500 is described as being performed by a system of one or more computers at one or more locations. For example, an update engine suitably programmed in accordance with this specification, such as update engine 226 of FIG. 2, may perform process 500.

[0091] The system performs process 500 in each training iteration, for example, during step 312 of Figure 3. For convenience, the system is said to perform steps 502-508 in the "current" iteration.

[0092] The system receives 502 a batch of experience tuples and respective state values ​​for second observations in the experience tuples for the current iteration, each of which includes a first observation of the environment, an action taken by the agent, a second observation of the environment after the action is taken, and a corresponding reward.

[0093] For each experience tuple, the system uses a Q-network to process the first observation in the experience tuple to generate a Q-value for the action in the experience tuple (504).

[0094] For each experience tuple, the system determines a target Q value for the action in the experience tuple using the state value for the second observation (506). The system can determine the target Q value based on the state value for the second observation and the reward. For example, the system can determine the target Q value as Q*(r,s')=r+γV π (s') (5) where r represents the reward, s' represents the second observation, γ is a discount factor (e.g., expressed as a positive floating-point value, e.g., less than 1), and V π (s') represents the state value of the second observation.

[0095] For each experience tuple, the system updates 508 the current values ​​of a set of Q-network parameters using the gradient of a Q-objective function that measures the error between (1) the Q-value for the action in the experience tuple and (2) a target Q-value for the action in the experience tuple. The system can update the current values ​​of the Q-network parameters using any suitable method, such as stochastic gradient descent with or without momentum, or ADAM. For example, the system can:

number

[0096] In some implementations, the system maintains a set of target Q-neural network parameter values ​​and a set of current Q-neural network parameter values. The system can generate a Q-value using the current set of Q-neural network parameter values ​​and a target Q-neural network value using the target Q-neural network value. The system can update the current Q-network parameter values ​​at each iteration using Equation 6. Every U iterations, where U is a positive integer, the system can update the target Q-network parameter values ​​to be equal to the current Q-network parameter values. Maintaining clear target and current Q-network parameter values ​​can regularize and stabilize training.

[0097] 6 is a flow diagram of an exemplary process for updating current values ​​of strategy network parameters. For convenience, process 600 is described as being performed by a system of one or more computers at one or more locations.

[0098] The system optionally performs process 600 in each training iteration, for example, during step 314 of Figure 3. For convenience, the system is said to perform steps 602-610 in the "current" iteration.

[0099] The system receives a batch of experience tuples for the current iteration (602). Each experience tuple may include a first observation of a state of the environment, an action taken by the agent in response to the first observation, a second observation of the state of the environment, and a corresponding reward.

[0100] For each experience tuple, the system processes the first observation in the experience tuple using a Q neural network to generate a Q value for the action in the experience tuple (604).

[0101] For each experience tuple, the system determines a state value for the first observation 606. The system can determine the state value for the first observation as a linear combination of the corresponding Q values, for example, using the method of Equations 3 and 4.

[0102] For each experience tuple, the system generates an advantage value based on the Q value and the state value (608). The system may generate an advantage value for an experience tuple as, for example, A π (a,s)=Q φ' (a,s)-V π (a,s) (7) where a represents the action, s represents the first observation, π represents the policy network, and A π (a,s) represents the advantage value, and Q φ' (a,s) represents the Q value generated using the target network parameters φ', and V π (a,s) represents the state value.

[0103] The system updates 610 the current values ​​of the policy network parameters using the gradient of the policy objective function based on the advantage values. The policy objective function may be based only on non-negative advantage values. Using only non-negative advantage values ​​may enable the system to update the current policy neural network parameter values ​​using only actions whose values ​​are estimated (e.g., by the advantage value) to be higher than the policy's average. The system may perform the update using any suitable gradient descent method, such as stochastic gradient descent, or ADAM. For example, the policy objective function may be:

number

[0104] In some implementations, the system can maintain a set of goal policy neural network parameters and a set of current policy neural network parameters. The system can update the current policy neural network parameter values ​​at each iteration using equation (8). Every V iterations, the system updates the goal policy neural network parameter values ​​to be the current policy neural network parameter values, where V is a positive integer. Maintaining the individual goal and current policy neural network parameters can regularize and stabilize training.

[0105] 7 is a flow diagram of an exemplary process for using an action selection system to select an action to be performed by an agent to interact with an environment. For convenience, process 700 is described as being performed by a system of one or more computers at one or more locations.

[0106] For convenience, process 700 is described in terms of selecting an action to be performed by an agent in the "current" time step.

[0107] The system receives current observations for the current time step 702. The current observations may include, for example, observations about the positions, velocities, and accelerations of objects in the environment, or about the joint positions, velocities, and accelerations of the joints of a robotic agent.

[0108] The system selects an action to be performed by the agent at the current time step using either an action selection system or an expert action selection policy, particularly a suboptimal expert action selection policy (704). The expert action selection policy may include a learned sequence of actions for performing a portion of a task, for example, for moving a gripper to a location in the environment or for lifting an object in the environment. A suboptimal expert action selection policy may be a policy that, when used to control an agent, can enable the agent to achieve at least partial task success. The system can choose probabilistically between using the action selection system or the suboptimal expert action selection policy, for example, with a predefined probability of choosing each or the other. The action selection system may be, for example, an action selection system trained by the training system 200 of FIG. 2, and a (suboptimal) expert action selection policy may be generated, for example, by orchestrating a waypoint tracking controller for a robotic agent interacting in a real-world environment. In some implementations, the waypoint tracking controller is a controller for controlling the actions of a robot based on a set of waypoints provided to the controller. For example, a waypoint tracking controller may be a trained controller that provides outputs to control a portion of a robot to move along a path defined by a set of one or more waypoints. The waypoints may be specified by a user through a user interface. This may provide a simple and intuitive way for a user to specify a desired behavior without the need for human demonstration or reward shaping. An expert action selection strategy may be generated by orchestrating, i.e., combining, waypoint tracking controllers.For example, a waypoint tracking controller may specify a particular behavior, such as controlling a portion of a robot to move along a path, and an expert action selection strategy may be generated by orchestrating, i.e., combining, the behaviors of multiple waypoint tracking controllers, e.g., to operate sequentially and / or in parallel. Examples of orchestrating waypoint tracking controllers are described in more detail in Rae Jeong et al., "Learning Dexterous Manipulation from Suboptimal Experts," arXiv:2010.08587v2, January 5, 2021. More generally, an expert action selection strategy may be generated by orchestrating, i.e., combining, the behaviors of any type of learned controller, e.g., a controller based on a pre-trained controller-action selection neural network.

[0109] If the system selects an action selection system, the action selection system selects an action (706). The action selection system may be one currently being trained, for example, by the training system 200 of FIG. 2. The action selection system may select an action to be taken by the agent by generating a set of final action scores for each action in the possible set of actions and selecting an action according to the final action scores, e.g., the action with the largest respective final action score. For example, the action selection system may use a policy neural network to process the current observations for the time step to generate a respective action score for each action in the set of possible actions, and a Q neural network to process the current observations for the time step to generate a respective Q value for each action in the set of possible actions. The action selection system may then determine a final action score for each action based on, for example, the action scores for the action and the Q value for the action, where each final action score is

number

[0110] In implementations in which the system maintains a set of target neural network parameters and a set of current neural network parameters for the Q-network, the policy network, or both during training, the action selection system can use the current neural network parameter values ​​for each of the policy and the Q-network.

[0111] If the system selects a suboptimal expert action selection policy, the suboptimal expert action selection policy selects an action 708. For example, the agent may be a robotic agent that interacts in an environment, and the expert action selection policy may be generated by orchestrating a waypoint tracking controller.

[0112] Based on the selected action, the system receives the next observation of the environment and a reward 710. For example, for a robotic agent interacting in an environment, the next observation may be the result of moving each of a set of robotic joints in a different way and seeing what effect that has on the environment.

[0113] The system generates a new experience tuple based on (1) the current observation, (2) the selected action, (3) the next observation, and (4) the reward (712). The system generates a new experience tuple from these four parts and adds the new experience tuple to the replay buffer.

[0114] Intertwining action selection between the action selection system and a suboptimal expert action selection policy can generate mixed policy-on “exploration” experience tuples and policy-off “expert” experience tuples for training data. The system can add these mixed experience tuples to a replay buffer for training. Training the action selection system with mixed training data (e.g., rather than policy-on training data alone) can enable the action selection system to be trained more quickly (e.g., with fewer training iterations) and achieve better performance (e.g., by enabling the agent to perform tasks more effectively). Training the action selection system more quickly can enable the training system to consume fewer computational resources (e.g., memory and computing power) during training than some conventional training systems.

[0115] This specification uses the term "configured" in the context of systems and computer program components. A system of one or more computers configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when operated, causes the system to perform the operation or action. One or more computer programs configured to perform a particular operation or action means that one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0116] Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, or in tangibly embodied computer software or firmware, or in computer hardware including the structures disclosed herein and their structural equivalents, or in a combination of one or more of these. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or controlling the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to an appropriate receiver apparatus for execution by a data processing apparatus.

[0117] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may be or further include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus may optionally include code that creates an execution environment for a computer program, e.g., code comprising processor firmware, a protocol stack, a database management system, an operating system, or any combination of one or more of these.

[0118] A computer program may also be called or described as a program, software, software application, app, module, software module, script, or code, and may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in portions of files that hold other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple cooperating files, e.g., files that store one or more modules, subprograms, or portions of code. A computer program may be deployed to be executed on one computer located at one site or on multiple computers, or may be distributed across multiple sites and interconnected by a data communications network.

[0119] As used herein, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components and installed on one or more computers at one or more locations. In some cases, one or more computers will be dedicated to a particular engine, and in other cases, multiple engines may be installed and running on the same computer or multiple computers.

[0120] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by special purpose logic circuitry, e.g., FPGAs, or ASICs, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0121] A computer suitable for running a computer program may be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Generally, a computer also includes one or more mass storage devices, e.g., magnetic, magneto-optical, or optical disks, for storing data, or is operatively coupled to the mass storage devices to receive data from, transfer data to, or both. However, a computer need not have such devices. Moreover, a computer may be incorporated in another device, e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0122] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0123] To enable user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, e.g., a mouse or trackball, for allowing the user to provide input to the computer. Other types of devices can also be used to enable user interaction; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Furthermore, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's device in response to a request received from the web browser. A computer can also interact with a user by sending text messages or other forms of messages to a personal device, e.g., a smartphone running a messaging application, and receiving a reply message in return from the user.

[0124] A data processing apparatus for implementing machine learning models may also include special purpose hardware accelerator units, for example for handling the common and numerically intensive parts of the machine learning training or production, i.e., inference, workload.

[0125] Machine learning models can be implemented and deployed using machine learning frameworks, for example the TensorFlow framework.

[0126] Embodiments of the subject matter described herein can be implemented in a computing system that includes back-end components, e.g., as data servers, or includes middleware components, e.g., application servers, or includes front-end components, e.g., client computers having graphical user interfaces or web browsers or apps through which users may interact with implementations of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0127] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., HTML pages, to a user device, e.g., for purposes of displaying the data to and receiving user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., a result of user interaction, may be received from the device at the server.

[0128] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Also, some features described herein in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as working in some combination and even initially claimed as such, one or more features from a claim can, in some cases, be deleted from that combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0129] Similarly, although operations may be illustrated in the figures or recited in the claims in a particular order, this should not be understood as requiring such operations to be performed in the particular order illustrated, or sequentially, or that all of the illustrated operations be performed to achieve desired results. In some situations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0130] Specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequence, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0131] 100 Action Selection System, System 102 Replay Buffer 104 Q neural network, policy neural network 106 Policy Neural Networks, Q-Networks 114 Agent 200 Training System 212 Sampling Engine 222 State Value Engine 226 Update Engine

Claims

1. 1. A method executed by one or more data processing devices for training an action selection system, comprising: the action selection system is used to select an action to be performed by an agent that interacts with an environment to perform a task; the action selection system includes a Q neural network and a policy neural network; The method includes, in each of a plurality of iterations: retrieving a batch of experience tuples from a replay buffer that characterize previous interactions of a simulated or real-world version of the agent with the environment, each experience tuple including: (i) a first observation that characterizes a state of the environment; (ii) an action taken by the agent in response to the first observation; (iii) a second observation that characterizes a state of the environment after the agent takes the action in response to the first observation; and (iv) a reward received as a result of the agent taking the action in response to the first observation; for each experience tuple, determining a state value for the second observation in the experience tuple; using the state value for the second observation in the experience tuple to determine an update to the current values ​​of a set of Q-neural network parameters of the Q-neural network. Including, determining a state value for the second observation includes determining the state value for the second observation as a linear combination of Q values ​​for actions sampled from a set of possible actions that the agent can take; determining the state value for the second observation as a linear combination of Q values ​​for the sampled actions; determining a temperature coefficient based on the Q value for the sampled action; and determining each modified Q-factor for each sampled action as a ratio of (i) the Q-factor for the sampled action and (ii) the temperature coefficient; applying a softmax function to the modified Q-values ​​to determine a weight function for each sampled action; determining the state value for the second observation as a linear combination of the Q values ​​for the sampled actions, the Q value for each sampled action being scaled by the weight function for the sampled action; A method carried out by

2. The state value for the second observation is: [Equation 1] In the above formula, V π (s) is the state value for the second observation, j indexes the sampled action, M is the number of sampled actions, and w j is the sampled action a j is the weighting factor for φ' (a j ,s) is the sampled action a j and each weighting coefficient w j teeth, [Equation 2] where k indexes the sampled action and η s The method of claim 1 , wherein: is the temperature coefficient.

3. Determining the temperature coefficient based on the Q-values ​​for the sampled actions includes, in each of one or more optimization iterations: determining a gradient of a dual function with respect to the temperature coefficient, the dual function depending on (i) the temperature coefficient and (ii) the Q value for the sampled action; adjusting the current value of the temperature coefficient using the gradient of the dual function for the temperature coefficient; 3. The method of claim 1 or 2, comprising:

4. The dual function is [Equation 3] It is calculated as follows: g(η s ) is the temperature coefficient η s where |B| denotes the number of experience tuples in the batch of experience tuples, ε is a regularization parameter, j indexes the sampled action, M is the number of sampled actions, and Q φ' (a j ,s) is the sampled action a j The method of claim 3, wherein the Q value is

5. using the state value for the second observation in the experience tuple to determine an update to a current value of a set of Q-neural network parameters of the Q-neural network; For each experience tuple, processing a first observation in the experience tuple using the Q neural network to generate a Q value for the action in the experience tuple; determining a target Q-value for the action in the experience tuple using the state value for the second observation in the experience tuple; For each experience tuple, determining a gradient of a Q-objective function that measures the error between (i) the Q-value for the action in the experience tuple and (ii) the target Q-value for the action in the experience tuple; using the gradient to determine the update to the current value of the set of Q neural network parameters; 5. The method of claim 1, comprising:

6. determining the target Q-value for the action in the experience tuple using the state value for the second observation in the experience tuple, determining the target Q-value as the sum of (i) the reward in the experience tuple and (ii) the product of the state value and a discount factor for the second observation in the experience tuple; and / or 6. The method of claim 5, wherein the error between (i) the Q value for the action in the experience tuple and (ii) the target Q value for the action in the experience tuple comprises a squared error between (i) the Q value for the action in the experience tuple and (ii) the target Q value for the action in the experience tuple.

7. The Q objective function is: [Equation 4] where B represents the batch of experience tuples, |B| is the number of experience tuples in the batch of experience tuples, each (s, a, r, s') is an experience tuple in the batch B of experience tuples, s is the first observation, a is the action, r is the reward, s' is the second observation, γ is a discount factor, and V π (s') is the state value for the second observation in the experience tuple, and Q φ The method of claim 6 , wherein (a, s) is the Q value for the action in the experience tuple.

8. determining, at each of the plurality of iterations, updates to current values ​​of a set of policy neural network parameters of the policy neural network; For each experience tuple, processing the first observation in the experience tuple using the Q neural network to generate a Q value for the action in the experience tuple; determining a state value for the first observation in the experience tuple; determining an advantage value for the experience tuple as the difference between (i) the Q value for the action in the experience tuple and (ii) the state value for the first observation in the experience tuple; and determining updates to the current values ​​of the set of policy neural network parameters of the policy neural network based only on the experience tuples having non-negative advantage values; 8. The method of any one of claims 1 to 7, further comprising:

9. determining the updates to the current values ​​of the set of policy neural network parameters of the policy neural network based only on the experience tuples having non-negative advantage values; determining a gradient of a policy objective function that depends only on the experience tuples that have non-negative advantage values; using the gradients to determine the updates to the current values ​​of the set of policy neural network parameters; 9. The method of claim 8, comprising:

10. for each experience tuple having a non-negative advantage value, the policy objective function depends on an action score for the action in the experience tuple that is generated by processing the first observation in the experience tuple using the policy neural network; The policy objective function is: [Equation 5] where B represents the batch of experience tuples, |B| is the number of experience tuples in the batch of experience tuples, each (s, a, r) ​​is an experience tuple in the batch B of experience tuples, s is the first observation, a is the action, and r is the reward; [Equation 6] is an indicator function, and A π (a,s) is the advantage value for the experience tuple, and π θ 10. The method of claim 9, wherein (a|s) is the action score for the action in the experience tuple generated by processing the first observation in the experience tuple using the policy neural network.

11. In each of one or more of the plurality of iterations, generating a plurality of new experience tuples using the action selection system, an expert action selection strategy, or both; adding the new experience tuple to the replay buffer; further comprising The step of generating a plurality of new experience tuples includes, at each of one or more time steps: receiving a current observation for the time step; selecting an action to be performed by the agent at the time step using the action selection system or the expert action selection strategy; receiving a next observation and reward resulting from the agent's performance of the selected action; generating a new experience tuple including the current observation, the selected action, the next observation, and the reward; Including, 11. The method of claim 1, wherein using the action selection system or the expert action selection strategy to select the action to be performed by the agent at the time step comprises probabilistically choosing between using the action selection system or the expert action selection strategy to select the action to be performed by the agent at the time step.

12. Selecting an action to be performed by the agent at a time step using the action selection system includes: processing the current observations for the time step using the policy neural network to generate a respective action score for each action in a set of possible actions; processing the current observations for the time step using the Q neural network to generate a respective Q value for each action in the set of possible actions; determining a final action score for each action based on (i) the action score for the action, and (ii) the Q-value for the action; selecting the action to be performed by the agent according to the final action score; 12. The method of claim 11, comprising:

13. The final action score for an action is: [Equation 7] where π(a|s) is the action score for the action, Q(s,a) is the Q value for the action, and η s The method of claim 12 , wherein is a temperature parameter.

14. one or more computers; and one or more storage devices communicatively coupled to the one or more computers, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 13.

15. 14. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 13.

Images