Training an action selection system using relative entropy Q-learning

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training an action selection system using reinforcement learning techniques. In one aspect, a method includes, at each of a plurality of iterations: obtaining a batch of experiences, each experience tuple including: a first observation, an action, a second observation, and a reward; for each experience tuple, determining a state value for the second observation, including: processing the first observation using a policy neural network to generate an action score for each action in a set of possible actions; sampling a plurality of actions from the set of possible actions according to the action scores; processing the second observation using a Q neural network to generate a Q value for each sampled action; and determining the state value for the second observation; and using the state value to determine an update to current values of Q neural network parameters.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of the filing date of U.S. Provisional Patent Application No. 63 / 057,826, filed on July 28, 2020, which is incorporated herein by reference in its entirety. Technical Field

[0003] This manual relates to processing data using machine learning models. Background Technology

[0004] Machine learning models receive input and generate outputs, such as predicted outputs, based on the received inputs. Some machine learning models are parametric models and generate outputs based on the received inputs and the values ​​of the model's parameters.

[0005] Some machine learning models are deep models that use multiple layers to generate the output of a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, each of which applies a non-linear transformation to the received input to generate the output. Summary of the Invention

[0006] This specification generally describes a system for implementing a computer program located on one or more computers at one or more locations for training an action selection system for controlling an agent interacting with its environment using reinforcement learning techniques. The reinforcement learning techniques described herein may be referred to as relative entropy Q-learning. The action selection system may include Q-neurons and policy neural networks, as described in more detail below.

[0007] According to a first aspect, a method is provided, executed by one or more data processing devices, for training an action selection system for selecting actions to be performed by an agent interacting with an environment to perform a task. The action selection system includes a Q-neural network and a policy neural network. The method includes, at each of a plurality of iterations, obtaining a batch of experience tuples representing previous interactions between the agent and the environment from a replay buffer. Each experience tuple includes: (i) a first observation representing the state of the environment, (ii) an action performed by the agent in response to the first observation, (iii) a second observation representing the state of the environment after the agent performs the action in response to the first observation, and (iv) a reward received as a result of the agent performing the action in response to the first observation. The method further includes, at each of the multiple iterations: for each empirical tuple, determining the state value of a second observation in the empirical tuple, including: processing the first observation in the empirical tuple using a policy neural network to generate a corresponding action score for each action in a set of possible actions that can be executed by the agent; sampling multiple actions from the set of possible actions based on the action scores; processing the second observation using a Q-neural network to generate a corresponding Q-value for each sampled action; and using the Q-values ​​of the sampled actions to determine the state value of the second observation. The method also includes, at each of the multiple iterations, using the state value of the second observation in the empirical tuple to determine an update to the current values ​​of the set of Q-neural network parameters for the Q-neural network.

[0008] In one implementation, the action selection system is used to select actions to be performed by a mechanical agent (e.g., a robot interacting with a real-world environment to perform a task). For example, the action selection system may be used to process observations relating to the real-world environment, and the selected action may be related to the action to be performed by the mechanical agent. Therefore, the method may also include using an action selection neural network to control the mechanical agent (e.g., a robot) to obtain observations from one or more sensor devices sensing the real-world environment (e.g., from image, distance, or position sensors, or from actuators of the mechanical agent), and using the action selection system to process the observations to select actions to control the mechanical agent to perform a task, thereby performing a task while interacting with the real-world environment.

[0009] In some implementations, determining the state value of the second observation using the Q value of the sampling action for each empirical tuple includes: determining the state value of the second observation as a linear combination of the Q values ​​of the sampling actions.

[0010] In some implementations, determining the state value of the second observation as a linear combination of the Q values ​​of the sampling actions includes: determining a temperature factor based on the Q values ​​of the sampling actions; determining a corresponding modified Q value for each sampling action as (i) the ratio of the Q value of the sampling action to (ii) the temperature factor; applying a softmax function to the modified Q values ​​to determine a weighting factor for each sampling action; and determining the state value of the second observation as a linear combination of the Q values ​​of the sampling actions, wherein the Q value of each sampling action is scaled by the weighting factor of the sampling action.

[0011] In some implementations, the state value of the second observation is calculated as:

[0012]

[0013] in This is the state value observed in the second observation. Index the sampling actions. It is the number of sampling actions. It is a sampling action Weighting factors It is a sampling action The Q value, and each weight factor The calculation is as follows:

[0014]

[0015] in Index the sampling actions, and It is a temperature factor.

[0016] In some implementations, determining the temperature factor based on the Q value of the sampling action includes, at each of one or more optimization iterations: determining the gradient of the dual function with respect to the temperature factor, wherein the dual function depends on: (i) the temperature factor, and (ii) the Q value of the sampling action; and using the gradient of the dual function with respect to the temperature factor to adjust the current value of the temperature factor.

[0017] In some implementations, the dual function is calculated as:

[0018]

[0019] in It is for temperature factor The dual function being evaluated, This indicates the number of empirical tuples in a batch of empirical tuples. It is a regularization parameter. Index the sampling actions. It is the number of sampling actions, and It is a sampling action The Q value.

[0020] In some implementations, using the state values ​​of a second observation in the empirical tuple to determine an update to the current value of the Q-neural network parameter set of the Q-neural network includes: for each empirical tuple: processing a first observation in the empirical tuple using the Q-neural network to generate a Q-value for an action in the empirical tuple; and using the state values ​​of the second observation in the empirical tuple to determine a target Q-value for an action in the empirical tuple; determining a gradient of a Q-objective function, wherein for each empirical tuple, the Q-objective function measures the error between (i) the Q-value of an action in the empirical tuple and (ii) the target Q-value of an action in the empirical tuple; and using the gradient to determine an update to the current value of the Q-neural network parameter set.

[0021] In some implementations, using the state value of a second observation in the experience tuple to determine the target Q value of the action in the experience tuple includes determining the target Q value as the sum of the following two items: (i) the reward in the experience tuple and (ii) the product of the discount factor and the state value of the second observation in the experience tuple.

[0022] In some implementations, the error between (i) the Q value of the action in the empirical set and (ii) the target Q value of the action in the empirical set includes the squared error between (i) the Q value of the action in the empirical set and (ii) the target Q value of the action in the empirical set.

[0023] In some implementations, the Q objective function is calculated as:

[0024]

[0025] in It is the number of experience tuples in the batch of experience tuples, each It is a batch of empirical tuples The empirical tuples in, where It is the first observation. It's an action. It is a reward and This is the second observation. It is a discount factor. It is the state value of the second observation in the empirical tuple, and It is the Q-value of the action in the empirical tuple.

[0026] In some implementations, the method further includes, at each of the plurality of iterations, determining an update to the current value of the policy neural network parameter set of the policy neural network, including: for each empirical tuple: using a Q-neural network to process a first observation in the empirical tuple to generate a Q-value for an action in the empirical tuple; determining a state value for the first observation in the empirical tuple; and determining the dominance value of the empirical tuple as the difference between (i) the Q-value of an action in the empirical tuple and (ii) the state value of the first observation in the empirical tuple; and determining an update to the current value of the policy neural network parameter set of the policy neural network based only on empirical tuples having non-negative dominance values.

[0027] In some implementations, determining the update of the current value of the policy neural network parameter set based solely on empirical tuples with non-negative dominance values ​​includes: determining the gradient of the policy objective function that depends only on empirical tuples with non-negative dominance values; and using the gradient to determine the update of the current value of the policy neural network parameter set.

[0028] In some implementations, for each empirical tuple with a non-negative advantage value, the policy objective function depends on the action score of the action in the empirical tuple generated by processing the first observation in the empirical tuple using a policy neural network.

[0029] In some implementations, the policy objective function is calculated as:

[0030]

[0031] in It is the number of experience tuples in the batch of experience tuples, each It is a batch of empirical tuples The empirical tuples in, where It is the first observation. It is an action and It's a reward. It is an indicator function. It is the advantage value of the empirical tuple, and It is the action score of the action in the empirical tuple generated by processing the first observation in the empirical tuple using a policy neural network.

[0032] In some implementations, the method further includes, at each of one or more iterations in a plurality of iterations: using an action selection system, an expert action selection strategy, or both to generate a plurality of new experience tuples; and adding the new experience tuples to the replay buffer.

[0033] In some implementations, generating multiple new experience tuples includes, at each of one or more time steps: receiving the current observation of the time step; using an action selection system or expert action selection strategy to select an action to be performed by the agent at the time step; receiving the next observation and the reward generated from the agent performing the selected action; and generating a new experience tuple that includes the current observation, the selected action, the next observation, and the reward.

[0034] In some implementations, using an action selection system or an expert action selection strategy to select the action to be performed by the agent at a time step includes randomly selecting between using an action selection system or an expert action selection strategy to select the action to be performed by the agent at a time step.

[0035] In some implementations, using an action selection system to select an action to be performed by the agent at a time step includes: using a policy neural network to process the current observation of the time step to generate a corresponding action score for each action in the set of possible actions; using a Q neural network to process the current observation of the time step to generate a corresponding Q value for each action in the set of possible actions; determining a final action score for each action based on (i) the action score of the action and (ii) the Q value of the action; and selecting an action to be performed by the agent based on the final action score.

[0036] In some implementations, the final action score of the action is calculated as follows:

[0037]

[0038] in It is the action score of the action. It is the Q value of the action, and It is a temperature parameter.

[0039] In some implementations, the agent is a robotic agent that interacts with the real-world environment, and the expert action selection strategy is generated by combining a waypoint tracking controller.

[0040] According to another aspect, a system is provided, comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the techniques described herein.

[0041] According to another aspect, a non-transitory computer storage medium is provided that stores one or more instructions, which, when executed by one or more computers, cause one or more computers to perform the operations of the techniques described herein.

[0042] Specific embodiments of the subject matter described in this specification may be implemented to achieve one or more of the following advantages.

[0043] The training system described in this specification can train an action selection system for controlling an agent interacting with its environment using on-policy and off-policy experience tuples representing previous interactions between the agent and its environment. Specifically, the training system can train the action selection system using mixed training data, which includes on-policy "exploration" experience tuples generated by the action selection system and off-policy "expert" experience tuples generated by expert action selection policies, such as those selecting actions relevant to the task. A particular advantage of the techniques described herein is that they can benefit from suboptimal expert action selection policies—those highly off-policy policies that select actions relevant only to a portion of the task—and can effectively combine these policies with on-policy "exploration" data representing the agent's interactions with its environment obtained through the method. In the context of a robotic agent interacting with a real-world environment, expert action selection policies can be generated by combining (suboptimal) expert action selection policies with a waypoint tracking controller. Training the action selection system on mixed training data (rather than, for example, only on training data with the same policy) allows the action selection system to be trained faster (e.g., with fewer training iterations) and achieve better performance (e.g., by enabling the agent to perform tasks more efficiently). By training the action selection system faster, the training system can consume fewer computational resources (e.g., memory and computing power) during training than some conventional training systems.

[0044] The training system described in this specification can train a Q-neural network on empirical tuples representing the agent's interaction with its environment using state values ​​determined through importance sampling using a policy neural network. For example, to determine the state value of an observation, the training system can use a policy neural network to process a score distribution over prior observations to generate a set of possible actions, sample multiple actions according to the score distribution, and determine the state value by combining the Q-values ​​of the sampled actions. Training the Q-neural network using state values ​​determined through importance sampling using a policy neural network can regularize and accelerate the training of the Q-network, thereby reducing the computational resource consumption of the training system.

[0045] The training system described in this specification can train a policy neural network on tuples of experience representing previous interactions between an agent and its environment, where the agent performs carefully chosen actions (e.g., actions associated with a non-negative advantage value). The advantage value for a given action can characterize the difference between the reward received by performing that action (e.g., a cumulative measure of reward) and the reward received by performing an average (e.g., randomly selected) action. Training a policy neural network on tuples of experience representing effective agent interactions with the environment can accelerate the training of action selection systems (thus reducing computational resource consumption during training) and improve the performance of the trained action selection system.

[0046] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0047] Figure 1 This is a block diagram of an example action selection system.

[0048] Figure 2 This is a block diagram of an example training system.

[0049] Figure 3 This is a flowchart of an example process for training an action selection system using relative entropy Q-learning.

[0050] Figure 4 This is a flowchart of an example process for determining the observed state value.

[0051] Figure 5 This is a flowchart of an example process for updating the current values ​​of Q network parameters.

[0052] Figure 6 This is a flowchart of an example process for updating the current values ​​of policy network parameters.

[0053] Figure 7 This is a flowchart illustrating an example process for using an action selection system to select actions to be performed by an agent to interact with the environment.

[0054] The same reference numerals and names in different figures indicate the same elements. Detailed Implementation

[0055] Figure 1 An example action selection system 100 is shown. Action selection system 100 is an example of a system implemented on one or more computers at one or more locations to implement computer programs in one or more of the systems, components and technologies described below.

[0056] System 100 selects actions 112 to be performed by agent 114 interacting with environment 116 at each of multiple time steps to achieve a goal. At each time step, system 100 receives data characterizing the current state of environment 116, such as an image of environment 116, and selects actions 112 to be performed by agent 114 in response to the received data. The data characterizing the state of environment 116 will be referred to as observation 120 in this specification. At each time step, the state of environment 116 at the time step (as characterized by observation 120) depends on the state of environment 116 at the previous time step and the actions 112 performed by agent 114 at the previous time step.

[0057] At each time step, system 100 may receive a reward 118 based on the current state of environment 116 at that time step and action 112 of agent 114. Typically, reward 118 may be represented as a numerical value. Reward 118 may be based on any event or aspect of environment 116. For example, reward 108 may indicate whether agent 114 has achieved a goal (e.g., navigating to a target location in environment 116 or completing a task) or the progress agent 114 has made toward achieving a goal (e.g., a task).

[0058] In some implementations, the environment is a real-world environment, and the agent is a mechanical agent that interacts with the real-world environment (e.g., moves within the real-world environment (by translation and / or rotation within the environment, and / or changes its configuration) and / or modifies the real-world environment). For example, the agent may be a robot that interacts with the environment (e.g., to locate an object of interest in the environment, move the object of interest to a designated location in the environment, physically manipulate the object of interest in the environment, or navigate to a designated destination in the environment); or the agent may be an autonomous or semi-autonomous land, air, or sea vehicle that navigates through the environment to a designated destination in the environment.

[0059] In these implementations, observation may include one or more of, for example, images, object location data, and sensor data (e.g., sensor data from images, distance or position sensors, or from actuators) that captures observations as the agent interacts with the environment.

[0060] For example, in the case of a robot, observations may include data characterizing the robot’s current state, such as one or more of the following: joint positions, joint velocities, joint forces, torques or accelerations (e.g., gravity-compensated torque feedback), and the overall or relative posture of the object held by the robot.

[0061] In the case of robots or other mechanical intelligent agents or vehicles, observation can similarly include the position, linear or angular velocity, force, torque or acceleration of one or more parts of the agent, and one or more of its overall or relative attitude. Observation can be defined in 1, 2, or 3 dimensions and can be absolute and / or relative.

[0062] Observations may also include, for example, data obtained by one or more sensor devices that sense the real-world environment; for example, sensed electronic signals, such as motor current or temperature signals; and / or image or video data, such as data from a camera or LIDAR sensor, or data from sensors in the environment that are placed separately from the agent.

[0063] In the case of electronic intelligent agents, observation may include data from one or more sensors and other sensors that monitor a part of a plant or service facility (such as current, voltage, power, temperature) and / or electronic signals that indicate the function of the electronic and / or mechanical items of the equipment.

[0064] Actions can be control inputs for controlling a robot, such as the torque of the robot's joints or higher-level control commands, or control inputs for controlling an autonomous or semi-autonomous land, air, or sea vehicle, such as the torque of the vehicle's control surfaces or other control elements or higher-level control commands.

[0065] In other words, actions can include, for example, position, velocity, or force / torque / acceleration data of one or more joints of a robot or a part of another mechanical agent. Actions may additionally or alternatively include electronic control data, such as motor control data, or more generally, data for controlling one or more electronic devices within an environment, the control of which affects the observed state of the environment. For example, in the case of autonomous or semi-autonomous land, air, or sea vehicles, actions can include actions that control the vehicle's navigation (e.g., steering) and movement (e.g., braking and / or acceleration).

[0066] In some implementations, the environment is a simulated environment, and the agent is implemented as one or more computers that interact with the simulated environment.

[0067] For example, the simulation environment can be a simulation of a robot or vehicle, and the action selection network can be trained on the simulation. For example, the simulation environment can be a motion simulation environment, such as a driving simulation or a flight simulation, and the agent can be a simulated vehicle navigating through the motion simulation. In these implementations, the action can be a control input that controls a simulated user or a simulated vehicle.

[0068] In another example, the simulated environment could be a video game, and the agent could be a simulated user playing the video game.

[0069] In yet another example, the environment can be a protein folding environment, such that each state is a corresponding state of the protein chain, and the agent is a computer system for determining how to fold the protein chain. In this example, actions are possible folding actions for folding the protein chain, and the desired outcome can include, for example, folding the protein so that it is stable and performs a specific biological function. As another example, the agent can be a mechanical agent that performs or controls protein folding actions autonomously selected by the system without human interaction. Observations can include direct or indirect observations of the protein's state and / or can be derived from simulations.

[0070] Typically, in a simulated environment, observations may include simulated versions of one or more of the previously described observations or observation types, and actions may include simulated versions of one or more of the previously described actions or action types.

[0071] Training an agent in a simulated environment allows it to learn from a large amount of simulated training data while avoiding the risks associated with training an agent in a real-world environment, such as harm to the agent due to performing inappropriately chosen actions. An agent trained in a simulated environment can then be deployed in a real-world environment. That is, the action selection system 100 can be trained on tuples representing interactions between the agent and the simulated environment. After being trained on these tuples, the action selection system 100 can be used to control a real-world agent interacting with a real-world environment.

[0072] In some other applications, intelligent agents can control actions in a real-world environment comprising multiple pieces of equipment, such as in a data center or a power grid trunk distribution or water distribution system, or in a manufacturing plant or service facility. Observations can then be related to the operation of the plant facility. For example, observations may include observations of the equipment's power or water usage, or observations of power generation or distribution control, or observations of resource usage or waste generation. Intelligent agents can control actions in the environment to improve efficiency (e.g., by reducing resource usage) and / or (e.g., by reducing waste) reduce the environmental impact of operations in the environment. Actions may include actions that control or impose operating conditions on multiple pieces of equipment in the plant / facility, and / or actions that result in changes to settings in the operation of the plant / facility (e.g., adjusting or turning on / off components of the plant / facility).

[0073] Alternatively, in any of the above embodiments, observations at any given time step may include data from previous time steps that may be helpful in characterizing the environment, such as actions performed at previous time steps.

[0074] The action selection system 100 selects the action 112 to be taken by the agent 114 in the environment 116 at each time step by processing the current observation 120 of the time step using a Q neural network 104 and a policy neural network 106.

[0075] The Q-neural network 104 processes inputs including the current observation 120 to generate a corresponding Q-value 108 for each action in the set of possible actions that can be performed by the agent. The Q-value for a given action is an estimate of the cumulative measure (e.g., the sum of time-discounted rewards) of the rewards that the agent would receive over a sequence of time steps if it started in the state represented by the current observation and performed the given action in response to the current observation.

[0076] Policy network 106 processes the current observation 120 to generate a set of action scores 110. Policy network 106 can generate a corresponding action score for each action in the set of possible actions.

[0077] Action selection system 100 selects the action to be performed by the agent at a time step based on a Q-value 108 and an action score 110. For example, action selection system 100 can combine the Q-value 108 and the action score 110 to generate a corresponding "final" action score for each possible action, and then use the final action score to select the action to be performed by the agent. (Reference) Figure 7 An example process for selecting an action to be performed by an agent using a Q-neural network 104 and a policy neural network 106 is described in more detail.

[0078] Action selection system 100 can be trained, for example, by training system 200. Training system 200 can use replay buffer 102, which stores training data, to train action selection system 100. The training data stored in replay buffer 102 can be, for example, experience tuples representing the agent's interaction with the environment. Each experience tuple in replay buffer 102 can include a first observation representing the initial state of the environment, an action taken by the agent to interact with the environment (e.g., action 112), a second observation representing the state of the environment after the agent has taken the action (e.g., observation 120), and a corresponding reward (e.g., reward 118).

[0079] Training system 200 can train action selection system 100 by updating the network parameter values ​​of Q-neural network 104 and optionally policy neural network 106 at each of multiple iterations. Training system 200 can update the network parameter values ​​of Q-neural network and policy neural network at each iteration by sampling batches of empirical tuples from replay buffer 102 and training Q-neural network and policy neural network on the sampled empirical tuples. Reference Figure 2 An example process for training an action selection system is described in more detail.

[0080] Figure 2 An example training system 200 is shown. The training system 200 is an example of a system implemented on one or more computers at one or more locations to implement computer programs of the systems, components and technologies described below.

[0081] Training system 200 trains a Q-network 104 and optionally a policy neural network 106 for controlling agents interacting in an environment to perform tasks. Training system 200 trains the Q-network 104 and policy neural network 106 of the action selection system by updating the current values ​​of the Q-network parameters and policy neural network parameters at each point in a series of iterations.

[0082] At each iteration, the training system 200 samples batches of experience tuples 204 (a set of experience tuples 204) from the replay buffer 102. Each experience tuple includes a first observation of the environment, an action taken by the agent in response to the first observation of the environment, a second observation of the environment after the action has been taken, and a reward for taking the action. For example, the system may randomly sample batches of experience tuples from the replay buffer 102.

[0083] At each iteration, for each experience tuple in the batch of experience tuples 204, the training system 200 uses the policy network 106 to process the first observation from the experience tuple to generate a corresponding set of action scores 210 for each action in the set of possible actions that the agent can take.

[0084] At each iteration, for each empirical tuple in the batch empirical tuple 204, the training system 200 samples multiple actions from the set of possible actions based on action scores. The training system 200 can use sampling engine 212 to sample M actions based on the action score of each first observation, where M>2 is a positive integer. For example, sampling engine 212 can process the action scores (e.g., using a soft-max function) to obtain a probability distribution over the set of possible actions, and then independently sample M actions from the probability distribution over the set of possible actions.

[0085] At each iteration, for each empirical tuple in the batch empirical tuple 204, the training system 200 uses the Q network 104 to process a second observation to generate a corresponding Q value, such as Q value 220, for each sampled action. For example, the Q network 104 may process the second observation from the empirical tuple to generate a corresponding Q value for each action in the set of possible actions, and the Q value corresponding to the sampled action can be determined from the generated Q value.

[0086] At each iteration, for each experience tuple in the batch of experience tuples 204, the training system 200 uses the state value engine 222 to generate a corresponding state value for the second observation. The state value can represent, for example, an estimate of the cumulative measure of rewards received by the agent over consecutive steps (e.g., the sum of time-discounted rewards) if the agent were initiated in that state. The state value engine 222 can process the corresponding Q-values ​​to determine the corresponding state value for the second observation. For example, the state value engine 222 can determine the state value as a linear combination of Q-values, as referenced below. Figure 3 Further detailed discussion is needed.

[0087] At each iteration, the training system 200 uses the update engine 226 to update the current values ​​of the Q-network parameters 228 of the Q-network 104. The update engine 226 processes the state value 224 to generate updates to the current values ​​of the Q-network parameters 228, for example, using the gradient of the Q-objective function that depends on the state value 224. (See reference) Figure 4 A sample procedure for updating the current values ​​of Q network parameters is described in more detail.

[0088] Optionally, at each iteration, the training system 200 can update the current values ​​of the policy neural network. For example, the training system can use the gradient of the policy objective function to update the parameter values ​​of the policy neural network, such as... Figure 6 As described, updating the values ​​of the policy neural network parameters can facilitate and regularize the training of the Q-neuron.

[0089] Figure 3 This is a flowchart of an example process for training an action selection system using relative entropy Q-learning. For convenience, process 200 will be described as being executed by a system of one or more computers located in one or more locations. For example, a training system appropriately programmed according to this specification (e.g., Figure 1 The training system 100 can execute process 200.

[0090] Before training, the system can initialize the network parameter values ​​of the policy neural network and the Q neural network in any appropriate manner. For example, the system can randomly initialize the network parameter values ​​of the policy neural network and the Q neural network.

[0091] At each iteration (i.e., across multiple training iterations), the system obtains a batch of experience tuples from the replay buffer. Each experience tuple includes (1) a first observation, (2) an action performed by the agent, (3) a second observation resulting from it, and (4) the corresponding reward (302). For example, the system may randomly sample the batch of experience tuples to provide a representative sample of the replay buffer across multiple iterations. Each experience tuple in the replay buffer may represent a previous interaction between the agent and the environment when the agent is controlled by an action selection system or when the agent is controlled by an expert action selection policy, for example, as referenced. Figure 7 As described.

[0092] At each iteration, for each empirical tuple, the system generates an action score (304) by processing the first observation in the empirical tuple using a policy neural network. Given a first observation s, the system generates a corresponding action score for each action in the set of possible actions. In some implementations, the system may maintain a target policy neural network parameter set and a current policy network parameter set. The system can use the target policy neural network values ​​to generate these action scores. The system may update the current policy neural network parameter values ​​at each iteration, as referenced... Figure 5 This will be discussed in further detail. The system can update the parameters of the target policy neural network to the current policy neural network parameter values ​​every U iterations, where U is a positive integer. Maintaining a separate set of target policy network parameters whose values ​​are periodically updated to the current policy neural network values ​​allows for stable and regularized training.

[0093] A policy neural network can have any suitable neural network architecture that enables it to perform the function it describes (i.e., process observations of the environment to generate a corresponding action score for each action in the set of possible actions). Specifically, a policy neural network can include any suitable type of neural network layers (e.g., fully connected layers, attention layers, convolutional layers, etc.) in any suitable number (e.g., 1 layer, 5 layers, or 25 layers) and be connected in any suitable configuration (e.g., as a linear sequence of layers). In a particular example, a policy neural network may include a sequence of convolutional neural network layers followed by fully connected layers, where the fully connected layers include corresponding neurons for each action in the set of possible actions.

[0094] At each iteration, for each empirical tuple, the system samples multiple actions based on action scores (306). The system can sample M actions from the empirical tuple based on action scores, where M>2 is a positive integer. For example, the system can process the action scores (e.g., using a soft-max function) to obtain a probability distribution over the set of possible actions, and then independently sample M actions from the probability distribution over the set of possible actions.

[0095] At each iteration, for each empirical tuple, the system generates a corresponding Q-value (308) for each sampled action by processing a second observation of the state of the environment. The system can use a Q-neural network to generate the corresponding Q-values. The system can process the second observation of the state to generate corresponding action scores for the set of possible actions and match the sampled actions with the appropriate action scores generated using the Q-neural network. In some implementations, the system can maintain a target Q-neural network parameter set and a current Q-neural network parameter set. The system can use the target Q-neural network values ​​to generate these Q-values. The system can update the current Q-neural network parameter values ​​at each iteration, as referenced... Figure 4 This will be discussed in further detail. The system can update the target Q-neural network parameter values ​​to the current Q-neural network parameter values ​​every V iterations, where V is a positive integer. Maintaining a separate set of target Q-neural network parameters whose values ​​are periodically updated to the current Q-neural network values ​​allows for stable and regularized training.

[0096] Q-neural networks can have any suitable neural network architecture that enables them to perform the function they describe (i.e., to process observations of the state of the environment to generate a corresponding Q-value for each action in the set of possible actions). Specifically, a Q-neural network can include any suitable type of neural network layers (e.g., fully connected layers, attention layers, convolutional layers, etc.) in any suitable number (e.g., 1 layer, 5 layers, or 25 layers) and be connected in any suitable configuration (e.g., as a linear sequence of layers). In a particular example, a Q-neural network may include a sequence of convolutional neural network layers followed by fully connected layers, where the fully connected layers include corresponding neurons for each action in the set of possible actions.

[0097] At each iteration, for each empirical tuple, the system uses the corresponding Q-value to determine the state value of the second observation (310). The system can use a state-value function to determine the corresponding state value of the second observation. For example, the state-value function can determine the corresponding state value of the second observation as a linear combination of the corresponding Q-values, as referenced. Figure 4 Further detailed discussion is needed.

[0098] At each iteration, the system uses the state values ​​to determine the update to the current Q-network parameter values ​​(312). The system can generate the update to the current Q-network parameter values ​​by determining the gradient of the Q-objective function. For example, for each empirical tuple, the Q-objective function can measure the error between the Q-value of the action in the empirical tuple and the target Q-value based on the state values, such as... Figure 5 This will be discussed in further detail later.

[0099] Optionally, at each iteration, the system can determine an update to the current policy neural network parameter values ​​(314). Updating the policy neural network parameters allows the system to better train and regularize the Q-neural network parameter values. For example, the system can use the gradient of the policy objective function to determine the update, such as... Figure 6 This will be discussed in further detail later.

[0100] At each iteration, the system determines whether the training termination criterion (316) has been met. If the termination criterion has not been met, the training loop returns to step 202. For example, the termination criterion could include the system performing a predefined number of iterations. Once the system has performed the predefined number of iterations, the system can terminate the training.

[0101] If the system determines that the termination criterion has been met, the system terminates the training cycle (318).

[0102] Figure 4 This is a flowchart of an example procedure for determining the state value of a second observation in an empirical tuple. For convenience, procedure 400 will be described as being executed by a system of one or more computers located in one or more locations. For example, a state value engine appropriately programmed according to this specification (e.g., Figure 2 The state value engine 222) can execute process 400.

[0103] The system during each training iteration (e.g., in Figure 3 During step 310 of the training process described in the text, process 400 is executed.

[0104] The system receives the corresponding Q-value (402) for the sampling action that includes a specific empirical tuple of the second observation. The Q-value can be generated using a Q-network, for example, such as... Figure 3 As described in step 308.

[0105] The system generates the temperature factor (404) based on the Q-value. The temperature factor can be initialized first and then updated using the gradient of its Langrangian dual function. For example, the temperature factor can be initialized to...

[0106] (1)

[0107] Where |B| is the number of experience tuples in the batch of experience tuples, n is the index of the experience tuples, M is the number of action samples, and m is the index of the action samples. The Q-value represents the state-action pair. The observation representing the state (here, the second observation), a mn Let m represent the sampling action of the nth empirical tuple, and Let represent the standard deviation across M sampling actions. For a given empirical tuple, the system can then use multiple gradient descent steps (e.g., using any suitable gradient descent method, such as ADAM) to optimize the dual function of the temperature factor, such as

[0108] (2)

[0109] in It is a temperature factor The dual function, |B|, represents the number of empirical tuples in the batch of empirical tuples. Let j represent the regularization parameter, j be the index of the sampling action, and M represent the number of sampling actions. It is sampling action a j The Q value.

[0110] The system generates a corresponding weighting factor (406) for each Q value based on the Q value and the temperature factor. The corresponding weighting factor can be a function of the Q value and the corresponding temperature factor. For example, sampling action a... j The corresponding weighting factors can be determined as follows:

[0111] (3)

[0112] Where w j Representing the weighting factors, j and k index the sampling actions. Represents the temperature factor, and Indicates sampling action a j The Q value.

[0113] The system uses appropriate weighting factors to determine the state values ​​as linear combinations of Q values ​​(408). For example, determining the state value can be represented as,

[0114] (4)

[0115] in Indicates that observation s is in the policy The state value below, j indexes the sampling action, w j Represents the weighting factor, and This represents the Q value.

[0116] Figure 5 This is a flowchart of an example procedure for updating the current values ​​of Q network parameters. For convenience, procedure 500 will be described as being executed by a system of one or more computers located in one or more locations. For example, an update engine appropriately programmed according to this specification (e.g., Figure 2 The update engine 226) can execute process 500.

[0117] The system at each training iteration (e.g., at) Figure 3 During step 312, process 500 is executed. For convenience, the system is referred to as executing steps 402-408 at the "current" iteration.

[0118] The system receives the batch of experience tuples for the current iteration and the corresponding state values ​​(502) of the second observations in the experience tuples. Each experience tuple includes a first observation of the environment, an action taken by the agent, a second observation of the environment after the action is taken, and the corresponding reward.

[0119] For each experience tuple, the system uses a Q-network to process the first observation in the experience tuple to generate the Q-value (504) of the action in the experience tuple.

[0120] For each experience tuple, the system uses the state value of the second observation to determine the target Q value of the action in the experience tuple (506). The system can determine the target Q value based on the state value of the second observation and the reward. For example, the system can determine the target Q value as follows:

[0121] (5)

[0122] Where r represents the reward. This indicates the second observation, where γ is a discount factor (e.g., represented as a positive floating-point value, such as less than 1), and This represents the state value of the second observation.

[0123] The system uses the gradient of the Q-objective function to update the current values ​​of the Q-network parameter set, where, for each empirical tuple, the Q-objective function measures the error (508) between (1) the Q-value of the action in the empirical tuple and (2) the target Q-value of the action in the empirical tuple. The system can use any suitable method (such as stochastic gradient descent with or without momentum, or ADAM) to update the current values ​​of the Q-network parameters. For example, the system can use the squared error Q-objective function to update the current Q-network parameter values, which is expressed as:

[0124] (6)

[0125] Where B represents the batch of empirical tuples, and |B| represents the number of empirical tuples in the batch. This indicates a first observation s, an action a, a reward r, and a second observation. The empirical tuple, This represents the target Q value, and The Q-value represents the action.

[0126] In some implementations, the system maintains a target Q-neural network parameter value set and a current Q-neural network parameter value set. The system can generate Q-values ​​using the current set of Q-neural network parameter values ​​and generate target Q-values ​​using the target Q-neural network values. The system can update the current Q-neural network parameter values ​​at each iteration using Equation 6. Every U iterations, the system can update the target Q-neural network parameter values ​​to be equal to the current Q-neural network parameter values, where U is a positive integer. Maintaining different target and current Q-neural network parameter values ​​can regularize and stabilize training.

[0127] Figure 6 This is a flowchart of an example procedure for updating the current values ​​of policy network parameters. For convenience, procedure 600 will be described as being executed by a system located on one or more computers at one or more locations.

[0128] The system can optionally at each training iteration (e.g., at...) Figure 3 During step 314, process 600 is executed. For convenience, the system is referred to as executing steps 602-610 at the “current” iteration.

[0129] The system receives the batch of experience tuples (602) for the current iteration. Each experience tuple may include a first observation of the state of the environment, the action taken by the agent in response to the first observation, a second observation of the state of the environment, and the corresponding reward.

[0130] For each experience tuple, the system uses a Q-neural network to process the first observation in the experience tuple to generate the Q-value (604) of the action in the experience tuple.

[0131] For each empirical tuple, the system determines the state value of the first observation (606). The system can determine the state value of the first observation as a linear combination of the corresponding Q values, for example, using the methods in Equations 3 and 4.

[0132] For each empirical tuple, the system generates an advantage value (608) based on the Q-value and the state value. The system can generate the advantage value of an empirical tuple as the difference between the Q-value and the state value, for example, expressed as...

[0133] (7)

[0134] Where 'a' represents the action, 's' represents the first observation, and 'π' represents the policy network. Indicates the dominance value. Indicates the use of target network parameters The generated Q value, and Indicates the state value.

[0135] The system updates the current values ​​of the policy network parameters using the gradient of the policy objective function based on the advantage value (510). The policy objective function can be based solely on non-negative advantage values. Using only non-negative advantage values ​​allows the system to update the current policy neural network parameter values ​​only using actions whose values ​​(e.g., through the advantage value) are estimated to be higher than the policy average. The system can perform the update using any suitable gradient descent method (such as stochastic gradient descent or ADAM). For example, the policy objective function can be expressed as,

[0136] (8)

[0137] Where B represents the batch of empirical tuples, and |B| represents the number of empirical tuples in the batch. Let represent the experience tuple, where s represents the first observation, a represents the action, and r represents the reward. This indicates an indicator function (e.g., a function that returns 1 if the condition is true or 0 if the condition is false). Indicates the dominance value, and This represents a policy neural network using the current policy network parameters θ. The generated action score.

[0138] In some implementations, the system can maintain a set of parameters for the target policy neural network and a set of parameters for the current policy neural network. The system can update the parameter values ​​of the current policy neural network at each iteration using equation (8). Every V iterations, the system updates the parameter values ​​of the target policy neural network to the parameter values ​​of the current policy neural network, where V is a positive integer. Maintaining separate parameters for the target policy neural network and the current policy neural network allows for regularization and stable training.

[0139] Figure 7 This is a flowchart of an example process for using an action selection system to select actions to be performed by an agent to interact with the environment. For convenience, process 600 will be described as being performed by a system of one or more computers located in one or more locations.

[0140] For convenience, process 700 will be described by referring to the action to be performed by the agent at the “current” time step.

[0141] The system receives the current observation at the current time step (702). The current observation may include, for example, observations about the position, velocity, and acceleration of objects in the environment or about the joint position, velocity, and acceleration of the robot agent's joints.

[0142] The system uses an action selection system or an expert action selection strategy (particularly a suboptimal expert action selection strategy) to select the action to be performed by the agent at the current time step (704). The expert action selection strategy may include a learned sequence of actions for performing a portion of a task (e.g., moving a gripper to a location in the environment, or picking up an object in the environment). The suboptimal expert action selection strategy may be a strategy that enables the agent to achieve at least partial success in a task when used to control the agent. The system may randomly choose between using the action selection system and the suboptimal expert action selection strategy, for example, with a predefined probability of selecting one or the other. The action selection system may be, for example, a system derived from... Figure 1 The training system 100 trains an action selection system, and (suboptimal) expert action selection strategies can be generated, for example, for robotic agents interacting in a real-world environment by combining a waypoint tracking controller. In an implementation, the waypoint tracking controller is a controller for controlling the robot's actions based on a set of waypoints already provided to the controller. For example, the waypoint tracking controller can be a learning controller that provides output for controlling a portion of the robot to move along a path defined by a set of one or more waypoints. Waypoints can have already been specified by the user via a user interface. This provides the user with a simple and intuitive way to specify the desired behavior without requiring human demonstration or reward shaping. Expert action selection strategies can be generated by combining (i.e., combining) the waypoint tracking controllers. For example, the waypoint tracking controllers can specify specific behaviors, such as controlling a portion of the robot to move along a path, and expert action selection strategies can be generated by combining (i.e., combining) the behaviors of multiple waypoint tracking controllers in, for example, sequentially and / or in parallel. See the following example, which describes the incorporation of waypoint tracking controllers, for a more detailed description: Rae Jeong et al., “Learning Dexterous Manipulation from Suboptimal Experts” arXiv:2010.08587v2, January 5, 2021. More generally, expert action selection policies can be generated by incorporating (i.e., combining) the behavior of any type of learned controller, such as a controller based on a previously trained controller action selection neural network.

[0143] If the system selects the action selection system, then the action selection system selects the action (706). The action selection system can be, for example, the one currently selected by... Figure 1The training system 100 trains the system. The action selection system can select the action to be taken by the agent by generating a set of final action scores for each action in the set of possible actions and selecting the action based on the final action score (e.g., the action with the highest corresponding final action score). For example, the action selection system can use a policy neural network to process the current observation at the time step to generate a corresponding action score for each action in the set of possible actions, and use a Q neural network to process the current observation at the time step to generate a corresponding Q value for each action in the set of possible actions. The action selection system can then determine the final action score for each action based on the action score and the Q value of the action, for example, where the corresponding final action score is determined as...

[0144] (9)

[0145] Where 'a' represents the action and 's' represents the current observation. Indicates the score of the action. This represents the Q value, and The temperature factor is represented (e.g., as in Equations 3 and 4). The action selection system can select the action to be performed by the agent based on the final action score (e.g., by selecting the action corresponding to the maximum final action score).

[0146] During training, in implementations where the system maintains a target neural network parameter set and a current neural network parameter set for the Q-network, policy network, or both, the action selection system can use the current neural network parameter values ​​of each of the policy network and the Q-network.

[0147] If the system selects a suboptimal expert action selection strategy, then the suboptimal expert action selection strategy selects an action (708). For example, the agent can be a robotic agent interacting in the environment, and the expert action selection strategy can be generated by combining a waypoint tracking controller.

[0148] The system receives the next observation and reward based on the environment of the selected action (710). For example, for a robotic agent interacting in the environment, the next observation could be the result of moving each of the robot's joint sets in a corresponding manner and seeing what effect it has on the environment.

[0149] The system generates new experience tuples (712) based on (1) the current observation, (2) the selected action, (3) the next observation, and (4) the reward. The system generates new experience tuples from the four parts and adds the new experience tuples to the replay buffer.

[0150] Interleaved action selection between the action selection system and the suboptimal expert action selection policy can generate a mixture of same-policy "exploratory" experience tuples and dissimilar-policy "expert" experience tuples from the training data. The system can add these mixed experience tuples to the replay buffer for training. Using mixed training data (instead of, for example, only on same-policy training data) to train the action selection system allows it to be trained faster (e.g., with fewer training iterations) and achieves better performance (e.g., by enabling the agent to perform tasks more efficiently). By training the action selection system faster, the training system can consume fewer computational resources (e.g., memory and computing power) during training compared to some conventional training systems.

[0151] This specification uses the term "configuration" in conjunction with system and computer program components. For a system of one or more computers configured to perform a particular operation or action, this means that software, firmware, hardware, or a combination thereof are installed on the system, causing the system to perform the operation or action in operation. For one or more computer programs configured to perform a particular operation or action, this means that one or more programs include instructions that, when executed by a data processing device, cause that device to perform the operation or action.

[0152] Embodiments of the subject matter and functional operation described herein may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed herein and their equivalents), or in a combination of one or more of these. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, for use by a data processing apparatus to operate or control the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical signals, optical signals, or electromagnetic signals, wherein the artificially generated propagation signals are generated to encode information for transmission to a suitable receiver device for use by a data processing apparatus.

[0153] The term "data processing apparatus" refers to data processing hardware and includes all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus may also be or include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may optionally include code that creates the runtime environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0154] A computer program, also referred to or described as a program, software, software application, application, module, software module, script, or code, can be written in any programming language, including compiled or interpreted languages, or declarative or procedural languages; it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored as a part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), as a single file dedicated to the program in question, or as multiple coordinating files (e.g., a file storing one or more modules, subroutines, or code portions). A computer program can be deployed to run on a single computer or on multiple computers located at a site or distributed across multiple sites and interconnected via a data communication network.

[0155] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in others, multiple engines may be installed on and run on the same computer(s).

[0156] The processes and logic flows described in this specification can be executed by one or more programmable computers, wherein the one or more programmable computers run one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuitry (e.g., FPGA or ASIC) or by a combination of special-purpose logic circuitry and one or more programmable computers.

[0157] A computer suitable for running computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into special-purpose logic circuitry. Typically, a computer will also include one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or be operatively coupled to receive data from or transfer data to such mass storage devices. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0158] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD ROM and DVD-ROM disks.

[0159] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound input, voice input, or tactile input. Furthermore, the computer can interact with the user by transmitting documents to and receiving documents from a device used by the user; for example, by transmitting a webpage to a web browser on the user's device in response to a request received from a web browser. Additionally, the computer can interact with the user by transmitting text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving response messages from the user in exchange.

[0160] The data processing apparatus used to implement machine learning models may also include, for example, dedicated hardware accelerator units for processing the ordinary and computationally intensive parts (i.e., inference, workload) of machine learning training or production.

[0161] Machine learning models can be implemented and deployed using machine learning frameworks such as TensorFlow.

[0162] Embodiments of the subject matter described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer having a graphical user interface, web browser, or application through which a user can interact with embodiments of the subject matter described herein), or any combination of one or more such back-end components, middleware components, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs) (e.g., the Internet).

[0163] A computing system may include clients and servers. Clients and servers are typically geographically separated and generally interact via a communication network. The client-server relationship arises from computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., HTML pages) to a user device, for example, for the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from that user. Data generated at the user device, such as the results of user interactions, may be received at the server from the device.

[0164] While this specification contains numerous specific details of implementation, these should not be construed as limiting the scope of any invention or the scope of what may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described herein in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from the claimed combination, and the claimed combination may be for sub-combinations or variations thereof.

[0165] Similarly, although operations are depicted in a specific order in the drawings and set forth in the claims, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or to perform all of the shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged as multiple software products.

[0166] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions set forth in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for training an action selection system, the action selection system being used to select actions to be performed by an agent interacting with an environment to perform a task, wherein the action selection system comprises a Q-neural network and a policy neural network, the method comprising, at each of a plurality of iterations: Obtain batches of experience tuples representing previous interactions between the agent and the environment from the replay buffer, wherein each experience tuple includes: (i) a first observation characterizing the state of the environment, (ii) an action performed by the agent in response to the first observation, (iii) a second observation characterizing the state of the environment after the agent performs the action in response to the first observation, and (iv) a reward received as a result of the agent performing the action in response to the first observation, wherein the first and second observations include: an image; or object position data captured by a distance or position sensor when the agent interacts with the environment; or sensor data from an actuator characterizing one or more of the robot's joint position, joint velocity, joint force, torque, acceleration, or the overall or relative posture of an object held by the robot; or sensor data from one or more sensors monitoring current, voltage, power, or temperature in a plant or service facility; or electronic signals representing the function of electronic items of the equipment; or electronic signals representing the function of mechanical items of the equipment; or analog versions of any of the above data; For each experience tuple, the state value of the second observation in the experience tuple is determined by sampling the importance of actions using a policy neural network; and The state value of the second observation in the empirical tuple is used to determine the update of the current value of the Q-neural network parameter set. The method further includes: The state values ​​of the second observation are determined as a linear combination of the Q values ​​of the importance sampling actions through the following steps: The temperature factor is determined based on the Q value of the sampling action; The corresponding modified Q value for each sampling action is determined as (i) the ratio of the Q value of the sampling action to (ii) the temperature factor; The softmax function is applied to the modified Q-value to determine the weight factor for each sampling action; and The state value of the second observation is determined as a linear combination of the Q values ​​of the sampling actions, wherein the Q value of each sampling action is scaled by a weighting factor of the sampling action. The state value of the second observation is calculated as follows: in This is the state value observed in the second observation. Index the sampling actions. It is the number of sampling actions. It is a sampling action Weighting factors It is a sampling action The Q value, and each weight factor Calculated as: in Index the sampling actions, and It is a temperature factor; and The Q objective function is calculated as follows: in It is the number of experience tuples in the batch of experience tuples, each It is a batch of empirical tuples The empirical tuples in, where It is the first observation. It's an action. It is a reward and This is the second observation. It is a discount factor. It is the state value of the second observation in the empirical tuple, and It is the Q-value of the action in the empirical tuple; and Specifically, determining the update of the current value of the Q-neural network parameter set using the state value of the second observation in the empirical tuple includes: for each empirical tuple, determining the gradient of the Q objective function, and using the gradient to determine the update of the current value of the Q-neural network parameter set.

2. The method according to claim 1, wherein, Determining the temperature factor based on the Q-value of the sampling action includes, at each of one or more optimization iterations: Determine the gradient of the dual function with respect to the temperature factor, wherein the dual function depends on: (i) the temperature factor, and (ii) the Q value of the sampling action; The current value of the temperature factor is adjusted using the gradient of the dual function with respect to the temperature factor.

3. The method according to claim 2, wherein, The dual function is calculated as follows: in It is for temperature factor The dual function being evaluated, This indicates the number of empirical tuples in a batch of empirical tuples. It is a regularization parameter. Index the sampling actions. It is the number of sampling actions, and It is a sampling action The Q value.

4. The method of claim 1, further comprising, at each of the plurality of iterations, determining an update to the current value of the policy neural network parameter set of the policy neural network, including: For each empirical tuple: A Q-neural network is used to process the first observation in the empirical tuple to generate the Q-value of the action in the empirical tuple; Determine the state value of the first observation in the empirical tuple; and The advantage value of the empirical tuple is determined as the difference between (i) the Q value of the action in the empirical tuple and (ii) the state value of the first observation in the empirical tuple; as well as The update of the current values ​​of the policy neural network parameter set is determined only based on empirical tuples with non-negative dominance values.

5. The method according to claim 4, wherein, Determining the update of the current values ​​of the policy neural network parameter set based solely on empirical tuples with non-negative dominance values ​​includes: Determine the gradient of the policy objective function that depends only on empirical tuples with non-negative advantage values; and The gradient is used to determine the update of the current values ​​of the policy neural network parameter set.

6. The method according to claim 5, wherein, For each empirical tuple with a non-negative advantage value, the policy objective function depends on the action score of the action in the empirical tuple generated by processing the first observation in the empirical tuple using a policy neural network.

7. The method according to claim 6, wherein, The objective function of the strategy is calculated as follows: in It is the number of experience tuples in the batch of experience tuples, each It is a batch of empirical tuples The empirical tuples in, where It is the first observation. It is an action and It's a reward. It is an indicator function. It is the advantage value of the empirical tuple, and It is the action score of the action in the empirical tuple generated by processing the first observation in the empirical tuple using a policy neural network.

8. The method of claim 1, further comprising, at each of one or more iterations of a plurality of iterations: Use action selection systems, expert action selection strategies, or both to generate multiple new experience tuples; and Add the new experience tuple to the replay buffer.

9. The method according to claim 8, wherein, Generating multiple new experience tuples includes each time step in one or more time steps: Receive the current observation at the time step; Use an action selection system or expert action selection strategy to select the action to be performed by the agent at the time step; Receive the next observation and reward generated from the agent performing the selected action; as well as Generate a new experience tuple that includes the current observation, the selected action, the next observation, and the reward.

10. The method according to claim 9, wherein, Using an action selection system or expert action selection strategy to select the action to be performed by the agent at the time step includes randomly selecting from among the actions to be performed by the agent at the time step.

11. The method according to claim 9, wherein, Using an action selection system to select the action to be performed by the agent at a time step includes: A policy neural network is used to process the current observation at the time step to generate a corresponding action score for each action in the set of possible actions. A Q-neural network is used to process the current observation at the time step to generate a corresponding Q-value for each action in the set of possible actions; The final action score for each action is determined based on (i) the action score and (ii) the Q-value of the action; and The action to be performed by the agent is selected based on the final action score.

12. The method according to claim 11, wherein, The final action score is calculated as follows: in It is the action score of the action. It is the Q value of the action, and It is a temperature parameter.

13. The method of claim 1, wherein the intelligent agent comprises a mechanical intelligent agent, the method further comprising using an action selection system to select an action to be performed by the mechanical intelligent agent when interacting with a real-world environment to perform a task in the real-world environment, wherein the observation is related to the real-world environment.

14. The method of claim 1, wherein the agent controls actions for controlling or applying operating conditions to the device item, the method further comprising using an action selection system to select actions that the agent will perform when interacting with a real-world environment including the manufacturing plant or service facility containing the device item, wherein the observation is related to the operation of the manufacturing plant or service facility.

15. The method according to claim 9, wherein, The agent is a robotic agent that interacts with the real-world environment, and the expert action selection strategy is generated by combining a waypoint tracking controller.

16. A system for training an action selection system, comprising: One or more computers; as well as One or more storage devices are communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of a corresponding method according to any one of claims 1-15.

17. A non-transitory computer storage medium storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations according to any one of claims 1-15.

Images