Systems and methods for selecting actions

Abstract

A system and method for selecting actions are disclosed. The method includes: receiving an observation of the current state of a real-world environment representing interaction between the robot agent and the robot agent performing a robot task; processing the observation using a neural network system to generate a policy output; and using the policy output to select an action to be performed by the robot agent in response to the observation. The neural network system includes a sequence of deep neural networks (DNNs), the sequence of DNNs including a simulated DNN trained on simulation and a first robot-trained DNN. The simulated DNN has been trained on interactions between a simulated version of the robot agent and a simulated version of the real-world environment to perform a simulated version of the robot task. The first robot-trained DNN has been trained on interactions between the robot agent and the real-world environment to perform the robot task, to determine training values ​​for the parameters of the first robot-trained DNN while keeping the training values ​​for the parameters of the simulated DNN fixed.

Description

[0001] This application is a divisional application of the invention patent application filed on October 10, 2017, with application number 201780074261.X and invention title "System, Method and Storage Medium for Selecting Actions in a Neural Network".

[0002] Cross-references to related applications

[0003] This application claims priority to U.S. Provisional Application Serial No. 62 / 406,363, filed October 10, 2016. The disclosure of the earlier application is considered part of the disclosure of this application and is incorporated herein by reference. Technical Field

[0004] This specification relates to the selection of actions to be performed by a reinforcement learning agent. Background Technology

[0005] Reinforcement learning agents interact with the environment by receiving observations that represent the current state of the environment and performing actions in response.

[0006] Some reinforcement learning systems use neural networks to select the action that an agent should perform in response to receiving any given observation.

[0007] Neural networks are machine learning models that use one or more layers of non-linear units to predict the output of a received input. Some neural networks are deep neural networks, which include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input based on the current values ​​of its corresponding set of parameters. Summary of the Invention

[0008] This specification describes a system implemented as a computer program on one or more computers in one or more locations.

[0009] Typically, an innovative aspect can be embodied in a system comprising a neural network system implemented by one or more computers. The neural network system is configured to receive observations of the current state of a real-world environment with which a robotic agent interacts to perform a robotic task, and to process the observations to generate a policy output that defines actions to be performed by the robotic agent in response to the observations. The neural network system includes a sequence of deep neural networks (DNNs). The sequence of DNNs includes a simulated-trained DNN, which has been trained on a simulated version of the interaction between a simulated version of the robotic agent and a simulated version of the real-world environment to perform a simulated version of the robotic task. The simulated-trained DNN includes a first plurality of index layers, and is configured to receive the observations and process them through each of the first plurality of index layers to generate a corresponding layer output for each of the first plurality of index layers. The sequence of DNNs also includes a first robot-trained DNN, which may have already been trained on the interaction between the robotic agent and the real-world environment to perform a robotic task. The first robot-trained DNN includes a second plurality of index layers. The DNN trained by the first robot is configured to receive the observation and process the observation through each of the second plurality of index layers to generate the policy output, and one or more of the layers in the second plurality of index layers are configured to receive the following as input: (i) the layer output generated by the previous layer of the DNN trained by the first robot, and (ii) the layer output generated by the previous layer of the DNN trained by the simulation, wherein the previous layer is a layer whose index is 1 smaller than the index of the layer.

[0010] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. Deep reinforcement learning algorithms are too slow to achieve performance on real-world robots, but can be demonstrated in simulated environments. By using the neural network system described in this specification to bridge the reality gap and transfer learned policies from simulation to the real world, action selection policies for robotic agents can be determined more quickly and efficiently with fewer computational resources than existing methods. In other words, the neural network system described in this specification allows for fast policy learning on real-world robots using deep reinforcement learning.

[0011] In some implementations, simulated DNNs are configured with sufficient capacity and depth to learn the task from scratch, while one or more robot-trained DNNs have smaller capacities, enabling maximum transfer across different tasks and faster learning. The capacity and structure of the simulated and robot-trained DNNs do not need to be identical, thus allowing for the solving of different tasks, including different input modalities. Furthermore, after training, the trained neural network system can have a smaller computational footprint when used for action selection, i.e., using fewer processing resources and less memory, while still efficiently selecting actions to be performed by the robotic agent. Moreover, while existing methods cannot adapt to changing network configurations or new input modalities, the neural network system described in this specification provides the flexibility to add new capacity (including new input connections) when transferring to new tasks through the use of progressive networks. This is also advantageous for bridging the reality gap to accommodate different inputs from simulated and real-world sensors.

[0012] In some implementations, the DNN trained by the first robot has a smaller capacity than the DNN trained in simulation. The capacity of a neural network can be calculated. It broadly relates to the number of parameters in the neural network, so the DNN trained by the first robot can have fewer parameters, such as weights, than the DNN trained in simulation. Empirically, this has been found not to hinder, but rather to incentivize learning while reducing computational load. Similarly, the DNN trained by the first robot can have a narrower architecture than the DNN trained in simulation, for example, fewer nodes in the corresponding layers.

[0013] In some implementations, the layers of both the DNN trained by the first robot and the DNN trained by simulation can include corresponding recurrent neural network layers. This helps to generate policy outputs based on the history of observations.

[0014] The policy output can be constrained to a discrete or continuous policy. For example, the output can provide a value in a continuous action space representing the action to be performed, or it can exist as a value representing the action probability for each possible action in a discrete action space. For example, the former could employ a DDPG DNN, and the latter an A3C DNN (the abbreviation of which is explained below). In the latter case, the output layer of the DNN can include a softmax layer to provide the policy and typically includes a linear output of the relevant value function. Other frameworks can also be used—for example, the policy can be implicit in the output from a Deep Q Network (DQN), such as as a value and advantage function output. Therefore, optionally, the action subsystem can select the action to be performed by the robot agent based on the policy output. This allows the robot agent to perform the selected action. If the robot agent has multiple degrees of freedom to control, multiple policy outputs can exist.

[0015] The system may include a second robot-trained DNN, which is laterally connected to a first robot-trained DNN and a simulation-trained DNN in a manner similar to that previously described. The second robot-trained DNN may receive different data (for observations) that also characterize the current state of the environment, along with or possibly replacing the observations. The second robot-trained DNN may be trained to perform different tasks on the first robot-trained DNN, or it may be trained to perform the same tasks, in which case the second robot-trained DNN may define the actions to be performed. Further DNNs may be linked in a similar manner.

[0016] Methods for training a system may include determining training values ​​for the parameters of one or more DNNs in a sequence of DNNs. Therefore, if a simulated DNN has already been trained, a robot-trained DNN can be trained in a real-world environment, enabling it to utilize features learned by the simulated DNN. The training values ​​for the parameters of the simulated DNN can be fixed to avoid forgetting the simulation-based training. A simulated version of the real-world environment can be used to train the simulated DNN.

[0017] Training a robot-trained DNN may involve receiving the following as inputs: (i) layer outputs generated by layers preceding the output layer of a first robot-trained DNN, and (ii) layer outputs generated by layers preceding the output layer of a simulated DNN; applying a first set of parameters to the layer outputs generated by layers preceding the output layer of the first robot-trained DNN; and applying a second set of parameters to the layer outputs generated by layers preceding the output layer of the simulated DNN. The method may further include initializing the values ​​of the second set of parameters to match the training values ​​of the parameters of the output layer of the simulated DNN, and optionally initializing the values ​​of the first set of parameters to zero. For example, the method may initialize some or all weights of the connections to the output layer of the DNN column from the last layer of the previous DNN column to the output weights of the previous DNN column, and optionally initialize some or all weights of the output layer of the in-column DNN column to zero. This facilitates exploration, particularly by increasing the chances of real-world rewards, to compensate for the risk that real-world rewards are so sparse that reinforcement learning may fail.

[0018] Methods of using this system may include receiving observations of the current state of a real-world environment that characterizes the robot agent’s interaction with it to perform robot tasks; processing the observations using a neural network system; and selecting an action to be performed by the robot agent in response to the observations using policy outputs.

[0019] According to another aspect of the present invention, a computer-implemented method is provided, comprising: receiving observations of the current state of a real-world environment in which a robot agent performing a robot task is interacting; processing the observations using a neural network system to generate a policy output, wherein the neural network system comprises a sequence of deep neural networks (DNNs), wherein the sequence of DNNs comprises: a simulated-trained DNN that has been trained on a simulated version of the robot agent to perform a simulated version of the robot task interacting with a simulated version of the real-world environment, wherein: the simulated-trained DNN includes a first plurality of index layers, and one or more computer-implemented simulated-trained DNNs are configured to receive the observations and process the observations through each of the first plurality of index layers to generate a respective layer output of each of the first plurality of index layers; and a first robot-trained DNN that has been trained on a simulated version of the robot agent performing a robot task to generate a policy output. The robot agent is trained to perform robot tasks and interact with a real-world environment to determine the training values ​​of the parameters of a first robot-trained DNN while keeping the training values ​​of the parameters of the simulated DNN fixed. The first robot-trained DNN includes a second plurality of index layers. The first robot-trained DNN, implemented by one or more computers, is configured to receive the observations and process them through each of the second plurality of index layers to generate a policy output. Each of the one or more layers of the second plurality of index layers implemented by one or more computers is configured to receive the following as inputs: (i) the layer output generated by the previous layer of the first robot-trained DNN, and (ii) the layer output generated by the previous layer of the simulated DNN, wherein the previous layer is a layer whose index is one smaller than the index of this layer; and to select an action to be performed by the robot agent in response to the observation using the policy output.

[0020] According to another aspect of the invention, a system is provided, comprising: one or more computers; and one or more non-transitory computer-readable media storing instructions, which, when executed by the one or more computers, cause the one or more computers to perform operations including: receiving an observation of the current state of a real-world environment in which a robotic agent performing a robotic task is interacting; processing the observation using a neural network system to generate a policy output, wherein the neural network system comprises a sequence of deep neural networks (DNNs), wherein the sequence of DNNs comprises: a simulated-trained DNN trained on a simulated version of the robotic agent interacting with a simulated version of the real-world environment for performing a robotic task, wherein: the simulated-trained DNN includes a first plurality of index layers, and the simulated-trained DNN implemented by the one or more computers is configured to receive the observation and process the observation through each of the first plurality of index layers to generate the first plurality of index layers. The respective layer outputs of each layer in the DNN; and a first robot-trained DNN, which has been trained on the robot agent to perform robot tasks in interaction with a real-world environment, to determine the training values ​​of the parameters of the first robot-trained DNN while keeping the training values ​​of the parameters of the simulated-trained DNN fixed, wherein the first robot-trained DNN includes a second plurality of index layers, the first robot-trained DNN implemented by one or more computers is configured to receive the observations and process the observations through each of the second plurality of index layers to generate policy outputs, and each of the one or more layers of the second plurality of index layers implemented by one or more computers is configured to receive the following as inputs: (i) the layer output generated by the previous layer of the first robot-trained DNN, and (ii) the layer output generated by the previous layer of the simulated-trained DNN, wherein the previous layer is a layer whose index is one smaller than the index of this layer; and to select an action to be performed by the robot agent in response to the observations using the policy outputs.

[0021] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0022] Figure 1 An example reinforcement learning system is shown.

[0023] Figure 2 An example architecture of a neural network system is shown.

[0024] Figure 3 This is a flowchart of an example process for training a neural network system.

[0025] Figure 4 This is a flowchart of an example process for selecting an action to be performed by an agent in response to an observation.

[0026] The same reference numerals and names in each of the figures indicate the same elements. Detailed Implementation

[0027] This specification provides a general description of a reinforcement learning system that selects actions to be performed by a reinforcement learning agent interacting with a real-world environment. In order to interact with the real-world environment, the agent receives data representing the current state of the real-world environment and performs actions from an action space (e.g., a discrete action space or a continuous action space) in response to the received data. The data representing the state of the real-world environment will be referred to as observations in this specification.

[0028] In some implementations, the robotic agent may be a robot that interacts with a real-world environment to perform a specific task. In other implementations, the robotic agent may be an autonomous or semi-autonomous vehicle that navigates through a real-world environment. In these implementations, the action may be a point in space that is a possible control input for controlling the robot or autonomous vehicle.

[0029] In some cases, observations use low-dimensional feature vectors that characterize the state of the environment to represent the state of the environment. In these cases, the values ​​of the different dimensions of the low-dimensional feature vectors can have varying ranges.

[0030] In some other cases, observation uses high-dimensional pixel inputs from one or more images that characterize the state of the environment (e.g., images captured by the agent's sensors when the agent interacts with the real-world environment) to characterize the state of the environment.

[0031] Figure 1 An example reinforcement learning system 100 is shown. The reinforcement learning system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, in which the systems, components and techniques described below are implemented.

[0032] The reinforcement learning system 100 selects actions to be performed by the reinforcement learning agent 102 interacting with the real-world environment 104 to complete a task. That is, the reinforcement learning system 100 receives observations, where each observation represents a corresponding state of the environment 104, and in response to each observation, selects an action to be performed by the agent 102 in response to the observation.

[0033] Specifically, the reinforcement learning system 100 uses a neural network system 106 and an action subsystem 108 to select actions.

[0034] The neural network system 106 is configured to receive observations 112 that characterize the current state of the real-world environment 104 to which the agent 102 interacts to perform a task. For example, the agent 102 is a robotic agent performing a robotic task (such as an arrival task or a transmitter (motion tracking) task).

[0035] To bridge the gap with reality and transfer learned strategies from simulation to the real world, the neural network system 106 includes a sequence of deep neural networks (DNNs), comprising a simulated DNN and one or more robot-trained DNNs. The simulated DNNs are configured with sufficient capacity and depth to learn the task from scratch, while the one or more robot-trained DNNs are configured with smaller capacity to achieve maximum transfer across different tasks and faster learning. The simulated and robot-trained DNNs do not need to have the same capacity and structure, thus allowing for the solving of different tasks, including different input modalities. Each DNN in the sequence comprises multiple index layers and an output layer. See below for reference. Figure 2 The architecture of the neural network system 106 is described in more detail.

[0036] The neural network system 106 is configured to process observations 112 using a DNN sequence to generate a policy output 110. Typically, the policy output 110 defines a probability distribution of a set of possible actions that an agent 102 can perform in response to observations 112 to complete a task. For example, in some embodiments, the agent 102 has multiple degrees of freedom, including one or more joints and one or more actuators (e.g., a robotic arm with multiple joints and fingers). In these embodiments, the policy output 110 defines a probability distribution of possible actions that control the different degrees of freedom of the agent 102 in response to observations, such as actions that control position changes in each degree of freedom of the agent 102.

[0037] In some implementations, the neural network system 106 is configured to use a DNN sequence to process observation 112 to generate a second policy output that defines a probability distribution of a set of possible actions that can be performed by agent 102 to complete a second distinct task.

[0038] To allow agent 102 to effectively interact with real-world environment 104, reinforcement learning system 100 trains neural network system 106 to determine training values ​​for the parameters of neural network system 106. Specifically, system 100 trains a simulated DNN to perform a task on interactions between a simulated version of the agent and a simulated version of the real-world environment to determine training values ​​for the parameters of the simulated DNN. System 100 then trains a robot-trained DNN to perform a task on interactions between the agent and the real-world environment to determine training values ​​for the parameters of the robot-trained DNN, while keeping the training values ​​for the parameters of the simulated DNN fixed. During training, the parameter values ​​of the output layer of the robot-trained DNN are initialized based on the training values ​​of the parameters of the output layer of the simulated DNN to improve exploration and accelerate learning. (See below for reference.) Figure 3 A more detailed example process for training the neural network system 106 is described below.

[0039] After the neural network system 106 generates the policy output 110, the action subsystem 108 is configured to receive the policy output 110 from the neural network system 106 and select an action 114 to be performed by the agent 102 based on the policy output 110. Specifically, the action subsystem 108 can select the action 114 by sampling the action 114 from the probability distribution defined by the policy output 110, for example, selecting the most probable action. The action subsystem 108 then causes the agent to execute the selected action 114.

[0040] Figure 2 It shows Figure 1 Example architecture of neural network system 106.

[0041] The neural network system 106 includes a sequence of deep neural networks (DNNs).

[0042] The sequence of DNNs includes a simulated deep neural network (DNN)222, which has been trained on the interaction between a simulated version of the robotic agent and a simulated version of the real-world environment to perform a simulated version of the first robotic task.

[0043] Typically, a simulated DNN 222 comprises multiple neural network layers arranged in a stacked manner. The simulated DNN 222 can be a feedforward neural network or a recurrent neural network. It can also include one or more convolutional neural network layers followed by fully connected layers or long short-term memory (LSTM) neural network layers.

[0044] like Figure 2 As shown, the simulated DNN 222 includes multiple first index layers, such as layer h1. (1) (204) and layer h2 (1)(206), and output layer 208. The simulated-trained DNN 222 is configured to receive input 202, which includes observations representing the current state of a real-world environment. For example, observations include visual data, such as one or more RGB images depicting the current state of a real-world environment. The simulated-trained DNN 222 is then configured to process the observations through each first index layer to generate a corresponding layer output for each layer. The corresponding layer output for each layer is then provided as input to the next layer.

[0045] The sequence of DNNs also includes a first robot-trained DNN 224. The first robot-trained DNN 224 can be a feedforward or recurrent neural network. The first robot-trained DNN 224 can include one or more convolutional neural network layers, followed by fully connected neural network layers or LSTM neural network layers.

[0046] In many cases, the first robot-trained DNN 224 and the simulation-trained DNN 222 do not need to have the same capacity or structure, and this can be advantageous when simulating real-world scenarios. The simulation-trained DNN 222 can be designed with sufficient capacity and depth to learn the robot task from scratch, but the robot-trained DNN 224 can have a minimal capacity to incentivize rapid learning and limit the growth of the total parameters. Therefore, in some cases, the simulation-trained DNN 222 is a standard-sized network, while the first robot-trained DNN 224 is a reduced-capacity network to accelerate training. This means that the first robot-trained DNN 224 has significantly fewer parameters than the simulation-trained DNN 222. For example, when both the simulation-trained DNN 222 and the first robot-trained DNN 224 are feedforward neural networks, the simulation-trained DNN 222 can have hundreds of thousands of parameters, for example, approximately 600,000 parameters, while the first robot-trained DNN 224 may only have tens of thousands of parameters, for example, approximately 40,000 parameters. In another example, when both the simulated DNN 222 and the first robot-trained DNN 224 are recurrent neural networks, the simulated DNN 222 may have approximately 300,000 parameters, while the first robot-trained DNN 224 may have approximately 40,000 parameters.

[0047] In some cases, the simulated-trained DNN 222 has a wide architecture, while the first robot-trained DNN 224 has a narrow architecture. For example, for each of one or more layers in a plurality of second index layers, the corresponding layer in a plurality of first index layers with the same index is a neural network layer of the same type but with a dimension greater than that of the layers in the plurality of second index layers. For example, the corresponding layer in a plurality of first index layers has far more nodes than the layers in a plurality of second index layers.

[0048] like Figure 2 As shown, the DNN 224 trained by the first robot includes multiple second index layers, for example, layer h1 (2) (210) and layer h2 (2) (212), and output layer 214.

[0049] The first robot-trained DNN 224 is configured to receive input 202, which includes observations (e.g., RGB images) representing the current state of the real-world environment. Typically, the first robot-trained DNN 224 is configured to process the input 202 jointly through each of a plurality of second index layers to generate a policy output 228, which defines actions to be performed by the robot agent to carry out a first robot task. The policy output defines corresponding changes in position for each of a plurality of degrees of freedom of the agent. Degrees of freedom include one or more joints of the agent and one or more actuators of the agent.

[0050] Specifically, the first layer of multiple second index layers (e.g., layer h1) (2) (210) is configured to receive input 202 and process input 202 to generate the first layer output.

[0051] To pass features learned from the simulated interaction of the agent for selecting actions to be performed by the agent, one or more layers following the first layer 210 in a plurality of second index layers are configured to receive the following as inputs: (i) the layer output generated by the previous layer of the DNN 224 trained by the first robot, and (ii) the layer output generated by the previous layer of the simulated DNN 222. The previous layer is a layer whose index is one less than the index of the layer. For example, layer h2 (2) (212) is configured to receive layer h1 of DNN 224 trained by the first robot. (2) (210) The generated layer output and the output from layer h1 (1) (204) The generated layer output is used as input. The system can be configured to apply a first set of parameters (weights) to the layer output generated by the previous layer of a DNN trained by a first robot, and to apply a second set of parameters (weights) to the layer output generated by the previous layer of a DNN trained by simulation. Optionally, the connections between the DNN “columns” of the sequence can include neural networks such as MLPs (Multilayer Perceptrons) to adapt the layer output of one DNN to the layer input of the next DNN, for example, by scaling and / or dimensionality reduction. Such adaptation may not be required when connecting to the output layer.

[0052] In some cases, both the multiple first index layers and the multiple second index layers include corresponding recurrent neural network (RNN) layers. In these cases, the RNN layers in the multiple second index layers are configured to receive the following as inputs: (i) the layer outputs of the layers preceding the RNN layers in the multiple first index layers, (ii) the internal states of the RNN layers in the multiple first index layers, and (iii) the layer outputs of the layers preceding the RNN layers in the multiple second index layers. For example, assuming that layer 206 of the multiple first index layers and layer 212 of the multiple second index layers are RNN layers, layer 212 is configured to receive the following as inputs: the layer output of layer 204, the internal state of layer 206, and the layer output of layer 210.

[0053] Specifically, output layer 214 is configured to receive the following as input: (i) layer outputs generated by layers preceding the output layer of the DNN 224 trained by the first robot, for example, by layer h2. (2) (212) The generated layer output, and (ii) The layer output generated by the layer preceding the output layer of the simulated DNN 222, for example, by layer h2 (1) (206) Generated Layer Output. Output layer 214 is configured to apply a first set of parameters to the layer output generated before the output layer of the DNN 224 trained by the first robot, and to apply a second set of parameters to the layer output generated before the output layer of the DNN 222 trained by simulation, to generate policy output 228. The first set of parameters and the second set of parameters are determined after training the DNN 224 trained by the first robot using reinforcement learning techniques on the interaction between the robot agent and the real-world environment. (See below for reference.) Figure 3 The training of the first robot's DNN 224 is described in more detail.

[0054] Optionally, the neural network system 106 may include a second robot-trained DNN 226, which has been trained on interactions between the robot agent and a real-world environment to perform a second robot task different from the first robot task. The second robot-trained DNN 226 includes multiple third index layers (e.g., layer h1). (3) (216) and layer h2 (3) (218)), and output layer 220. The DNN trained on the second robot is configured to receive additional data representing the current state as input along with observations (e.g., RGB images) included in input 202. For example, the additional data is proprioceptive data that includes proprioceptive features of the robot agent, such as joint angles and velocities of each of the robot agent's joints and actuators.

[0055] By using the additional data as input, the DNN trained on the second robot allows the neural network system 106 the flexibility to add new capacities (including new input modalities) when transitioning to new tasks. This is superior to existing methods, which cannot adapt to changing network configurations or new input modalities. This is also advantageous for bridging the reality gap, adapting to different inputs from simulated and real-world sensors.

[0056] In one variant, the DNN trained by the second robot may receive only additional data without receiving observations, and may implicitly receive observations via lateral connections to the DNN trained by the first robot and / or the DNN trained by simulation.

[0057] The first of multiple third index layers (e.g., layer h1) (3) (216) is configured to process the received input 202 and additional data to generate layer outputs. One or more layers following the first layer 216 in the second robot-trained DNN 226 are configured to receive the following as inputs: (i) the layer output generated by the previous layer of the first robot-trained DNN 224, (ii) the layer output generated by the previous layer of the simulated-trained DNN 222, and (iii) the layer output generated by the previous layer of the second robot-trained DNN 226. For example, layer h2 of the second robot-trained DNN 226... (3) (218) is configured to receive the following as input: layer h1 of a DNN 224 trained by the first robot. (2) (210) The generated layer output is derived from layer h1 of the simulated trained DNN. (1) (204) The generated layer output 222, and layer h1 of the DNN 226 trained by the second robot. (3) (216) The generated layer output.

[0058] One or more layers following the first layer 216 are configured to process the corresponding inputs to generate the corresponding layer outputs. The layer outputs are then provided as input to the next layer, or, if there are no layers following the current layer (i.e., the current layer is output layer 220), the layer outputs are provided as the second policy output 230. The second policy output 230 defines the actions to be performed by the robotic agent to carry out the second robotic task.

[0059] Figure 3 It is used to train neural network systems (e.g., Figure 1 A flowchart of an example process 300 for a neural network system 106 is provided. For convenience, process 300 will be described as being executed by a system of one or more computers located in one or more locations. For example, a reinforcement learning system (e.g., ...) properly programmed according to this specification. Figure 1 The reinforcement learning system 100 can execute process 300.

[0060] The system first trains a simulated DNN of the neural network system to perform a robotic task (i.e., a real-world task) by interacting with a simulated version of the robotic agent and a simulated version of the real-world environment, in order to determine the training values ​​of the parameters of the simulated DNN (step 302). The simulated version of the robotic agent is a virtual 3D model that has a similar appearance to a real-world robotic agent and is capable of simulating the movements of a real-world robotic agent. The simulated version of the real-world environment is a virtual 3D environment that renders the real-world environment and objects within it. During training, the simulated version of the robotic agent is programmed to move or interact with virtual objects in the simulated environment to complete the simulated version of the real-world task. The system can simulate training input data. For example, if the input data in the real-world environment includes observations captured by a camera, the simulated input data is provided by a component capable of rendering possible observations.

[0061] To allow neural network systems to effectively adapt to variations in real-world tasks, environments, and robotic agents, the system can introduce variance into the training of the simulated-trained DNN. Specifically, real-world robotic agents may differ, for example, due to manufacturing inconsistencies. Furthermore, different instances of a task may differ, for example, because the object being moved may have slightly different dimensions, and real-world agents may have different capabilities for performing selected actions due to manufacturing defects. Therefore, the system can train the simulated-trained DNN on simulated tasks that are not entirely identical to real-world tasks, allowing the simulated-trained DNN to generalize to unpredictable situations that may occur in the real world.

[0062] This system can incorporate variance into the training of a simulated DNN in various ways. For example, the system can apply random noise to an action being performed by an agent before the system selects an action. As another example, the system can change the dimensions of objects in a simulated version of the environment between instances of a simulated task.

[0063] This system can use first-order reinforcement learning techniques to train a simulated DNN. For example, the system can use the Asynchronous Advantage Actor-Critic (A3C) framework to train a simulated DNN. The training method using the A3C framework is described in detail in "Asynchronous methods for deep reinforcement learning" by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu (presented at the International Conference on Machine Learning (ICML) 2016). As another example, the system can use policy gradient Actor-Critic algorithms (e.g., Deep Deterministic Policy Gradient (DDPG) algorithms) to train a simulated DNN. The training method using the DDPG algorithm is described in detail in Continuous control with Deep Reinforcement Learning by Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra (available at https: / / arxiv.org / abs / 1509.02971).

[0064] Compared to real-world robots, simulation training is accelerated by fast rendering, multi-threaded learning algorithms, and the ability to train continuously without human intervention. Using simulation, the system can efficiently explore various training parameters. For example, the system can explore the learning rate and entropy cost, which are randomly and uniformly sampled on a logarithmic scale.

[0065] Then, the system trains the first robot's trained DNN to perform the first robot task by interacting with the robot agent in the real-world environment, in order to determine the training values ​​of the parameters of the first robot's trained DNN (step 304). Typically, the system first initializes the parameter values ​​of the first robot's trained DNN to predetermined or randomly generated initial values.

[0066] In real-world training, there is a risk that the rewards received by the system as a result of actions performed by the robotic agent may be so sparse or nonexistent that convergence is low or nonexistent. Therefore, to maximize the probability of rewards during exploration in the real-world domain, the system initializes the first robot-trained DNN such that the robot agent's initial policy is identical to that of the simulated-trained DNN. Figure 2 In the example neural network system 106, when the parameters of the first robot-trained DNN 224 are instantiated, the output layer 214 has input connections from layers 212 and 206 of the simulated-trained DNN. Unlike the other parameters of the first robot-trained DNN, which are randomly initialized, the output layer 214 initializes the value of a first parameter applied to the layer output of layer 212 to zero, and the output layer 214 initializes the value of a second parameter applied to the layer output of layer 206 to the training values ​​matching the parameters of the output layer of the simulated-trained DNN.

[0067] After initializing the parameter values ​​of the first robot-trained DNN, the system trains the first robot-trained DNN to perform robot tasks through interactions between the agent and the real-world environment, thereby determining the training values ​​of the parameters of the first robot-trained DNN while keeping the training values ​​of the simulated DNN fixed. In addition to observations representing the state of the real-world environment, the first robot-trained DNN can be trained on additional inputs. These additional inputs can be proprioceptive data, including the proprioceptive features of the robot agent. Proprioceptive features can include the angles and velocities of each joint in the robot agent's joints and actuators.

[0068] The system can use a second reinforcement learning technique to train the DNN trained by the first robot on the interactions of the agent. For example, the system can use the Asynchronous Advantage Actor-Critic (A3C) framework, the policy gradient Actor-Critic algorithm (such as the Deep Deterministic Policy Gradient (DDPG) algorithm), or other reinforcement learning techniques to train the DNN trained by the first robot. The second reinforcement learning technique can be the same as or different from the first reinforcement learning technique.

[0069] Then, the system trains a second robot-trained DNN on interactions between the robot agent and the real-world environment to perform a second, different robot task, in order to determine the training values ​​of the parameters of the second robot-trained DNN (step 306). The system trains the second robot-trained DNN on the second, different robot task to extend the capabilities of the entire system.

[0070] The system first initializes the parameter values ​​of the DNN trained by the second robot to randomly generated initial values.

[0071] The system uses a third reinforcement learning technique to train the DNN trained by the second robot. This third reinforcement learning technique utilizes the lateral neural connections between the simulated-trained DNN and the first robot-trained DNN. The third reinforcement learning technique can be, for example, using the Asynchronous Advantage Actor-Critic (A3C) framework, the policy gradient Actor-Critic algorithm (e.g., the Deep Deterministic Policy Gradient (DDPG) algorithm), or other reinforcement learning techniques. Trained in this way, the DNN trained by the second robot can leverage features or representations learned in the first robot task, encoded in either the simulated-trained DNN or the first robot-trained DNN, to improve the learning speed for the second robot task. For example, the first robot task might be retrieving a stationary object, while the second task might be retrieving a dynamically moving object.

[0072] In some implementations, in addition to observations representing the state of the real-world environment, a second robot-trained DNN can be trained on additional inputs. These additional inputs can be proprioceptive data, including proprioceptive features of the robot agent. Proprioceptive features can include the angles and velocities of each of the robot agent's joints and actuators. In some implementations, during training, the second robot-trained DNN may receive only the additional data and not the observations, and may implicitly receive the observations via lateral connections to the first robot-trained DNN and / or the simulated-trained DNN.

[0073] Figure 4 This is a flowchart of an example process 400 for selecting an action to be performed by an agent in response to an observation. For convenience, process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a reinforcement learning system appropriately programmed according to this specification (e.g., Figure 1 The reinforcement learning system 100 can execute process 400.

[0074] The system receives an observation (step 402) of the current state of the real-world environment to which the agent interacts to perform a robotic task. This observation uses high-dimensional pixel inputs from one or more images representing the current state of the environment, such as images captured by the agent's sensors as the agent interacts with the real-world environment.

[0075] The system then processes the observations using a neural network system to generate a policy output (step 404). The system processes the observations based on the training values ​​of the parameters of the neural network system. Typically, the policy output defines a probability distribution of a set of possible actions that can be performed by the agent in response to the observation. For example, in some embodiments, the agent has multiple degrees of freedom, including one or more joints and one or more actuators (e.g., a robotic arm with multiple joints and fingers). In these embodiments, the policy output defines a probability distribution of possible actions that control the different degrees of freedom of the agent in response to the observation, such as actions that control the positional changes of each degree of freedom of the agent. For example, in the case of a robotic arm, actions could include velocity commands for the arm and finger joints.

[0076] The system uses the policy output to select the action to be performed by the agent in response to the observation (step 406). Specifically, the system selects the action by sampling actions from a probability distribution defined by the policy output.

[0077] Then, the system causes the agent to perform an action (step 408).

[0078] This specification uses the term "configured" in relation to system and computer program components. For a system of one or more computers configured to perform a specific operation or action, this means that the system has software, firmware, hardware, or a combination thereof installed thereon that causes the system to perform the operation or action during operation. For one or more computer programs configured to perform a specific operation or action, this means that one or more programs include instructions that, when run by a data processing device, cause the device to perform the operation or action.

[0079] The embodiments of the subject matter and functional operation described in this specification may be implemented using digital electronic circuits, computer software or firmware tangibly implemented, computer hardware (including the structures disclosed in this specification and their equivalents), or a combination of one or more of these. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs (i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing apparatus or for controlling the operation of a data processing apparatus). The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or further, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, generated to encode information for transmission to a suitable receiver device for execution by a data processing apparatus.

[0080] The term "data processing apparatus" refers to data processing hardware and includes all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus may also be or may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0081] A computer program (which may also be referred to or described as a program, software, software application, app, module, software module, script, or code) can be written in any programming language (including compiled or interpreted languages, or declarative or procedural languages), and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to said program, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected via a data communication network.

[0082] The processes and logic described in this specification can be executed by one or more programmable computers running one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic can also be executed by special-purpose logic circuitry (e.g., FPGA or ASIC) or a combination of special-purpose logic circuitry and one or more programmable computers.

[0083] A computer suitable for running computer programs can be based on a general-purpose microprocessor, a special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. Essential components of a computer are the central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented or integrated therein by dedicated logic circuitry. Typically, a computer will also include, or be operatively coupled to, receiving data from or transferring data to one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or both. However, a computer may not necessarily have such devices. Furthermore, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), etc.

[0084] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD ROMs and DVD-ROMs.

[0085] To provide interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, that the user can use to provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending and receiving documents from a device used by the user; for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and then receiving response messages from the user.

[0086] The data processing apparatus for implementing machine learning models may also include, for example, a dedicated hardware accelerator unit for processing the common and computationally intensive parts of machine learning training or production, namely inference and workloads.

[0087] Machine learning frameworks can be used to implement and deploy machine learning models, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0088] Embodiments of the subject matter described in this specification can be implemented in computing systems that include back-end components (e.g., as data servers), or middleware components (e.g., application servers), or front-end components (e.g., client computers having a graphical user interface, web browser, or app that a user can use to interact with embodiments of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. Components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0089] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is established by means of computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., HTML pages) to a user device, for example, for the purpose of displaying data to a user interacting with the device as a client and receiving user input from the user. Data generated at the user device, such as the result of user interaction, may be received at the server from the device.

[0090] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in the context of a single embodiment may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may have been described above as functioning in certain combinations and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0091] Similarly, although operations are depicted in a specific order in the accompanying drawings and recited in a specific order in the claims, this should not be construed as requiring such operations to be performed in the shown specific order or sequential order, or requiring the performance of all illustrated operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0092] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method comprising: receiving, from a robotic agent executing a robotic task, an observation of a current state of a real-world environment with which the robotic agent is interacting, wherein the observation is visual data; processing the observation using a neural network system to generate a policy output, wherein the neural network system comprises a sequence of deep neural networks (DNNs), wherein the sequence of DNNs comprises: a simulated, trained DNN that has been trained on interactions of a simulated version of the robotic agent with a simulated version of the real-world environment for a simulated version of the robotic task, wherein: the simulated, trained DNN comprises a first plurality of indexed layers, and the one or more computer-implemented simulated, trained DNNs are configured to receive the observation and process the observation through each of the first plurality of indexed layers to generate a respective layer output for each of the first plurality of indexed layers; and a first robotic, trained DNN that has been trained on interactions of the robotic agent with the real-world environment for the robotic task to determine trained values of parameters of the first robotic, trained DNN while fixing trained values of parameters of the simulated, trained DNN, wherein the first robotic, trained DNN comprises a second plurality of indexed layers, the one or more computer-implemented first robotic, trained DNNs are configured to receive the observation and process the observation through each of the second plurality of indexed layers to generate the policy output, and each of the one or more layers of the second plurality of indexed layers is configured to receive, as input, (i) a layer output generated by a preceding layer of the first robotic, trained DNN and (ii) a layer output generated by a preceding layer of the simulated, trained DNN, wherein the preceding layer is a layer whose index is one less than the index of the layer; and selecting, using the policy output, an action to be performed by the robotic agent in response to the observation.

2. The method of claim 1, wherein, the first robotic, trained DNN has a smaller capacity than the simulated, trained DNN.

3. The method of claim 1, wherein, for each of the one or more layers of the second plurality of indexed layers, a corresponding layer of the first plurality of indexed layers having the same index is a neural network layer of the same type as the layer of the second plurality of indexed layers but of a larger dimension.

4. The method of claim 1, wherein, each of the first plurality of indexed layers and the second plurality of indexed layers comprises a respective recurrent neural network layer.

5. The method of claim 4, wherein, the recurrent neural network layer of the second plurality of indexed layers is configured to receive, as input, (i) a layer output of a preceding layer of the recurrent neural network layer of the first plurality of indexed layers, (ii) an internal state of the recurrent neural network layer of the first plurality of indexed layers, and (iii) a layer output of a preceding layer of the recurrent neural network layer of the second plurality of indexed layers.

6. The method of claim 1, wherein, each of the one or more layers of the second plurality of indexed layers is configured to receive, as input, (i) a layer output generated by a preceding layer of the first robotic, trained DNN and (ii) a layer output generated by a preceding layer of the one or more computer-implemented simulated, trained DNNs, each layer being further configured to: applying the respective first set of parameters to a layer output generated by a preceding layer of the DNN trained by the first robot; and applying the respective second set of parameters to a layer output generated by a preceding layer of the DNN trained by the simulation.

7. The method of claim 1, wherein, The policy output defines a respective change in position for each of a plurality of degrees of freedom of the robotic agent.

8. The method of claim 7, wherein, The plurality of degrees of freedom includes one or more joints of the robotic agent and one or more actuators of the robotic agent.

9. The method of claim 1, wherein, The sequence of DNNs further includes: a second robot-trained DNN, wherein: the second robot-trained DNN includes a third plurality of indexing layers, and each of one or more layers of the third plurality of indexing layers implemented by one or more computers is configured to receive as input (i) a layer output generated by a preceding layer of the DNN trained by the first robot, (ii) a layer output generated by a preceding layer of the DNN trained by the simulation, and (iii) a layer output generated by a preceding layer of the second robot-trained DNN.

10. The method of claim 9, wherein: the second robot-trained DNN implemented by one or more computers is configured to receive different data characterizing a current state of the observation; and the second robot-trained DNN implemented by one or more computers is configured to process the different data through the third plurality of indexing layers to generate a second policy output defining an action to be performed by the robotic agent in performing a second different robotic task.

11. The method of claim 10, wherein, The different data is proprioceptive data.

12. One or more non-transitory computer-readable media having stored thereon instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: receiving an observation performed by a robotic agent executing a robotic task that characterizes a current state of a real-world environment with which the robotic agent is interacting, wherein the observation is visual data; processing the observation using a neural network system to generate a policy output, wherein the neural network system includes a sequence of deep neural networks (DNNs), wherein the sequence of DNNs includes: a simulation-trained DNN that has been trained on interactions of a simulated version of the robotic agent with a simulated version of the real-world environment in performing a simulated version of the robotic task, wherein: the simulation-trained DNN includes a first plurality of indexing layers, and the simulation-trained DNN implemented by one or more computers is configured to receive the observation and process the observation through each of the first plurality of indexing layers to generate a respective layer output for each of the first plurality of indexing layers; and a first robot-trained DNN that has been trained on interactions of the robotic agent with the real-world environment in performing the robotic task to determine training values of parameters of the first robot-trained DNN while fixing training values of parameters of the simulation-trained DNN, wherein the first robot-trained DNN includes a second plurality of indexing layers, the DNN trained by the one or more computers is configured to receive the observation and process the observation through each of a first plurality of indexed layers to generate a layer output for each of the first plurality of indexed layers, and each of one or more layers of the second plurality of indexed layers implemented by the one or more computers is configured to receive, as input, (i) a layer output generated by a preceding layer of the DNN trained by the one or more computers and (ii) a layer output generated by a preceding layer of the DNN trained by the one or more computers, wherein the preceding layer is a layer whose index is one less than the index of the layer; and the policy output is used to select an action to be performed by the robotic agent in response to the observation.

13. A system comprising: one or more computers; and one or more non-transitory computer-readable media having stored thereon instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising: receiving an observation characterizing a current state of a real-world environment with which a robotic agent performing a robotic task is interacting, wherein the observation is visual data; processing the observation using a neural network system to generate a policy output, wherein the neural network system comprises a sequence of deep neural networks (DNNs), wherein the sequence of DNNs comprises: a DNN trained by simulation that has been trained on interactions of a simulated version of the robotic agent with a simulated version of the real-world environment for a simulated version of the robotic task, wherein: the DNN trained by simulation comprises a first plurality of indexed layers, and the DNN trained by simulation is configured to receive the observation and process the observation through each of the first plurality of indexed layers to generate a respective layer output for each of the first plurality of indexed layers; and a DNN trained by the robotic agent that has been trained on interactions of the robotic agent with the real-world environment for the robotic task to determine trained values of parameters of the DNN trained by the robotic agent while fixing trained values of parameters of the DNN trained by simulation, wherein the DNN trained by the robotic agent comprises a second plurality of indexed layers, the DNN trained by the one or more computers is configured to receive the observation and process the observation through each of the second plurality of indexed layers to generate the policy output, and each of one or more layers of the second plurality of indexed layers implemented by the one or more computers is configured to receive, as input, (i) a layer output generated by a preceding layer of the DNN trained by the one or more computers and (ii) a layer output generated by a preceding layer of the DNN trained by simulation, wherein the preceding layer is a layer whose index is one less than the index of the layer; and the policy output is used to select an action to be performed by the robotic agent in response to the observation.

14. One or more non-transitory computer-readable media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a neural network system, the neural network system comprising a sequence of deep neural networks (DNNs) to determine trained values of parameters of one or more of the DNNs in the sequence of DNNs, wherein, A neural network system implemented by one or more computers is configured to receive an observation, which is visual data, that characterizes a current state of a real-world environment with which a robotic agent is interacting that is performing a robotic task, and process the observation to generate a policy output that defines an action to be performed by the robotic agent in response to the observation, wherein the sequence of DNNs includes: a simulated-trained DNN, wherein the simulated-trained DNN implemented by one or more computers includes a first plurality of indexing layers and is configured to receive the observation and process the observation through each of the first plurality of indexing layers to generate a respective layer output for each of the first plurality of indexing layers; and a first robot-trained DNN that has been trained on interactions of the robotic agent with the real-world environment for performing the robotic task to determine training values for parameters of the first robot-trained DNN while holding training values for parameters of the simulated-trained DNN fixed, the first robot-trained DNN includes a second plurality of indexing layers, the first robot-trained DNN implemented by one or more computers is configured to receive the observation and process the observation through each of the second plurality of indexing layers to generate the policy output, and each of one or more layers of the second plurality of indexing layers implemented by one or more computers is configured to receive as input: (i) a layer output generated by a preceding layer of the first robot-trained DNN, and (ii) a layer output generated by a preceding layer of the simulated-trained DNN, wherein the preceding layer is a layer whose index is one less than the index of the layer, wherein the operations include: training the first robot-trained DNN on interactions of the robotic agent with the real-world environment for performing the robotic task to determine training values for parameters of the first robot-trained DNN while holding training values for parameters of the simulated-trained DNN fixed.

15. The one or more non-transitory computer-readable media of claim 14, wherein an output layer of the first robot-trained DNN implemented by one or more computers is configured to: receive as input: (i) a layer output generated by a preceding layer of the output layer of the first robot-trained DNN, and (ii) a layer output generated by a preceding layer of the output layer of the simulated-trained DNN; apply a first set of parameters to the layer output generated by the preceding layer of the output layer of the first robot-trained DNN; and apply a second set of parameters to the layer output generated by the preceding layer of the output layer of the simulated-trained DNN, and wherein the operations further include: initializing values of the second set of parameters to match the training values for the parameters of the output layer of the simulated-trained DNN.

16. The one or more non-transitory computer-readable media of claim 15, wherein, the operations further include: initializing values of the first set of parameters to zero.

17. The one or more non-transitory computer-readable media of claim 15, wherein, the operations further include: initializing parameter values for layers of the first robot-trained DNN other than the output layer of the first robot-trained DNN to random values.

18. The one or more non-transitory computer-readable media of claim 14, wherein, the operations further include: training the simulated-trained DNN includes training the simulated-trained DNN on interactions of a simulated version of the robotic agent using a first reinforcement learning technique.

19. The one or more non-transitory computer-readable media of claim 18, wherein, training the simulated-trained DNN includes training the simulated-trained DNN on interactions of a simulated version of the robotic agent using a first reinforcement learning technique.

20. The one or more non-transitory computer-readable media of claim 19, wherein, training the first robot-trained DNN includes training the first robot-trained DNN on interactions of the robotic agent using a second reinforcement learning technique.

21. A system comprising: one or more computers; and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training a neural network system comprising a sequence of deep neural networks (DNNs) to determine training values for parameters of one or more DNNs in the sequence of DNNs, wherein the neural network system implemented by the one or more computers is configured to receive an observation characterizing a current state of a real-world environment with which a robotic agent performing a robotic task is interacting and to process the observation to generate a policy output defining an action to be performed by the robotic agent in response to the observation, wherein the observation is visual data, wherein the sequence of DNNs comprises: a simulated-trained DNN, wherein the simulated-trained DNN implemented by the one or more computers includes a first plurality of indexing layers and is configured to receive the observation and process the observation through each of the first plurality of indexing layers to generate a respective layer output for each of the first plurality of indexing layers; and a first robot-trained DNN that has been trained on interactions of the robotic agent with the real-world environment for performing the robotic task to determine training values for parameters of the first robot-trained DNN while holding training values for parameters of the simulated-trained DNN fixed, the first robot-trained DNN includes a second plurality of indexing layers, the first robot-trained DNN implemented by the one or more computers is configured to receive the observation and process the observation through each of the second plurality of indexing layers to generate the policy output, and each of one or more layers of the second plurality of indexing layers implemented by the one or more computers is configured to receive, as input, (i) a layer output generated by a preceding layer of the first robot-trained DNN and (ii) a layer output generated by a preceding layer of the simulated-trained DNN, wherein the preceding layer is a layer whose index is one less than the index of the layer, wherein the operations comprise: training the first robot-trained DNN on interactions of the robotic agent with the real-world environment for performing the robotic task to determine training values for parameters of the first robot-trained DNN while holding training values for parameters of the simulated-trained DNN fixed.

22. A computer-implemented method comprising: receiving, by a robotic agent executing a robotic task, an observation characterizing a current state of a real-world environment with which it is interacting, wherein the observation is visual data; processing the observation using a neural network system to generate a policy output, wherein the neural network system comprises a sequence of deep neural networks (DNNs), wherein the sequence of DNNs comprises: a simulated, trained DNN that has been trained on interactions of a simulated version of the robotic agent with a simulated version of the real-world environment for a simulated version of the robotic task, wherein the simulated, trained DNN is configured to receive and process the observation to generate one or more layer outputs, and a first robot-trained DNN that has been trained on interactions of the robotic agent with the real-world environment for the robotic task to determine training values of parameters of the first robot-trained DNN while fixing training values of parameters of the simulated, trained DNN, wherein the first robot-trained DNN is configured to receive and process the observation and the one or more layer outputs generated by the simulated, trained DNN to generate the policy output, and selecting, using the policy output, an action to be performed by the robotic agent in response to the observation, wherein the first robot-trained DNN and the simulated, trained DNN have different capabilities.

23. The method of claim 22, wherein, The first robot-trained DNN has a smaller capacity than the simulated, trained DNN.

24. The method of claim 22, wherein the simulated, trained DNN comprises a first plurality of indexed layers, and the one or more computer-implemented simulated, trained DNNs are configured to process the observation through each layer of the first plurality of indexed layers to generate a respective layer output for each layer of the first plurality of indexed layers; and wherein the first robot-trained DNN comprises a second plurality of indexed layers, and the one or more computer-implemented first robot-trained DNNs are configured to process the observation through each layer of the second plurality of indexed layers to generate the policy output.

25. The method of claim 24, wherein, For each of one or more layers of the second plurality of indexed layers, a corresponding layer of the first plurality of indexed layers having the same index is a neural network layer of the same type as the layer of the second plurality of indexed layers but of a larger dimension.

26. The method of claim 24, wherein, Each of the first plurality of indexed layers and the second plurality of indexed layers comprises a respective recurrent neural network layer.

27. The method of claim 26, wherein, The recurrent neural network layer of the second plurality of indexed layers is configured to receive, as input, (i) a layer output of a previous layer of the recurrent neural network layer of the first plurality of indexed layers, (ii) an internal state of the recurrent neural network layer of the first plurality of indexed layers, and (iii) a layer output of a previous layer of the recurrent neural network layer of the second plurality of indexed layers.

28. The method of claim 24, wherein, Each of the one or more layers of the second plurality of indexing layers implemented by one or more computers is configured to receive as input (i) a layer output generated by a preceding layer of the DNN trained by the first robot and (ii) a layer output generated by a preceding layer of the DNN trained by the simulation, and wherein each of the one or more layers of the second plurality of indexing layers is further configured to: apply a respective first set of parameters to the layer output generated by the preceding layer of the DNN trained by the first robot; and apply a respective second set of parameters to the layer output generated by the preceding layer of the DNN trained by the simulation, wherein the preceding layer is a layer whose index is one less than the index of the layer.

29. The method of claim 24, wherein, The policy output defines a respective change in position for each of a plurality of degrees of freedom of the robotic agent.

30. The method of claim 29, wherein, The plurality of degrees of freedom includes one or more joints of the robotic agent and one or more actuators of the robotic agent.

31. The method of claim 24, wherein, The sequence of DNNs further includes: a DNN trained by a second robot, wherein: the DNN trained by the second robot includes a third plurality of indexing layers, and each of the one or more layers of the third plurality of indexing layers implemented by one or more computers is configured to receive as input (i) a layer output generated by a preceding layer of the DNN trained by the first robot, (ii) a layer output generated by a preceding layer of the DNN trained by the simulation, and (iii) a layer output generated by a preceding layer of the DNN trained by the second robot.

32. The method of claim 31, wherein: the DNN trained by the second robot is configured to receive different data characterizing a current state of the observation; and the DNN trained by the second robot is configured to process the different data through the third plurality of indexing layers to generate a second policy output, the second policy output defining an action to be performed by the robotic agent in performing a second different robotic task.

33. The method of claim 32, wherein, The different data is proprioceptive data.

34. One or more non-transitory computer-readable media having stored therein instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: receiving an observation performed by a robotic agent performing a robotic task, the observation characterizing a current state of a real-world environment with which the robotic agent is interacting, wherein the observation is visual data; processing the observation using a neural network system to generate a policy output, wherein the neural network system includes a sequence of deep neural networks (DNNs), wherein the sequence of DNNs includes: a DNN trained by simulation, the DNN trained by simulation having been trained on interactions of a simulated version of the robotic agent performing a simulated version of the robotic task with a simulated version of the real-world environment, wherein the DNN trained by simulation is configured to receive the observation and process the observation to generate one or more layer outputs, and the sequence of DNNs further includes: a DNN trained by a second robot, wherein: the DNN trained by the second robot includes a third plurality of indexing layers, and each of the one or more layers of the third plurality of indexing layers implemented by one or more computers is configured to receive as input (i) a layer output generated by a preceding layer of the DNN trained by the first robot, (ii) a layer output generated by a preceding layer of the DNN trained by the simulation, and (iii) a layer output generated by a preceding layer of the DNN trained by the second robot. a first robot-trained DNN that has been trained on interactions of a robot agent with a real-world environment for performing a robot task to determine trained values of parameters of the first robot-trained DNN while keeping trained values of parameters of the simulated-trained DNN fixed, wherein the first robot-trained DNN is configured to receive the observation and one or more layer outputs generated by the simulated-trained DNN and process the observation and the one or more layer outputs to generate a policy output, and selecting, using the policy output, an action to be performed by the robot agent in response to the observation, wherein the first robot-trained DNN and the simulated-trained DNN have different capabilities.

35. A system comprising: one or more computers; and one or more non-transitory computer-readable media having stored therein instructions, that, when executed by the one or more computers, cause the one or more computers to perform operations comprising: receiving, from a robot agent performing a robot task, an observation characterizing a current state of a real-world environment with which the robot agent is interacting, wherein the observation is visual data; processing the observation using a neural network system to generate a policy output, wherein the neural network system comprises a sequence of deep neural networks (DNNs), wherein the sequence of DNNs comprises: a simulated-trained DNN that has been trained on interactions of a simulated version of a robot agent with a simulated version of a real-world environment for performing a simulated version of a robot task, wherein the simulated-trained DNN is configured to receive the observation and process the observation to generate one or more layer outputs, and a first robot-trained DNN that has been trained on interactions of a robot agent with a real-world environment for performing a robot task to determine trained values of parameters of the first robot-trained DNN while keeping trained values of parameters of the simulated-trained DNN fixed, wherein the first robot-trained DNN is configured to receive the observation and one or more layer outputs generated by the simulated-trained DNN and process the observation and the one or more layer outputs to generate a policy output, and selecting, using the policy output, an action to be performed by the robot agent in response to the observation, wherein the first robot-trained DNN and the simulated-trained DNN have different capabilities.

36. One or more non-transitory computer-readable media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a neural network system, the neural network system comprising a sequence of deep neural networks (DNNs) to determine trained values of parameters of one or more of the DNNs in the sequence of DNNs, wherein, a neural network system implemented by one or more computers is configured to receive, from a robot agent performing a robot task, an observation characterizing a current state of a real-world environment with which the robot agent is interacting, and process the observation to generate a policy output, wherein the observation is visual data, the policy output defining an action to be performed by the robot agent in response to the observation, wherein the sequence of DNNs comprises: a simulated-trained DNN configured to receive the observations and process the observations to generate one or more layer outputs, wherein the simulated-trained DNN is trained on interactions of a simulated version of the robotic agent with a simulated version of the real-world environment for performing a simulated version of the robotic task; and a first robotic-trained DNN trained on interactions of the robotic agent with the real-world environment for performing the robotic task to determine training values of parameters of the first robotic-trained DNN while holding training values of parameters of the simulated-trained DNN fixed, wherein the first robotic-trained DNN is configured to receive the observations and the one or more layer outputs generated by the simulated-trained DNN and process the observations and the one or more layer outputs to generate a policy output; and wherein the operations comprise: training the first robotic-trained DNN on interactions of the robotic agent with the real-world environment for performing the robotic task to determine training values of parameters of the first robotic-trained DNN while holding training values of parameters of the simulated-trained DNN fixed, wherein the first robotic-trained DNN and the simulated-trained DNN have different capabilities.

37. The one or more non-transitory computer-readable media of claim 36, wherein an output layer of the first robotic-trained DNN implemented by the one or more computers is configured to: receive, as input, (i) a layer output generated by a preceding layer of the output layer of the first robotic-trained DNN and (ii) a layer output generated by a preceding layer of the output layer of the simulated-trained DNN; apply a first set of parameters to the layer output generated by the preceding layer of the output layer of the first robotic-trained DNN; and apply a second set of parameters to the layer output generated by the preceding layer of the output layer of the simulated-trained DNN, and wherein the operations further comprise: initializing values of the second set of parameters to match the training values of the parameters of the output layer of the simulated-trained DNN.

38. The one or more non-transitory computer-readable media of claim 37, wherein, the operations further comprise: initializing values of the first set of parameters to zero.

39. The one or more non-transitory computer-readable media of claim 37, wherein, the operations further comprise: initializing parameter values of layers of the first robotic-trained DNN other than the output layer of the first robotic-trained DNN to random values.

40. The one or more non-transitory computer-readable media of claim 36, wherein, the operations further comprise: training the simulated-trained DNN on interactions of the simulated version of the robotic agent with the simulated version of the real-world environment for performing the simulated version of the robotic task to determine the training values of the parameters of the simulated-trained DNN.

41. The one or more non-transitory computer-readable media of claim 40, wherein, training the simulated-trained DNN includes training the simulated-trained DNN on the interactions of the simulated version of the robotic agent using a first reinforcement learning technique.

42. The one or more non-transitory computer-readable media of claim 41, wherein, training the first robotic-trained DNN includes training the first robotic-trained DNN on the interactions of the robotic agent using a second reinforcement learning technique.

43. A system comprising: one or more computers; and One or more non-transitory computer-readable media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a neural network system comprising a sequence of deep neural networks (DNNs) to determine trained values of parameters of one or more of the DNNs in the sequence of DNNs, wherein the neural network system implemented by the one or more computers is configured to receive an observation characterizing a current state of a real-world environment with which a robotic agent performing a robotic task is interacting and to process the observation to generate a policy output, wherein the observation is visual data, the policy output defining an action to be performed by the robotic agent in response to the observation, wherein the sequence of DNNs comprises: a simulated trained DNN configured to receive the observation and to process the observation to generate one or more layer outputs, wherein the simulated trained DNN is trained on interactions of a simulated version of the robotic agent performing a simulated version of the robotic task with a simulated version of the real-world environment; and a first robot-trained DNN trained on interactions of the robotic agent performing the robotic task with the real-world environment to determine trained values of parameters of the first robot-trained DNN while holding trained values of parameters of the simulated trained DNN fixed, wherein the first robot-trained DNN is configured to receive the observation and the one or more layer outputs generated by the simulated trained DNN and to process the observation and the one or more layer outputs to generate the policy output; and wherein the operations comprise: training the first robot-trained DNN on interactions of the robotic agent performing the robotic task with the real-world environment to determine trained values of parameters of the first robot-trained DNN while holding trained values of parameters of the simulated trained DNN fixed, wherein the first robot-trained DNN and the simulated trained DNN have different capabilities. One or more non-transitory computer-readable media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a neural network system comprising a sequence of deep neural networks (DNNs) to determine trained values of parameters of one or more of the DNNs in the sequence of DNNs, wherein the neural network system implemented by the one or more computers is configured to receive an observation characterizing a current state of a real-world environment with which a robotic agent performing a robotic task is interacting and to process the observation to generate a policy output, wherein the observation is visual data, the policy output defining an action to be performed by the robotic agent in response to the observation, wherein the sequence of DNNs comprises: a simulated trained DNN configured to receive the observation and to process the observation to generate one or more layer outputs, wherein the simulated trained DNN is trained on interactions of a simulated version of the robotic agent performing a simulated version of the robotic task with a simulated version of the real-world environment; and a first robot-trained DNN trained on interactions of the robotic agent performing the robotic task with the real-world environment to determine trained values of parameters of the first robot-trained DNN while holding trained values of parameters of the simulated trained DNN fixed, wherein the first robot-trained DNN is configured to receive the observation and the one or more layer outputs generated by the simulated trained DNN and to process the observation and the one or more layer outputs to generate the policy output; and wherein the operations comprise: training the first robot-trained DNN on interactions of the robotic agent performing the robotic task with the real-world environment to determine trained values of parameters of the first robot-trained DNN while holding trained values of parameters of the simulated trained DNN fixed, wherein the first robot-trained DNN and the simulated trained DNN have different capabilities.

Images