Data-driven robot control

Abstract

Methods, systems, and apparatus for data-driven robot control include computer programs encoded on a computer storage medium. One method includes: maintaining robot experience data; acquiring annotated data; training a reward model on the annotated data; generating task-specific training data for a specific task, including, for each experience in a second subset of experiences in the robot experience data: processing observations in the experiences using the trained reward model to generate reward predictions, and associating the reward predictions with the experiences; and training a policy neural network on task-specific training data for the specific task, wherein the policy neural network is configured to receive network inputs including observations and generate policy outputs that define a control policy for the robot to perform the specific task.

Description

Technical Field

[0001] This manual relates to the use of neural networks to control robots. Background Technology

[0002] A neural network is a machine learning model that uses one or more non-linear units to predict the output from a given input. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as the input for the next layer in the network—the next hidden layer, or output layer. Each layer of the network generates its output from the received input based on the current values ​​of its corresponding set of weights. Summary of the Invention

[0003] This specification describes a system implemented as a computer program on one or more computers at one or more locations, which trains a policy neural network to control a robot, i.e., to select actions to be performed by the robot while it interacts with its environment, so that the robot performs a specific task.

[0004] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages.

[0005] The techniques described in this specification allow systems to reuse past experiences for learning new tasks. Specifically, starting with a limited amount of annotated experience, the system is able to generate a large amount of training data and then train the policy neural network entirely offline for the new task. This limits further wear and tear on the physical robot, as no additional robot interaction is required once a limited amount of annotated experience has been generated. Furthermore, this approach is stable and does not require manually designed rewards for the new task.

[0006] More specifically, robot experience data (also known as endless memory or NES) contains camera and sensor data recorded by the robot and accumulates as the robot (or multiple robots) learns and solves new tasks. The system can use this accumulated data to train a neural network to control the robot to perform new tasks using only minimal demonstration data from the robot or another agent performing the new task, without requiring additional interaction between the robot and its environment. This results in a robust control strategy for the new task and allows the robot to perform the new task effectively.

[0007] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0008] Figure 1 An example neural network training system is shown.

[0009] Figure 2 This is an operation diagram of a neural network training system.

[0010] Figure 3 This is a flowchart of an example process for training a policy neural network.

[0011] Figure 4 This shows a sample user interface for users to annotate task episodes.

[0012] The same reference numerals and reference numerals in different figures indicate the same elements. Detailed Implementation

[0013] Figure 1 An example neural network training system 100 is shown. System 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the systems, components and techniques described below can be implemented.

[0014] System 100 trains a policy neural network 110, which is used to control robot 112, that is, to select actions to be performed by robot 112 when it interacts with environment 114, so that robot 112 performs a specific task. The robot may be, for example, an industrial robot, a warehouse robot, or an autonomous or semi-autonomous vehicle. The environment may be a real-world environment.

[0015] For example, a specific task may include navigating robot 112 to different locations in the environment, locating robot 112 to different objects, picking up or manipulating different objects, or moving different objects to one or more designated locations.

[0016] Each input to the policy neural network 110 can include an observation representing the interaction with the agent, i.e., the robot, and the output of the policy neural network (“policy output”) can define the action to be performed by the agent in response to the observation, for example, defining the output as a probability distribution relative to the possible actions to be performed by the agent, or deterministically defining the output of the action.

[0017] Observations can include, for example, one or more of the following: images, object position data, and sensor data used to capture observations as the agent interacts with the environment, such as data from image, distance, or position sensors, or sensor data from actuators. For example, in the case of a robot, observations can include data characterizing the robot's current state, such as one or more of the following: joint positions, joint velocities, joint forces, torque, or acceleration, for example, gravity-compensated torque feedback, and the global or relative pose of an object held by the robot. In other words, observations can similarly include one or more of the position, linear or angular velocity, force, torque, or acceleration, and global or relative pose of one or more parts of the agent. Observations can be defined in one, two, or three dimensions and can be absolute and / or relative. Observations can also include, for example, sensed electronic signals, such as motor current or temperature signals; and / or image or video data, such as image or video data from camera or lidar sensors, for example, data from sensors of the agent or from sensors located separately from the agent in the environment.

[0018] Actions can be control inputs used to control robots, such as control commands for robot joints or higher levels, or torques for autonomous or semi-autonomous land, air, or sea vehicles, such as torques for vehicle control surfaces or other control elements or higher levels.

[0019] In other words, motion can include, for example, position, velocity, or force / torque / acceleration data of one or more joints of a robot or a part of another mechanical agent. Motion data may additionally or alternatively include electronic control data, such as motor control data, or more generally, data for controlling one or more electronic devices within the environment, the control of which affects the observed environmental state.

[0020] In one example, observations each include one or more images of the environment captured by one or more cameras, such as the robot's camera sensors, one or more cameras located at different locations in the robot's external environment, or both, and low-dimensional proprioceptive features of the robot.

[0021] As a specific example, each input to the policy neural network 110 can include an action and an observation, and the output of the policy neural network 110 can be a Q-value, which represents the predicted reward that the robot will receive as a result of performing an action in response to an observation.

[0022] Rewards refer to a cumulative measure of the rewards received by the agent (robot 112), such as the sum of time-discounted rewards. Typically, rewards are scalar values ​​and characterize, for example, the progress the agent has made in completing a task.

[0023] As a specific example, the reward can be a sparse binary reward, which is zero unless the task is successfully completed, and 1 if the task is successfully completed as a result of performing an action.

[0024] As another specific example, the reward can be a dense reward, which measures the robot's progress in completing the task from the individual observations received during the period of attempting to perform the task. That is, when the environment is in the state represented by the observation, the individual observation can be associated with a non-zero reward value that indicates the robot's progress in completing the task.

[0025] System 100 can then control robot 112 based on the Q-values ​​of actions in the action set, for example, by selecting the action with the highest Q-value as the action to be performed by robot 112.

[0026] As another specific example, each input to the policy neural network 110 can be an observation, and the output of the policy neural network 110 can be a probability distribution over a set of actions, where the probability of each action represents the likelihood that performing that action in response to an observation will maximize the predicted reward. The system 100 can then control the robot 112 based on this probability, for example, by selecting the action with the highest probability as the action to be performed by the robot 112, or by sampling actions from the probability distribution.

[0027] In some cases, to allow for fine-grained control of the agent, system 100 can treat the space of actions to be performed by the robot, i.e., the set of possible control inputs, as a continuous space. Such a setup is called a continuous control setup. In these cases, the output of policy neural network 110 can be parameters of a multivariate probability distribution in the space, such as the mean and covariance of a multivariate normal distribution, or it can directly define the action.

[0028] In some cases, such as in actor-critic type systems, during training, system 100 may utilize a critic neural network that optionally shares some parameters with the policy neural network 110, and is based, for example, on parameters of one or more Q-value distributions of the observer-action input-output as described above, or on the distribution of the critic's Q-values. For example, system 100 may implement a distributed deep deterministic policy gradient reinforcement learning system (arXiv: 1804.08617).

[0029] The policy neural network 110 can have any suitable architecture that allows the policy neural network 110 to process observations to generate policy outputs.

[0030] As a specific example, when the observation includes high-dimensional sensor data, such as image or laser data, the policy neural network 110 can be a convolutional neural network. As another example, when the observation only includes relatively low-dimensional inputs, such as sensor readings characterizing the robot's current state, the policy neural network can be a multilayer perceptron. As yet another example, when the observation includes both high-dimensional sensor data and low-dimensional inputs, the policy neural network 110 can include a convolutional encoder for encoding the high-dimensional data, a fully connected encoder for encoding the low-dimensional data, and a policy subnetwork that operates on the combination of encoded data, such as concatenation, to generate a policy output.

[0031] For example, in one particular implementation, the policy neural network 110 includes a convolutional neural network followed by a spatial softmax layer that encodes the image into a set of keypoint coordinates with onto which proprioceptive features are appended. The policy neural network 110 then embeds the resulting features into a linear layer and applies layer normalization to generate final features. The policy neural network 110 then processes the final features using a policy head implemented as a recurrent neural network to generate a probability distribution or parameters of that probability distribution. A critic neural network, also implemented as a recurrent neural network, is also capable of processing the final features to generate a Q-value output.

[0032] In order to allow system 100 to train neural network 110, system 100 maintains robot experience data 120. Typically, robot experience data 120 is data characterizing the robot's interaction with environment 114.

[0033] Robot experience data 120 includes experience 122, each experience 122 further including observations and actions performed by the robot in response to the observations.

[0034] The robot experience data 120 can include a large amount of experience 122 collected while one or more robots perform various tasks or randomly interact with the environment. However, the robot experience data 120 is generally not associated with task-specific rewards that are needed to train the policy neural network 110 through reinforcement learning. That is, although task-specific rewards are needed to train the policy neural network 110 to control the robot 112 to perform specific tasks, such rewards are not available in the robot experience data 120.

[0035] More specifically, robot experience data 120 will typically include a small amount of demonstration data of a specific task being performed by the robot. However, this demonstration data is not associated with any task-specific rewards for that particular task. See below for reference. Figure 2 Describe how this demo data was generated.

[0036] Furthermore, robot experience data 120 will typically include additional extensive experience data collected while one or more robots are performing different tasks or interacting randomly with their environment. As a specific example, robot experience data 120 can include data collected from interactions between multiple robots performing multiple different tasks simultaneously. For instance, system 100 may have previously trained one or more other policy neural networks to control the robots to perform other tasks, and robot experience data 120 can include any data collected as a result of previous training. Therefore, a large portion of the data in experience data 120 is typically data collected while the robots are performing tasks different from a specific task (or interacting randomly with their environment).

[0037] Therefore, although a large amount of data 120 is available for system 100, system 100 cannot directly use data 120 to train policy neural network 110.

[0038] In order to allow system 100 to train policy neural network 110 using data 120, system 100 obtains annotation data 130, which assigns a corresponding task-specific reward for a specific task to each experience in a first subset 122 of the robot experience data 120.

[0039] Specifically, system 100 obtains annotations for the demonstration data, i.e., for one or more segments in which the robot successfully performs a specific task, and optionally, for one or more segments in which the robot fails to perform a specific task. Further optionally, the system is also able to obtain annotated data on the robot's experience of not attempting to perform a specific task, such as the robot performing different tasks or randomly interacting with the environment.

[0040] As a specific example, system 100 can acquire annotation data through interaction with a set of human users 170. For instance, to obtain a reward for experience in a segment of a task performed by the robot, system 100 can provide a user interface that can be presented to human users 170 on a user device that allows the human users to watch segments of the robot performing the task and to provide instructions to system 100 on how the robot successfully performed the specific task. System 100 can then generate a task-specific reward that identifies the reward.

[0041] The following will refer to Figure 2 and Figure 3 A more detailed description of obtaining the annotated data.

[0042] System 100 trains a reward model 140 on annotated data 130. This reward model 140 receives input observations as input and generates reward predictions as outputs. These predictions are task-specific rewards that should be assigned to a specific task from the input observations. In some implementations, the reward model is a reward ranking model (or more simply, a ranking model). That is, the model is trained to rank rewards within task segments (instances where the robot performs a task), rather than regressing to task-specific rewards in the annotated data.

[0043] Therefore, after training, the trained reward model 140 is able to predict task-specific rewards for observations, even if those observations are not generated while the robot is performing a specific task.

[0044] The reward model 140 can have any suitable architecture that allows the model 140 to process observations to generate reward predictions. In particular, the reward model 140 can have an architecture similar to that of the policy neural network 110, but with a different output layer that allows the reward model 140 to generate outputs as single values ​​rather than potential multi-valued policy outputs.

[0045] System 100 generates task-specific training data 150 for a specific task, which associates each of multiple experiences with a task-specific reward for the specific task.

[0046] Specifically, for each experience in the second subset of experiences in the robot's experience data, system 100 processes the observations in the experience using a trained reward model 140 to generate a reward prediction and associates the reward prediction with the experience.

[0047] The system can generate a set of training data 150, which includes a second subset of experiences associated with reward predictions, and optionally includes a first subset of experiences and associated rewards obtained through annotated data 130.

[0048] In some cases, the second subset of experience does not overlap with the first subset; that is, it only includes experience from data 120 that has not been annotated by data 130 using reward annotations. In other cases, the second subset includes some or all of the data from the first subset, for example, because the reward predicted by the trained reward model 140 may be more accurate for a particular task than the annotations provided by a single human user.

[0049] By generating training data 150 in this manner, system 100 is able to generate a large amount of training data for a specific task based on only a small amount of labeled experience. Including a mixture of task-specific experience and other experience extracted from NES in the second subset can improve the final performance of the trained policy neural network.

[0050] System 100 then trains policy neural network 110 on task-specific training data 150 for a specific task, for example, using heteropolistic reinforcement learning (i.e., generating experience using a policy different from the current policy of the policy neural network). In this implementation, such training is performed completely detached from the policy. The system is able to train policy neural network 110 on a large amount of data with minimal additional robot-environment interactions, i.e., minimal environmental interactions other than those already reflected in the robot experience data 120.

[0051] The following is for reference. Figure 2 and 3 The operation of system 100 is described in more detail from robot experience data 120, that is, from data without any task-specific rewards for a specific task, to training the policy neural network 110.

[0052] After the system 100 has trained the policy neural network 110, the system 100 is able to control the robot 112 while the robot 112 performs a specific task using the trained policy neural network 110.

[0053] Alternatively or additionally, the system can provide data for a specified trained policy neural network 110, i.e., training values ​​of the parameters of the neural network, for use in controlling the robot while it performs a specific task.

[0054] In any of these cases, system 100 can then acquire the experience generated as a result of controlling the robot to perform a specific task using the trained policy neural network 110 and add this experience to robot experience data 120. In this way, the experience becomes available to train another neural network to control the robot to perform another task. Therefore, the amount of experience in robot experience data 120 can continue to increase and continue to be reused by system 100 to learn new tasks, even if no experience data 120 may be associated with the task reward of the new task or generated while the robot is performing the new task.

[0055] Optionally, more annotated data can be obtained for new empirical data, and the training process can be iterated to further train the policy neural network.

[0056] Figure 2 Figure 200 shows the stages of training the policy neural network 110.

[0057] In Figure 200, the training phases are divided into those phases that require robot control (phases A, F, and G depicted above the dashed lines dividing the graph into "robot" and "cloud") and those phases that can be performed without robot control, for example, in the cloud or on a different computer system (phases B, C, D, and E depicted below the dashed lines dividing the graph into "robot" and "cloud").

[0058] The training phases are also divided into those phases specific to the specific task that the policy neural network 110 is being trained on (phase CG and a portion of phase A in the "task-specific" region outside the "task-independent" box) and those phases that are not related to the task that the policy neural network is being trained on, i.e., those phases that can be performed without knowing the specific task (phase B and an optional portion of phase A inside the "task-independent" box).

[0059] In addition, these stages are divided into stages that require user intervention (stages A and C in the "Human Operator" area outside the "No Human Operator" box) and those stages that are unrelated to the task the policy neural network is being trained on, i.e., those stages that can be performed without knowing the specific task (stages B and DG in the "No Human Operator" box).

[0060] In Phase A, the system collects demonstration data on the robot performing specific tasks. For example... Figure 2 As illustrated in the example, the collection of demonstration data involves a "human operator," such as a user who remotely operates the robot to perform a specific task. For example, system 100 can allow a user to control the robot by submitting control inputs to the robot via a network connection while viewing video of the environment transmitted via a network connection, for example, from the robot's perspective or from a third-person perspective relative to the robot, and can record the control inputs, i.e., actions and observations corresponding to different frames in the video, to generate demonstration data.

[0061] However, in other examples, the system is able to collect demonstration data without a human operator, for example, by using simpler control strategies, such as hard-coded, rule-based strategies, to control the robot.

[0062] In addition, during phase A, the system can optionally collect data on the robot's random interactions with the environment or its exploration of the environment according to certain exploration strategies; that is, data unrelated to the task.

[0063] In Phase B, the system will demonstrate that the data is stored in Never Endless Memory (NES), that is, stored in storage. Figure 1 The collected robot experience data is stored in one or more physical or logical data storage devices. (See above reference.) Figure 1The robot experience data stored in the perpetual memory can include experience collected simultaneously by different robots performing different tasks, i.e., tasks different from a specific task. This memory is called "perpetual" because experience generated as a result of training a neural network for a task can be continuously reused to train the neural network for new tasks using the described techniques. However, in practice, experience collection may cease once the policy neural network has been satisfactorily trained.

[0064] In phase C, the system acquires annotation data, such as annotation data generated by user (“human operator”) submitting annotation input to the system. The annotation data includes annotations defining task-specific rewards for the demonstration data and, optionally, other experiential data in perpetual memory. References will follow below. Figure 3 A more detailed description of obtaining the annotated data.

[0065] In phase D, the system learns on annotated data, i.e., trains the reward model R. As described above, the reward model maps received observations to predicted task-specific rewards for a particular task. Therefore, after training, the reward model can be used to predict task-specific rewards, even for experience not collected while the robot was performing a particular task.

[0066] The following will refer to Figure 3 A more detailed description of training the reward model.

[0067] In phase E, the system learns the policy (“Q / pi”) used to control the agent. Specifically, refer to the above and below. Figure 3 As described in more detail, the system generates task-specific training data using the trained reward model, and then trains a policy neural network on the task-specific training data. The system is then able to use the policy defined by the output of the trained policy neural network as the final policy for controlling the agent.

[0068] Optionally, the system can then execute stages F and G.

[0069] In stage F, the system performs a specific task segment, that is, controls the robot using the learned policy, i.e., uses the trained policy neural network to perform the specific task. For example, the system can enable the robot to perform a specified number of task segments while controlling the robot using the trained policy neural network, i.e., enabling the robot to perform actions defined by the output of the trained policy neural network.

[0070] Then, in stage G, the system evaluates the performance of the learned policy, i.e., the performance of the trained policy neural network, to determine, for example, whether the learned policy is of sufficiently high quality for deployment in a production environment. As an example, the system can evaluate the policy neural network based on the proportion of successfully executed task segments, the average reward obtained per task segment, or whether any task segment results in unwanted wear and tear on the robot or damage to other objects in the environment.

[0071] The system can then add the experience data generated as a result of execution phase F to the perpetual memory, i.e., for training new policy neural networks for new tasks. If the system determines that the learned policy is of sufficiently high quality for deployment, it can also add experience generated after the policy has been deployed to the perpetual memory, provided the system has access to this experience.

[0072] Therefore, as more policy neural networks are trained for more new tasks, the endless memory can continue to grow.

[0073] Figure 3 This is a flowchart of an example process 300 for training a policy neural network. For convenience, process 300 will be described as being executed by a system of one or more computers located in one or more locations. For example, a suitably programmed neural network training system, such as... Figure 1 The neural network training system 100 is capable of executing process 300.

[0074] The system maintains robot experience data characterizing the robot's interaction with the environment (step 302). Specifically, the data includes experience, each experience of which in turn includes observations and corresponding actions performed by the robot in response to the observations.

[0075] The system obtains annotation data, which assigns task-specific rewards for a specific task to each experience in the first subset of experiences in the robot's experience data (step 304).

[0076] Typically, the first subset of experience in robot experience data includes demonstration experience collected when the robot performs one or more segments of a specific task.

[0077] For example, a first subset of experience could include experience collected while the robot is being controlled by a user, e.g., remotely operated, and performing one or more segments. Alternatively or additionally, a first subset of experience could include experience collected while the robot is controlled by different computer systems using different strategies, e.g., strategies based on hard-coded rules or simpler machine learning strategies.

[0078] In some implementations, the first subset of experience can also include experience collected when the robot performs tasks different from the specific task or interacts randomly with the environment. That is, the system can obtain annotated data that assigns task-specific rewards for a specific task to the experience collected while the robot performs different tasks. This ensures that the annotated data better covers the state space of all possible states of the environment.

[0079] In order to obtain annotation data, the system is able to receive annotation input from one or more users, who are rewarded for each of the defined experiences in the first subset.

[0080] In order to obtain a given annotation input, the system is able to provide a representation of one or more experiences from a first subset of the experience data to be presented to the user in a user interface for presentation on the user device, and then obtain input submitted by the user through the user device, which defines a reward for one or more experiences.

[0081] As a specific example, when the reward is sparse, the system can provide video of the robot performing a task segment for presentation. The user can then submit input indicating whether the task was successfully performed, and the system can map the input to a reward value of 1 for the last experience in the segment if the task was successfully performed, and to a reward value of 0 for the last experience in the segment if the task was not successfully performed.

[0082] As another specific example, when the reward is a dense reward, the system can provide video of segments in which the robot performs a task for presentation, and the user can submit input that correlates individual frames of the video—that is, individual experiences within a segment—with a measure of the agent's progress in completing the task from the start of the frame. The system can then map these inputs to dense reward values.

[0083] For example, the system can present a user interface that allows users to draw curves indicating progress in completing a specific task over time, with the interface displaying the frame corresponding to the current cursor position. The system can then map the user-drawn curves to curves of numerical reward values, which are appropriately scaled to obtain task-specific rewards for experience within a given segment. Allowing users to submit annotations by drawing curves while watching video allows them to accurately capture the robot's progress without needing to individually mark every frame of the video with reward values.

[0084] Figure 4 A sample user interface 400 is shown, illustrating the user's annotation of task segments.

[0085] As in Figure 4As seen in the example, the user has, for example, used an input device or by submitting touch input to plot a reward curve 410, which indicates the robot's progress in completing the task at any given time within the segment while watching the video of the segment. Figure 4 In the example, the dashed line 430 at the top of interface 400 indicates that the robot has successfully performed the task. Figure 4 As shown, the dot plot represents a video frame displayed to the user at a given point along the reward curve 410. For example, point 420 is located in a segment where the user, while viewing frame 440, indicates that the robot is still far from completing the task; that is, the point on the curve is far from the success region 430. Frame 440, displayed to the user at this time corresponding to point 420, indicates that the robot has not yet interacted with any objects in the environment. Therefore, by interacting with interface 400 to plot the reward curve 410, the user can easily utilize reward annotations for all observations in a segment without explicitly assigning numerical reward values ​​to the observations.

[0086] Return to Figure 3 The system trains a reward model on annotated data. The reward model receives input observations as input and generates reward predictions as outputs. These reward predictions are predictions of task-specific rewards that should be assigned to the input observations for a specific task (step 306).

[0087] Typically, the system trains the reward model using an in-segment approach, rather than training the model to directly regress task-specific rewards. This can be beneficial when multiple users submit annotated data because while annotators may have high consistency in relative experience quality (in terms of the value of completing a specific task) within a segment, they are often inconsistent in the overall scale of task-specific rewards.

[0088] As a specific example, given two experiences in the same event, the system trains a reward model such that the reward predictions for the two experiences satisfy two conditions.

[0089] First, if the first experience is successful based on the task-specific reward, then the estimated reward generated by the reward model should also be considered successful; if the first experience is unsuccessful based on the task-specific reward, then the estimated reward generated by the reward model should also be considered unsuccessful. When the task-specific reward for an experience is higher than the annotation success threshold, the system can determine that the experience is successful based on the task-specific reward; and when the task-specific reward for an experience is lower than the annotation failure threshold, the system can determine that the experience is unsuccessful based on the reward prediction. Similarly, when the reward prediction for an experience is higher than the prediction success threshold, the system can determine that the experience is successful based on the reward prediction; and when the reward prediction for an experience is lower than the prediction failure threshold, the system can determine that the experience is unsuccessful based on the reward prediction. The threshold is a hyperparameter used in training the reward prediction model.

[0090] Secondly, if the task-specific reward for the first experience is higher than the task-specific reward for the second experience by a first threshold, then the predicted reward for the first experience should be higher than the predicted reward for the second experience by a second threshold, where the first threshold and the second threshold are hyperparameters for training the reward prediction model.

[0091] In particular, to satisfy these conditions, the system trains the reward model to optimize the so-called hinge loss function, which measures the difference in reward predictions between different experiences from the same task segment. In other words, the reward model can be considered a (reward) ranking model that ranks rewards within the same task segment.

[0092] More specifically, the hinge loss function can be the sum or weighted sum of two components.

[0093] The first component trains the model to generate a reward prediction that satisfies the second condition, and for a given pair of experiences from the same event, it can be equal to 0 if the task-specific reward for the first experience is not higher than the task-specific reward for the second experience by a first threshold, and can be equal to (i) zero or (ii) the difference between the predicted reward for the first experience and the predicted reward for the second experience plus the second threshold if the task-specific reward for the first experience is higher than the task-specific reward for the second experience by a first threshold.

[0094] The second component trains the model to generate reward predictions that satisfy the first condition and are able to include two items.

[0095] For a given experience, if the experience is unsuccessful according to the task-specific reward, the first term can be equal to zero, and if the experience is successful according to the task-specific reward, it can be equal to the largest of the following: (i) zero or (ii) the difference between the predicted success threshold and the predicted reward for the experience.

[0096] For a given experience, if the experience is not unsuccessful based on the task-specific reward, then the second term can be equal to zero, and if the experience is unsuccessful based on the task-specific reward, then it is equal to the largest of the following two: (i) zero or (ii) the difference between the predicted reward for the experience and the predicted failure threshold.

[0097] The system can use supervised learning to train a reward model to minimize this loss function.

[0098] The system generates task-specific training data for a specific task, and uses the trained reward model to associate each of the multiple experiences with a task-specific reward for the specific task (step 308).

[0099] Specifically, the system is able to select a second subset of experiences from the robot's experience data, and for each experience in the second subset, process the observations in the experience using a trained reward model to generate a reward prediction and associate the reward prediction with the experience. This produces labeled experiences associated with task-specific rewards, i.e., reward predictions generated by the trained reward model. Therefore, the system is able to generate hundreds of hours of useful training data using, for example, a relatively small amount of annotated data from human demonstrations.

[0100] Then, the system trains a policy neural network on task-specific training data for a specific task (step 310).

[0101] For example, the system can train a policy neural network on a specific task using off-policy reinforcement learning techniques. Because this technique is "off-policy," meaning it does not require generating any experience on the neural network being trained on using the current version of the neural network, the system can train a policy neural network on task-specific training data without using the neural network to control the robot to perform a specific task.

[0102] This system is capable of training policy neural networks using any suitable heterogeneous policy reinforcement learning technique. As a specific example, the system is capable of using batch reinforcement learning techniques, such as a technique that relies on distributed policy gradients to update the policy neural network, or a technique that relies on non-distributed deterministic policy gradients to update the policy neural network. Batch reinforcement learning is described in more detail below: Sascha Lange, Thomas Gabel, and Martin Riedmiller, Batchreinforcementlearning, In Reinforcementlearning, pp. 45–73, Springer, 2012. The following text describes distributed policy gradients and non-distributed deterministic policy gradients in more detail: Gabriel Barth-Maron, Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, DhruvaTB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap, Distributed distributed deterministic policy gradients, In the International Conference on Learning Representation, 2018.

[0103] This specification uses the term "configured" in relation to system and computer program components. For a system of one or more computers configured to perform a specific operation or action, this means that software, firmware, hardware, or a combination thereof are installed on the system, which, in operation, causes the system to perform the operation or action. For one or more computer programs configured to perform a specific operation or action, this means that the one or more programs include instructions that, when executed by a data processing device, cause that device to perform the operation or action.

[0104] The embodiments of the subject matter and functional operation described in this specification can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in one or more combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof. Alternatively or additionally, program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical signals, optical signals, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiving device for execution by the data processing apparatus.

[0105] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The device may also be or further include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0106] A computer program, also known as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages; and the computer program can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A program may, but is not necessarily, correspond to a file in a file system. A program can be stored as a portion of a file that holds other programs or data, for example, as one or more scripts stored in a markup language document, in a single file dedicated to the program involved, or in multiple collaborating files, such as a file storing one or more modules, subroutines, or code portions. A computer program can be deployed to execute on a single computer or on multiple computers located in one location or distributed across multiple locations and interconnected by a communication network.

[0107] In this specification, the term "database" is used broadly to refer to any collection of data: data that does not need to be structured in any particular way, or not at all, and that can be stored on storage devices in one or more locations. Thus, for example, an indexed database can comprise multiple collections of data, each of which can be organized and accessed in different ways.

[0108] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and run on the same one or more computers.

[0109] The processes and logic flows described in this specification can be executed by one or more programmable processors, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuitry, such as FPGAs or ASICs, or by a combination of special-purpose logic circuitry and one or more programmed computers.

[0110] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and memory can be supplemented by or incorporated into special-purpose logic circuitry. Typically, a computer will also include, or be operatively coupled to, receiving data from or transferring data to one or more mass storage devices for storing data, or both, such as magneto-, magneto-optical, or optical discs. However, a computer does not necessarily have to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, GPS receiver, or portable storage device, such as a Universal Serial Bus (USB) flash drive, to name a few.

[0111] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, such as: semiconductor storage devices, such as EPROM, EEPROM, and flash memory devices; disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0112] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having: a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user; and a keyboard and pointing devices such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents to and from a device used by the user; for example, by sending a webpage to a web browser in response to a request received from a web browser on the user's device. Furthermore, the computer can interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving response messages from the user in return.

[0113] The data processing apparatus used to implement machine learning models can also include, for example, dedicated hardware accelerator units, which are common and computationally intensive components used to process machine learning training or production (i.e., inference, workloads).

[0114] It is possible to use machine learning frameworks, such as TensorFlow, Microsoft Cognitive Toolkit, Apache Singa, or Apache MXNet, to implement and deploy machine learning models.

[0115] Embodiments of the subject matter described herein can be implemented in computing systems that include, for example, backend components as data servers, or middleware components such as application servers, or frontend components such as client computers with graphical user interfaces, web browsers, or apps, or any combination of one or more such backend, middleware, or frontend components, through which users can interact with embodiments of the subject matter described herein via the graphical user interface, web browser, or app. The components of the system can be interconnected via digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”).

[0116] A computing system can include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is established by means of computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data, such as HTML pages, to a user device for purposes such as displaying data to a user interacting with the device as a client and receiving user input from it. It is possible to receive data generated on the user device, such as the result of user interaction, from the device at the server.

[0117] While this specification contains numerous specific details of implementation, these should not be construed as limiting the scope of any invention or the scope of claims, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented separately or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases, one or more features from the claimed combination can be removed from the combination, and the claimed combination can be adapted for sub-combinations or variations thereof.

[0118] Similarly, although operations are depicted in the accompanying drawings and described in a specific order in the claims, this should not be construed as requiring such operations to be performed in the specific order shown or sequentially, or that all illustrated operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0119] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in different orders and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method, comprising: Maintain robot experience data characterizing robot interaction with the environment, the robot experience data including multiple experiences, each of the multiple experiences including an observation and an action performed by the corresponding robot in response to the observation; Obtain annotation data, which assigns a task-specific reward for a specific task to each experience in a first subset of the robot's experience data, wherein the first subset of experiences includes experiences from multiple different task segments of the specific task. A reward model is trained on the annotated data. The reward model receives input observations as input and generates reward predictions as output. The reward predictions are predictions of task-specific rewards that should be assigned to the input observations for a specific task. Training the reward model includes training the reward model to optimize a loss function that measures the following for a given pair of experiences from the same task segment in the plurality of different task segments: (i) the difference between the corresponding reward prediction generated by the reward model for a first observation in a first experience in the given pair and the corresponding reward prediction generated by the reward model for a second observation in a second experience in the given pair; and (ii) the difference between the corresponding task-specific reward for the first experience in the given pair and the corresponding task-specific reward for the second observation in the second experience in the given pair. Task-specific training data is generated for the specific task, which associates each of a plurality of experiences with a task-specific reward for the specific task, including for each experience in a second subset of the experiences in the robot's experience data: The observed information in the experience is processed using the trained reward model to generate a reward prediction, and Associate the reward prediction with the experience; as well as A policy neural network is trained on task-specific training data for the specific task, wherein the policy neural network is configured to receive network inputs including observations and generate policy outputs that define a control policy for the robot to perform the specific task.

2. The method according to claim 1, further comprising: The robot is controlled while performing the specific task using a trained policy neural network.

3. The method according to claim 1, further comprising: Provide data for a specified trained policy neural network to control the robot while it performs the specific task.

4. The method of claim 2, further comprising: To obtain experience as a result of controlling the robot to perform the specific task using the policy neural network; as well as Add the experience to the robot's experience data.

5. The method according to claim 1, wherein, The robot experience data includes data collected from interactions between multiple robots while performing multiple different tasks.

6. The method according to claim 1, wherein, The experience in the robot's experience data is not associated with any reward for any of the plurality of different tasks.

7. The method according to claim 5, wherein, The second subset of empirical data is collected as a result of the robot performing one or more tasks different from the specific task described above.

8. The method according to claim 1, wherein, Training the policy neural network includes using heterogeneous policy reinforcement learning techniques to train the policy neural network.

9. The method according to claim 1, wherein, The first subset of the experience data includes demonstration experiences collected during one or more segments of the robot performing the specific task.

10. The method according to claim 9, wherein, The robot is controlled by the user while performing one or more segments of the specific task.

11. The method according to claim 1, wherein, The annotation data obtained includes: Provide a representation of one or more experiences from the first subset of the experience data to present to the user; and The user provides input defining the reward for the one or more experiences.

12. The method according to any one of claims 1 to 11, wherein, The loss function includes another term, which is the difference between a predicted success threshold for a given experience measure of success based on the experience, and the corresponding predicted reward generated by the reward model for a given observation in the given experience.

13. A system comprising one or more computers and one or more storage devices storing instructions, the instructions, when executed by the one or more computers, causing the one or more computers to perform operations of a corresponding method according to any one of claims 1 to 12.

14. A computer-readable storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operation of a corresponding method according to any one of claims 1 to 12.

Images