Controlling agents using tokenized goal images

Abstract

Methods and systems for controlling agents, e.g., robots, using tokenized goal images. One of the methods includes receiving a goal image; tokenizing the goal image to generate a plurality of visual tokens; at each of a plurality of time steps: obtaining one or more observation images characterizing a state of the environment at the time step; tokenizing each of the one or more observation images; generating a sequence of input tokens that comprises the plurality of visual tokens that represent the goal image and the plurality of visual tokens that represent the one or more observation images; processing the sequence of input tokens to generate an output sequence of output tokens from the discrete vocabulary of tokens that represents an action to be performed by the agent in response to the observation images; and causing the agent to perform the selected action.

Description

CONTROLLING AGENTS USING TOKENIZED GOAL IMAGESCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to GR Application No. 20230100490, filed on June 20, 2023. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application.BACKGROUND

[0002] This specification relates to processing data using machine learning models.

[0003] Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.

[0004] Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output.SUMMARY

[0005] This specification generally describes a system implemented as computer programs on one or more computers in one or more locations that controls an agent, e.g., a robot or other mechanical agent, that is interacting in an environment (e.g., a real-world environment) by selecting actions to be performed by the agent and then causing the agent to perform the actions. When the agent is a robot, the robot can be any appropriate type of robot, e.g., a robotic arm, a quadruped robot, a vehicular robot, e.g., an autonomous vehicle, and so on.

[0006] In particular, the system controls the agent to perform a particular task based on outputs generated by a sequence modeling neural network that has been trained to control the agent in response to (i) observation images characterizing the environment and (ii) a goal image characterizing a goal state of the environment, e.g., of a state of the environment in which the particular task is successfully completed.

[0007] More specifically, the system tokenizes the observation images and the goal image to generate (i) visual tokens representing the observation images and (ii) visual tokens representing the goal image and then conditions the sequence modeling neural network on the visual tokens.

[0008] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0009] Many existing systems for controlling robots and other agents utilize task-specific policies that can each control the agent to perform a single corresponding task. This is because, even though the cost of task design and robot experience generation is very high, leveraging heterogeneous robot data at scale has remained a challenging problem in the field of robotics.

[0010] This specification, on the other hand, describes a self-improving generalist policy that can control a robot or other agent to perform any of a variety of tasks, e.g., a variety of vision-based robotic manipulation tasks.

[0011] In particular, this specification describes techniques for conditioning a sequence modeling neural network, e.g., a large transformer sequence model, on tokenized representations of goal images and observation images. By employing this conditioning, the neural network can effectively adapt, e.g., through fine-tuning, to a broad set of new downstream tasks.

[0012] For example, the same model can be used to control robots to perform multiple precise and dexterous vision-based tasks, including tasks that are performed with different robot embodiments, i.e., physical configurations of robots, with different degrees of freedom, various observation and action specifications, and operating at different control frequencies. The described agent handles these variations natively and without requiring common action or observation representations between the tasks by leveraging (i) the sequence modeling neural network’s ability to input and output variable-length sequences based on context and (ii) the conditioning on the goal image. In other words, by representing images and actions as discrete tokens and using the sequence modeling neural network to process input sequences of tokens and output sequences of tokens and by conditioning the sequence modeling neural network on visual tokens representing goal images, the system can use the same neural network to flexibly control different robots to perform multiple different tasks.

[0013] More specifically, the described neural network is able to successfully adapt to multiple new tasks - including new robot embodiments, unseen behaviours, objects and perceptual variants, and sim-to-real - via fine-tuning on a small dataset of new episodic experience, e.g., as few as between 100 to 1000 demonstrations. This significantly reduces the cost of acquiring new skills and onboarding new embodiments. That is, because the neural network does not need to be re-trained from scratch for each new task, even if it has acorresponding new robot embodiment, and can instead be fine-tuned on a very small amount of training data.

[0014] This specification also describes how the neural network can be used to gather additional data that is later added to train new iterations of the agent. This self-improvement process makes for a more capable agent, improving the system’s cross-task transfer and finetuning capabilities to even more tasks, and demonstrating better performance on existing tasks.

[0015] In some implementations, the system makes use of vector quantization (VQ), e.g., a VQ-GAN encoder (Vector Quantized Generative Adversarial Networks encoder), when tokenizing input images. This choice of encoder enables fast training and iteration.Moreover, by pre-training the encoder on a large set of diverse images that include both general-purpose images and agent-specific images, the system can improve the generalization of the sequence modeling neural network to new tasks, e.g., that require interacting with different types of environment or objects.

[0016] Additionally, by specifying tasks via visual goal-conditioning, the system achieves the desirable property that any image in a trajectory can be labelled as a valid “hindsight goal” for all time steps leading up to it. This means that hindsight goals in existing data can be extracted without additional human supervision and that even suboptimal data collected by the agent can be incorporated back into the training set for self-improvement. For example, the system can “relabel” a successful training sequence with the final state from another successful, semantically similar trajectory to improve the diversity of the training data. Additionally, visual goals provide an intuitive interface to indicate to the robot which task it should perform.

[0017] Moreover, this specification describes techniques for autonomously performing task resets, i.e., for switching from one task to another, e.g., when generating training data. This significantly increases the amount of training data that can be collected in a given amount of time and with a given amount of real-world environment interaction while minimizing wear and tear and damage risk to the physical robot.

[0018] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0019] FIG. 1 shows an example action selection system.

[0020] FIG. 2 is a flow diagram of an example process for controlling an agent.

[0021] FIG. 3 is a flow diagram of an example process for training the sequence modeling neural network.

[0022] FIG. 4 shows an example training pipeline.

[0023] FIG. 5 is a flow diagram of an example process for determining whether to switch tasks when controlling an agent.

[0024] FIG. 6 shows an example of the performance of the described techniques.

[0025] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0026] FIG. 1 shows an example action selection system 100. The action selection system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0027] The action selection system 100 uses a sequence modeling neural network 120 to control an agent 104 interacting with an environment 106 to perform a task in the environment 106. For example, the agent 104 can be a robot, e.g., a robotic arm, a quadruped robot, a humanoid robot, or other type of robot that is controllable by the system 100.

[0028] Examples of agents, environments, and tasks will be described below.

[0029] When controlling the agent 104, the system 100 controls the agent 104 to accomplish a task by selecting actions 108 to be performed by the agent 104 at each of multiple time steps during the performance of an episode of the task.

[0030] An “episode” of a task is a sequence of interactions during which the agent attempts to perform an instance of the task starting from some starting state of the environment. In other words, each task episode begins with the environment being in an initial state, e.g., a fixed initial state or a randomly selected initial state, and ends when the agent has successfully completed the task or when some termination condition is satisfied, e.g., the environment enters a state that has been designated as a terminal state or the agent performs a threshold number of actions without successfully completing the task.

[0031] At each time step during any given task episode, the system 100 receives an observation 110 characterizing the current state of the environment 106 at the time step and, in response, selects an action 108 to be performed by the agent 104 at the time step. After the agent 104 performs the action 108, the environment 106 transitions into a new state.

[0032] The observation 110 can include any appropriate information that characterizes the state of the environment. As one example, the observation 110 can include sensor readings from one or more sensors configured to sense the environment. For example, the observation 110 can include one or more images captured by one or more cameras, measurements from one or more proprioceptive sensors, and so on. As used herein, an image may be any still or moving image, i.e., the image may be part of a video, in 2D or 3D, and may be a monochrome, color or hyperspectral image, i.e., comprising monochrome or color pixels. As defined herein, an “image” includes a point cloud e.g. from a LIDAR system, and a “pixel” includes a point of the point cloud. The image may, for example, represent physical objects in a real -world environment.

[0033] In some cases, the system 100 receives an extrinsic reward 152 (also referred to as a “task” reward) from the environment in response to the agent performing the action.

[0034] Generally, the reward is a scalar numerical value and characterizes a progress of the agent towards completing the task.

[0035] As a particular example, the reward can be a sparse binary reward that is zero unless the task is successfully completed and one if the task is successfully completed as a result of the action performed.

[0036] As another particular example, the reward can be a dense reward that measures a progress of the agent towards completing the task as of individual observations received during the episode of attempting to perform the task, i.e., so that non-zero rewards can be and frequently are received before the task is successfully completed.

[0037] In some cases, the reward can be generated by a reward model. As one example of this, the reward model may be learned using a success detector that detects successful behavior from observations of the environment, e.g., as described in “Vision-Language Models as Success Detectors” arXiv:2303.07280.

[0038] More specifically, to control the agent 104 to perform a particular task, the system 100 receives a goal image 130 characterizing a goal state of the environment 106, e.g., of a state of the environment 106 in which the particular task is successfully completed.

[0039] The goal image 130 can, for example, be received from another agent in the environment or from a control system of the agent 104. For example, another agent in the environment 106 can capture an image of the environment 106 and can provide it to the system 100. As another example, the control system can receive an input from a user specifying the goal image 130, e.g., an image that has been captured by a camera of the user or that the user has selected from a library of goal images.

[0040] The system 100 tokenizes the goal image 130 to generate a plurality of visual tokens 132 that represent the goal image 130.

[0041] Each visual token 132 is selected from a discrete vocabulary of tokens, e.g., a vocabulary that includes a fixed number of visual tokens that represent images and, optionally, a fixed number of additional tokens that each represent a respective discretized numerical value.

[0042] Tokenizing images will be described in more detail below.

[0043] In order to then perform the task, at each of a plurality of time steps, the system 130 uses the sequence modeling neural network 120 to process at least the visual tokens 132 representing the goal image and tokens representing the observation 110 at the time step to select the action 108 to be performed by the agent 104 at the time step. For example, the sequence modeling neural network 120 can represent the action 108 as a sequence of discretized tokens, i.e., as a sequence of tokens that each represent a discretized numerical value.

[0044] The sequence modeling neural network 120 can have any appropriate architecture that allows the neural network to map an input sequence of tokens from a vocabulary to an output sequence of tokens from the vocabulary.

[0045] As a particular example, the sequence modeling neural network 120 can be an autoregressive neural network, e.g., a causal Transformer neural network e.g., an encoder-decoder Transformer neural network with a causal Transformer decoder or a causal decoder-only Transformer neural network, that auto-regressively generates the tokens in the output sequence.

[0046] In other words, in these examples the sequence modeling neural network 120 can be an auto-regressive neural network that generates output sequences of tokens, e.g., an autoregressive self-attention neural network that includes causally-masked self-attention layers.

[0047] In some implementations, the auto-regressive neural network 120 includes a transformer neural network. In general, a transformer neural network may be a neural network that has a succession of self-attention neural network layers. For example, the neural network 120 can have the architecture of a large language model (LLM) neural network.

[0048] A self-attention neural network layer has an attention layer input for each element of the input and is configured to apply an attention mechanism over the attention layer input to generate an attention layer output for each element of the input; there are many possible attention mechanisms that may be used. The transformer neural network may be referred toas a decoder-only Transformer neural network in the sense that it can be based on just the decoder of the original transformer architecture (Vaswani et al., 2017 arXiv: 1706.03762).

[0049] In more detail, the neural network 120 can be referred to as an auto-regressive neural network, i.e., because the neural network auto-regressively generates an output sequence of tokens. More specifically, the auto-regressively generated output is created by generating each particular token in the output sequence conditioned on a current input sequence that includes at least some of the tokens that precede the particular token in the output sequence, i.e., the tokens that have already been generated for any previous positions in the output sequence that precede the particular position of the particular token.

[0050] For example, the neural network 120 can be an auto-regressive attention neural network that includes (i) a plurality of attention blocks that each apply a self-attention operation and (ii) an output subnetwork that processes an output of the last attention block to generate a score distribution over tokens in the vocabulary, e.g., a score distribution used for selecting an output token, e.g., by sampling from the score distribution or selecting a most likely token according to the score distribution. The self-attention operation applied by some or all of the attention blocks can be causally-masked, so that, for each position in the sequence, only the tokens at the position and at positions preceding the position are assigned non-zero attention weights.

[0051] In this example, the neural network 120 can have any of a variety of Transformerbased neural network architectures. Examples of such architectures include those described in J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models, arXiv preprint arXiv:2203.15556, 2022; J.W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, H. F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A.Wu, E. Eisen, S. M. Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Martens, X. L. Li, A. Kuncoro, A.Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d’ Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, B. A. Hechtman, L. Weidinger, I. Gabriel, W. S. Isaac, E. Lockhart, S. Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving. Scaling language models:Methods, analysis & insights from training gopher. CoRR, abs / 2112.11446, 2021; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 1910.10683, 2019; Daniel Adiwardana, Minh- Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like opendomain chatbot. CoRR, abs / 2001.09977, 2020; and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.

[0052] More specifically, the neural network 120 can include a plurality of layers that include a plurality of attention layers.

[0053] Each attention layer receives a respective hidden state for each of the input positions in the input sequence and updates the respective hidden states for each of the input positions by applying an attention mechanism to the respective hidden states. That is, each hidden state corresponds to a respective input token within the input sequence, i.e., the hidden state for a given input position corresponds to the token at the given input position.

[0054] For the first attention layer in the neural network 120, the respective hidden states can be the input tokens in the input sequence or the input tokens after having been modified by one or more initial layers of the neural network. For the subsequent attention layers in the sequence, the respective hidden states can be the outputs of a preceding layer within the attention neural network.

[0055] As a particular example, the system 100 can use a neural network having the architecture described in Reed, et al, “A Generalist Agent”, arXiv:2205.06175, the entire contents of which are hereby incorporated by reference herein in their entirety.

[0056] Generally, an action or other data item may be tokenized by converting the elements of the data item into respective tokens from a vocabulary that is made up of a discrete set of tokens, with each token having a respective predetermined or learned embedding. An embedding is an ordered collection of numerical values having a pre-determined dimensionality, e.g., a vector of floating point or other numeric values having a predetermined number of entries.

[0057] An action may be discrete or continuous and may include multiple action dimensions. For example, each action dimension can correspond to a respective degree of freedom (DoF) of a robot being controlled by the system 100. In these cases, the system canconvert the action into tokens by discretizing each of the action dimensions so that the value of each of the action dimensions corresponds to one of the tokens in the vocabulary.

[0058] Using the neural network 120 to generate the action 108 will be described in more detail below.

[0059] When the agent 104 is a robot or other mechanical agent, the system 100 can be implemented local to the agent 104 or remote from the agent 104.

[0060] When the system is local to the mechanical agent, the system 100 can directly control the agent to perform the action 108.

[0061] When the system 100 is remote from the agent 104, the control system of the agent, e.g., the robot, can obtain the observation 110 and then provide the observation 110 to the system 100, e.g., over a data communication network. The system 100 can then select the action 108 as described above and transmit data identifying the action 108 to the control system of the agent, which can control the agent using the identified action 108.

[0062] Some examples of agents and environments now follow.

[0063] In some implementations, the environment is a real-world environment and the agent is a mechanical agent interacting with the real-world environment to perform a particular task. For example, the agent may be a robot interacting with the environment to accomplish a goal, e.g., to locate an object of interest in the environment, to move an object of interest to a specified location in the environment, to physically manipulate an object of interest in the environment in a specified way, or to navigate to a specified destination in the environment; or the agent may be an autonomous or semi-autonomous land, air, or sea vehicle navigating through the environment to a specified destination in the environment.

[0064] The actions may be control inputs to control a robot, e.g., torques for the joints of the robot or higher-level control commands, or the autonomous or semi-autonomous land or air or sea vehicle, e.g., torques to the control surface or other control elements of the vehicle or higher-level control commands.

[0065] For example, as described above, an action can include one or more action dimensions for each of multiple DoFs of the robot. As one example, when the robot is a robotic arm, some of the action dimensions can correspond to controls for the end-effector of the robot and one or more of the action dimensions can correspond to controls for the gripper of the robot or of a robotic hand of the robot. As a particular example, the robot can be a 5- DoF robot that has 4 DoFs for the end-effector and 1 for a gripper. As another particular example, the robot can be a 7-DoF robot that has 6 DoFs for the end-effector and 1 for agripper. As another particular example, rather than having one DoF for the gripper, the robot can have 3 or more DoFs for a robotic hand.

[0066] In other words, the actions can include for example, position, velocity, or force / torque / accel eration data for one or more joints of a robot or parts of another mechanical agent. Actions may additionally or alternatively include electronic control data such as motor control data, or more generally data for controlling one or more electronic devices within the environment the control of which has an effect on the observed state of the environment. For example in the case of an autonomous or semi-autonomous land, air, or sea vehicle the actions may include actions to control navigation e.g. steering, and movement e.g., braking and / or acceleration of the vehicle.

[0067] In some implementations the environment is a simulated environment and the agent is implemented as one or more computer programs interacting with the simulated environment. For example, the environment can be a computer simulation of a real -world environment and the agent can be a simulated mechanical agent navigating through the computer simulation.

[0068] For example, the simulated environment may be a motion simulation environment, e.g., a driving simulation or a flight simulation, and the agent may be a simulated vehicle navigating through the motion simulation. In these implementations, the actions may be control inputs to control the simulated user or simulated vehicle. As another example, the simulated environment may be a computer simulation of a real-world environment and the agent may be a simulated robot interacting with the computer simulation.

[0069] Generally, when the environment is a simulated environment, the actions may include simulated versions of one or more of the previously described actions or types of actions.

[0070] In some cases, the system can be used to control the interactions of the agent with a simulated environment, and the system can train the parameters of the neural network used to control the agent based on the interactions of the agent with the simulated environment. After the neural network is trained based on the interactions of the agent with a simulated environment, the policy neural network can be used to control the interactions of a real-world agent with the real -world environment, i.e., to control the agent that was being simulated in the simulated environment. Training the neural network based on interactions of an agent with a simulated environment (i.e., instead of a real-world environment) can avoid wear-and- tear on the agent and can reduce the likelihood that, by performing poorly chosen actions, the agent can damage itself or aspects of its environment.

[0071] FIG. 2 is a flow diagram of an example process 200 for controlling the agent. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, an action selection system, e.g., the action selection system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 200.

[0072] The system receives a goal image that characterizes a goal state to be reached as a result of the agent performing a particular task in the environment (step 202).

[0073] The system tokenizes the goal image to generate a plurality of visual tokens that represent the goal image (step 204).

[0074] As described above, each visual token is selected from a discrete vocabulary of tokens, e.g., a vocabulary that includes a fixed number of visual tokens that represent images and, optionally, a fixed number of additional tokens that each represent a respective discretized numerical value.

[0075] “Tokenizing” an image refers to processing the intensity values of the pixels of the image to generate a representation of the image as a set of multiple tokens from a discrete vocabulary of tokens.

[0076] The tokens in the vocabulary can have any appropriate format such that each token is unique from each other token and can be processed as input by a machine learning model.

[0077] For example, the tokens can each be one-hot encoded vectors that each have a “one” in along a different dimension. As another example, the tokens can each be different integers.

[0078] In some implementations, each visual token in the vocabulary corresponds to a respective latent vector in a codebook of latent vectors. A “codebook” is a set that includes a fixed number of vectors.

[0079] In these implementations, the system can tokenize the goal image by processing the goal image using an image encoder neural network to generate an encoder output that includes a plurality of encoded vectors. The system can then identify, for each encoded vector, a latent vector from the codebook that is nearest to the encoded vector.

[0080] The image encoder neural network can generally have any appropriate architecture. For example, the image encoder neural network can be a convolutional neural network or a vision Transformer neural network.

[0081] The system can then select, as the plurality of visual tokens, the visual tokens from the vocabulary that correspond to the nearest latent vector to each of the plurality of encodedvectors, i.e., so that the visual tokens include, for each encoded vector, the nearest latent vector to the encoded vector.

[0082] This process is generally referred to as vector quantization (VQ).

[0083] The system then performs the following steps to control the agent at each of a plurality of time steps, e.g., at each time step in a sequence of time steps until termination criteria are satisfied.

[0084] The system obtains one or more observation images characterizing a state of the environment at the time step (step 206). For example, the observation images can be images captured by a camera sensor of the robot or by a camera sensor located in the environment.

[0085] The system tokenizes each of the one or more observation images to generate a plurality of visual tokens selected from the discrete vocabulary of tokens that represent the one or more observation images (step 208). In particular, the system can tokenize each observation image in the same manner as described above for the goal images.

[0086] The system generates a sequence of input tokens that includes the plurality of visual tokens that represent the goal image and the plurality of visual tokens that represent the one or more observation images (step 210). For example, the sequence of input tokens can include the visual tokens that represent the goal image followed by the visual tokens that represent the one or more observation images.

[0087] The sequence can also optionally include tokens representing information from previous time steps, e.g., visual tokens representing previous observation images received at one or more previous time steps, previous actions performed by the agent, and / or reward values received as a result of the agent performing previous actions.

[0088] For example, when used, the system can tokenize the actions by discretizing each numeric value that defines a given action and then mapping the discretized numeric value to the corresponding token in the vocabulary.

[0089] Similarly, when used, the system can tokenize the reward values by discretizing each reward value and then mapping the discretized numeric value to the corresponding token in the vocabulary.

[0090] In some implementations, the one or more observation images are a portion of a larger observation for the time step.

[0091] For example, the system can also receive a proprioceptive observation that includes a plurality of numerical values characterizing a state of the agent at the time step, e.g., that includes sensor readings from one or more proprioceptive sensors of the agent. In this example, the system can generate a plurality of discretized tokens that represent theproprioceptive observation by discretizing each of the plurality of numerical values in the proprioceptive observation and then include the plurality of discretized tokens in the input sequence for the time step.

[0092] In some implementations, the input sequence can include additional information.

[0093] For example, in some implementations, in addition to receiving the goal image, the system can also receive a natural language instruction for the agent that characterizes the task to be performed by the agent. For example, the system can receive a single natural language instruction before performing the task or can periodically receive new natural language instructions as the agent performs the task. In this example, the system can tokenize the most recently received natural language instruction and can then include one or more tokens representing the natural language instruction for the agent as part of the input sequence.

[0094] As another example, in some implementations, instead of or in addition to receiving the goal image, the system can also receive a video demonstration of the task being performed. For example, the video demonstration can be of a human performing the task or of a robot controlled by another system performing the task. In this example, the system can tokenize the images in the video demonstration and can then include one or more tokens representing the video demonstration of the task being performed as part of the input sequence.

[0001] The system processes the sequence of input tokens using the sequence modeling neural network to generate an output sequence of output tokens from the discrete vocabulary of tokens that represents an action to be performed by the agent in response to the observation images (step 212).

[0002] For example, the output sequence can represent the action as a sequence of discretized tokens, i.e., as a sequence of tokens that each represent a discretized numerical value.

[0003] As a particular example, as described above, an action for a robot can include one or more action dimensions for each of multiple DoFs of the robot. Thus, the output sequence can include a respective token for each of the action dimensions that each correspond to a respective numerical value. By mapping the tokens to their corresponding numerical values, the system can generate an action that specifies a respective control input for each of the DoFs of the robot.

[0004] As described above, the sequence modeling neural network can have any appropriate architecture that allows the neural network to map an input sequence of tokens from a vocabulary to an output sequence of tokens from the vocabulary.

[0005] As a particular example, the sequence modeling neural network can be an autoregressive neural network, e.g., a causal Transformer neural network e.g., an encoder-decoder Transformer neural network with a causal Transformer decoder or a causal decoder-only Transformer neural network, that auto-regressively generates the tokens in the output sequence. That is, the neural network auto-regressively generates each token in the output sequence conditioned on the tokens in the input sequence and any preceding tokens in the output sequence.

[0006] The system then causes the agent to perform the selected action (step 214). For example, the system can cause the agent to perform the selected action, e.g., by providing instructions to the agent that when executed cause the agent to perform the action, by submitting a control input directly to the appropriate controls of the agent, by providing data identifying the action to a control system for the agent, or using another appropriate control technique.

[0007] As described above, when controlling the agent, the system can continue performing these steps until termination criteria are satisfied. For example, the termination criteria can be satisfied when the task is successfully completed, when a threshold number of iterations have been performed or when the environment enters a terminal state.

[0008] The system can determine that a task has been successfully completed in any of a variety of ways.

[0009] For example, the system can receive a user input indicating that the task has been completed.

[0010] As another example, the system can apply one or more rules to the observation at a given time step, e.g., to the image at the time step, to the proprioceptive information at the time step, or both, to determine whether the task has been successfully completed.

[0011] As yet another example, the system can use a success detection model (success detector), e.g., a vision-based reward model, to process the observation image at the time step to generate a prediction of whether the task has been successfully completed.

[0012] In some cases, e.g., when the system is controlling the agent while generating training data for further training of the sequence modeling neural network or generating data for evaluating the performance of the sequence modeling neural network, the system can determine whether to autonomously switch the task that the system is performing, i.e., to switch from performing the particular task to performing a different task in order to increase the quality and diversity of the generated data.

[0013] This is described in more detail below with reference to FIG. 5.

[0014] Generally, prior to controlling the agent to perform the particular task, the system or another system has trained the sequence modeling neural network.

[0015] One example of performing this training is described below with reference to FIG. 3.

[0016] FIG. 3 is a flow diagram of an example process 300 for training the sequence modeling neural network. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the action selection system 100 of FIG. 1 or a different system of one or more computers, appropriately programmed in accordance with this specification, can perform the process 300.

[0017] The system obtains first training data that includes a plurality of training input sequences (302).

[0018] Each training input sequence has been generated from interactions of a respective agent with a respective environment and corresponds to a respective task.

[0019] Moreover, each training input sequence includes, for each of a respective plurality of training time steps: (i) visual tokens from the vocabulary representing a goal image and generated by tokenizing the goal image; (ii) visual tokens from the vocabulary representing one or more observation images for the training time step and generated by tokenizing each of the observation images; and (iii) tokens from the vocabulary representing an action performed by the respective agent at the training time step.

[0020] Generally, the training sequences can be generated using a variety of different interactions, e.g., interactions by different agents, interactions with different environments, interactions while performing different tasks, interactions while controlled by different policies, and so on.

[0021] That is, the training sequences can represent a variety of diverse interactions with the environment. Some examples of differences between training sequences in the first training data that allow the first training data to capture the diverse range of interactions now follow.

[0022] As one example, two or more of the training sequences in the first training data can have been generated from interactions of different agents.

[0023] That is, the training sequences can represent interactions of multiple different agents with the environment. For example, the different agents can include two or more different robots with different action spaces. For example, the different robots can include robots withmultiple different DoFs, e.g., one robot with 5 DoFs, another robot with 7 DoFs, another robot with 14 DoFs, and so on.

[0024] Moreover, even if two robots have the same action space, e.g., the same number of DoFs, the proprioceptive observations for the robots can have different dimensionalities. Moreover, even if two robots have the same action space, the physical and kinematic characteristics between the embodiments of the two robots can result in the action distributions being different between the two different robots.

[0025] As another example, two or more of the training sequences in the first training data can have been generated from interactions of respective agents with different environments. That is, the training sequences can represent interactions with multiple different environments. For example, the different environments can include both one or more real- world environments and one or more simulated environments. Thus, the training data can include both real-world and simulated training data. In the case of robotics, robots trained in simulation can have different action spaces than robots trained in the real-world or the same action space.

[0026] As another example, two or more of the training sequences in the first training data can correspond to different tasks. That is, the training sequences can represent interactions associated with multiple different tasks. For example, the different tasks can include different robotic manipulation tasks. More specifically, the different robotic manipulation tasks can require manipulating different types of objects.

[0027] Some examples of different tasks when the agent is a robot now follow. In these examples, the tasks include tasks from one or more different task families.

[0028] For example, the task families can include one or more of object stacking tasks, tower building tasks, pyramid building tasks, and inverted pyramid building tasks. The task families stacking, tower building, pyramid building, and inverted pyramid building consist of building structures with either objects or gears. They differ in difficulty, but in all cases require dexterous and precise movements to ensure that the structure remains stable after completion.

[0029] As another example, the task families can include an object lifting task family. The lifting task family consists of picking up a specific object in a basket with multiple objects. The objects can be, e.g., fruits, vegetables, or gears.

[0030] As another example, the task families can include object insertion and removal task families that require inserting an object in a specified location or removing an object that is at a specified location.

[0031] Additionally, each training input sequence has been generated from interactions of the respective agent while the respective agent has been controlled by a respective policy. To improve the diversity of the training data, two or more of the training sequences in the first training data can have been generated while the respective agents were controlled by different policies. Examples of such policies include (i) a policy neural network for the corresponding task trained through reinforcement learning and (ii) human-teleoperated policies.

[0032] The training sequences can also include sequences generated while an agent was controlled using the sequence modeling neural network, i.e., so that the sequence modeling neural network can be re-trained on training data generated using previous versions of the neural network. This is described in more detail below.

[0033] As a particular example, the system can use both expert and non-expert data for training, with different subsets of the data being collected in different ways. For example, the system can use three types of data generation: (i) data produced by specialist RL agents, e.g., employed in simulation; (ii) human-teleoperated expert data, e.g., used for physical world tasks; and iii) self-generated data. The primary difference between the two expert types of trajectories is that agent data provides fairly smooth and efficient trajectories due to the way the RL agent acts in the world, while teleoperated data often includes pauses as teleoperators employ behaviours similar to a bang-bang controller.

[0034] In some cases, the system can generate some of the sequences in the training data by re-labeling an initial training sequence to modify the goal image in the training sequence.

[0035] For example, a given input sequence can correspond to a respective task episode.

[0036] For a first training sequence generated from the given input sequence, the system can select the goal image in the first training sequence by selecting, as the goal image, an image of a last state in the corresponding task episode. Because the task episode is guaranteed to be “successful” with respect to reaching the last state in the episode, this allows the system to include even unsuccessful episodes (i.e., in terms of completing the task), in the training data.

[0037] Additionally, if the given input sequence is a successful sequence, i.e., has been determined to be a successful sequence of performing an instance of a corresponding task, for a second training sequence generated from the given input sequence, the system can select the goal image to be included in the second training sequence by identifying a different task episode corresponding to the same task as the second training sequence, e.g., a different task episode where the same task was successfully performed; and selecting, as the goal image, an image of the last state in the different task episode. Selecting this type of semantically-equivalent goal can assist the neural network in generalizing to different instances of the same task, e.g., different instances of the same task that require manipulating or otherwise interacting with different objects in the environment.

[0038] The system then trains the sequence modeling neural network on the first training data, e.g., by training the sequence modeling neural network using an objective that requires predicting tokens from training sequences conditioned on earlier sequences in the training sequences (step 304).

[0039] The system can perform this training in any of a variety of ways.

[0040] For example, training the sequence modeling neural network on the first training data can include training the sequence modeling neural network to predict, for each training time step in each training sequence, the tokens representing the action performed at the training time step conditioned on at least the tokens representing the goal image and the tokens representing one or more observation images for the training time step (step 306). Optionally, the conditioning can also include tokens from one or more preceding time steps, e.g., as described above. For example, the training objective for the training can include an action token prediction objective that measures, for each training time step in each training sequence and for each action token representing the action token at the training time step, the log probability assigned to the action token by the sequence modeling neural network by processing the tokens that precede the action token in the training input sequence.

[0041] As another example, training the sequence modeling neural network on the first training data can include training the sequence modeling neural network to predict, for each of at least a subset of the training time steps in each of the training sequences, the tokens representing the one or more observation images at a future time step that is after the training time step in the training sequence conditioned only on the tokens at the training time step and, optionally, one or more tokens from one or more preceding time steps (and not on tokens at any time steps that are after the training time step in the training sequence) (step 308).

[0042] In some implementations, the future time step is a time step that is k time steps after the training time step in the training sequence, where k is greater than 1. Setting k greater than 1, e.g., to 3, 5, or 7, allows the system to account for the fact that images one step apart within a trajectory can frequently look very similar, making one step prediction too reliant on memorizing the most recent observation image.

[0043] For example, the training objective for the training can include an action token prediction objective that measures, for one or more particular training time steps of the training time steps in each training sequence and for each visual token representing the one ormore observation images at a training time step that is k time steps after the particular time step in the sequence, the log probability assigned to the visual token by the sequence modeling neural network by processing the tokens at the preceding time step and any tokens that precede the tokens in the input sequence (and not any tokens that are at time steps that are after the particular time step).

[0044] As a particular example, a training sequence f of tokens that has Z+l time steps can be represented as follows:where x is a proprioceptive observation, l is a set of one or more observation images, g is a goal image, a is an action, and / ., AT, N, Q denote the number of tokens required to encode proprioceptive inputs, observation images, goals, and actions, respectively. The goal images g can be fixed within the trajectory and repeated at each time step while the other quantities will generally vary across time steps.

[0045] When the training objective for the training includes objectives for both steps 306 and 308, the overall objective L 6; D) can be expressed as:log (P,kI xli1, lif, gin where 0 are the parameters of the neural network and D is the training data set. As can be seen from this objective, in some cases, the system does not condition the objective on the action tokens at any given time step. In some other cases, the system does condition the objective on the action tokens.

[0046] By repeatedly performing this training, the system can train the neural network to effectively condition on any of a variety of goal images, i.e., to, given a goal image, accurately generate action tokens that result in the agent performing the task specified by the goal image.

[0047] In some implementations, the image encoder neural network that is used to tokenize input images has been pre-trained prior to training the sequence modeling neural network and has been held fixed during the training of the sequence modeling neural network. For example, the image encoder neural network can be pre-trained using a reconstruction loss or contrastive loss.

[0048] For example, the image encoder neural network can have been trained using a VQ- GAN framework. That is, the image encoder neural network can be trained to represent an image as a plurality of latent vectors of a cookbook. Optionally, the latent vectors in thecodebook can be learned jointly during the training of the encoder neural network and then held fixed during the training of the sequence modeling neural network. Training an image encoder neural network using a VQ-GAN framework is, for example, described in Esser et al. 2021 arXiv:2012.09841 and van den Oord et al. 2017 arXiv: 1711.00937.

[0049] In some cases, the system can pre-train the encoder on a diverse collection of images to improve the generalization of the sequence modeling neural network that uses the outputs of the encoder after the pre-training. For example, the system can train the encoder on (i) a dataset that consists of a large number of general purpose images, and (ii) one or more data sets that include images of robots or other agents being controlled to perform tasks. For example, these data sets can include simulated robots, video games or other simulated environments, real-world robots, e.g. robots performing visual manipulation tasks, and so on.

[0050] In some implementations, after training the sequence modeling neural network on the first training data, the training system obtains second training data for a particular task and trains the sequence modeling neural network on the second training data (step 310). Generally, the system can train the sequence modeling neural network on the second training data using the same training objective as when training on the second training objective or on a different objective. For example, the system can train the neural network to predict only the action tokens as described or to predict both the action tokens and the visual tokens as described above.

[0051] As another example, the system can generate this training data by controlling the agent to perform the particular task using the sequence modeling neural network and then train the neural network on the objective described above or using reward values for the particular task through reinforcement learning or a combination of the above objective and reinforcement learning.

[0052] As another example, the system can obtain demonstration data for the particular task, e.g., data that has been generated by a human-teleoperated policy or by an expert, already trained policy for the task and then train the sequence modeling neural network on the objective described above on the demonstration data.

[0053] Thus, the system uses the second training data to fine-tune the sequence modeling neural network to improve the performance of the neural network in controlling the agent to perform the particular task.

[0054] Optionally, the system can then use the fine-tuned neural network to generate additional data that can be used as additional training that can be used to improve theperformance of the neural network on new downstream tasks. An example of how this can be done is now described with reference to FIG. 4.

[0055] FIG. 4 shows an example 400 of a training pipeline that can be performed by the system to train the sequence modeling neural network in order to generate a self-improving agent, i.e., that repeatedly incorporates data from previous tasks and other robot interactions in order to improve the performance of the neural network in controlling a robot for the next task.

[0056] As shown in FIG. 4, the system first obtains 402 a large training data set that represents interactions by a diverse set of robots while performing a diverse set of tasks.

[0057] The system then trains 404 the sequence modeling neural network on this large training data set.

[0058] After this training, the system collects 406 demonstration data for a new task and, optionally, a new robot. For example, the new robot can have a different physical configuration, i.e., can be a different robot embodiment, than any of the robots already represented in the first training data set. The different physical configuration can result in a different action space, i.e., different degrees of freedom, a different observation space, e.g., different available proprioceptive or image information, or both. In the example 400, the system collects 406 demonstration data that is generated by users tele-operating a robot to perform the new task. More generally, however, the system can obtain any appropriate demonstration data of a robot preforming the new task.

[0059] The system fine-tunes 408 the neural network on the demonstration data, e.g., on the same objective used during the training of the neural network on the large training data set.

[0060] The system then uses the fine-tuned neural network to self-generate 410 more training data, e.g., by using the neural network to control the robot (or more than one robot) to perform the new task.

[0061] The system then expands 412 the current training data set by adding the selfgenerated data to the current training data set.

[0062] Thus, the system can continually expand the training data with data from new tasks, thereby improving the ability of the neural network to generate to new tasks, even when those new tasks require controlling robots to perform different types of movements, controlling novel robots with different hardware configurations, or both.

[0063] This specification also describes techniques for controlling a robot to perform multiple tasks and, in particular, for determining when to switch from controlling the robot to perform one task to controlling the robot to perform another task. These techniques can beused while evaluating one or more policies for controlling the robot, while generating training data for the one or more policies, and so on.

[0064] In particular, resetting the environment for a single task requires bringing the state from the end state back into the set of valid start states for that task. However, manually programming such reset routines is a highly non-trivial endeavour (in many cases performing a reset is almost as complicated as solving the task itself) resulting in a significant complication when collecting data for use in training or evaluating a policy.

[0065] The system can account for this by leveraging by the fact that the set of end states for some tasks overlap with the set of start states of other tasks. Thus the system can “re-use” tasks trained for a given task as reset mechanisms for tasks whose end states overlap with the valid start states for another task.

[0066] In particular, the system implements an autonomous reset mechanism based on this observation that uses policy pools.

[0067] Instead, the system can implement autonomous reset using a set of policy pools.

[0068] A policy pool is a collection of policies (or policies implicitly defined by a pool of goal images) with overlapping start and end states. By pooling multiple policies in this way, the system can achieve automated resets, increase the robot utilization (by reducing the need for explicit human resets) and increase the diversity of initial conditions for evaluation and data collection.

[0069] The system can utilize two types of policy pools: stateless policy pools, in which the policies are executed in some order regardless of the state of the environment when a given episode terminates (e.g. for lifting tasks); and a state-based policy pool, which samples the next policy to execute based on the state of the environment (e.g. performing a remove task when the initial state corresponds to a successful insertion).

[0070] An example of the latter type of policy pool is described below with reference to FIG. 5.

[0071] FIG. 5 is a flow diagram of an example process 500 for determining when to switch tasks while controlling an agent. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, an action selection system, e.g., the action selection system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 500.

[0072] In particular, while controlling the robot using a policy for a current task, the system obtains an observation characterizing a current state of the environment at a current time step(step 502). For example, as described above, the observation can include image(s), proprioceptive information, or both. The policy can be, e.g., a goal-conditioned policy that, for each of a plurality of tasks, is conditioned on a respective goal input for the task. An example of such a policy is the sequence modeling neural network described above. Thus, the policy for the current task is specified by conditioning the goal-conditioned policy on a goal input for the current task.

[0073] The system determines, from the observation, whether termination conditions for a first task are satisfied as of the current time step (step 504).

[0074] The first task can be the same as or different from the current task.

[0075] That is, in some cases, even though the system is controlling the agent using the policy for the current task, termination conditions for a different task, e.g., a sub-task of the current task or another different task, may be satisfied.

[0076] For example, the system can determine, for each of multiple tasks, whether respective termination conditions for the task are satisfied at the current time step.

[0077] For example, the termination conditions for a given task can be satisfied when the task is successfully performed as of the current time step.

[0078] In some of these cases, to determine whether termination conditions for a given task are satisfied, the system processes at least a portion of the observation using a success detector to generate a success score for the current task. For example, the success detector can process the observation images in the observation, the proprioceptive data in the observation, or both. The system can then determine, based on the success score, whether the given task has been successfully completed. As a particular example, the success detector can be a machine learning model that has been trained to process the one or more images in the observation to generate a success score that indicates whether the task has been successfully completed.

[0079] When the termination conditions are satisfied, the system identifies a second task that has initial conditions that overlap with the termination conditions for the first task (step 506).

[0080] As one example, one task can have initial conditions that overlap with termination conditions for another task when the states in which the other task is successfully completed are included in a set of valid initial states for the task.

[0081] In particular, the system can maintain, for each of multiple tasks, a set of other tasks that have initial conditions that overlap with the termination conditions for the first task.One example of this is that an insertion task can be determined to overlap with a set of one or more removal tasks.

[0082] The system then controls the robot using a policy for the second task starting from the current state of the environment (step 508). For example, the system can continue controlling the robot using the policy for the second task until termination conditions for the second task or a different one of the multiple tasks are satisfied. For example, when the policy is a goal-conditioned policy, the policy for the second task is specified by conditioning the goal-conditioned policy on a goal input for the second task.

[0083] Similarly, when the termination conditions for the first task are not satisfied, the system does not switch the task that is being performed. For example, at a particular time step that is different from the current time step, the system can obtain an observation characterizing a particular state of the environment. The system can then determine, from the observation, that termination conditions for the first task are not satisfied as of the particular time step. In response, the system can continue controlling the robot using the policy for the current task starting from the particular state of the environment, i.e., can refrain from switching the task.

[0084] The system can thus autonomously reset the task that is being performed by the robot when termination conditions for a given task are satisfied. This results in automated resets without requiring a manually programmed reset routine, increases the robot utilisation (by reducing the need for explicit human resets and ensuring that more of the robot utilization is to perform tasks rather than moving to valid states for tasks) and increases the diversity of initial conditions for evaluation and data collection. That is, by automatically switching tasks as described above, the system can significantly improve the quality of the data that is generated, either for evaluating control policies for certain tasks or for training a neural network to control agents, e.g., the sequence modeling neural network as described above or a different neural network.

[0085] FIG. 6 shows an example 600 of the performance of various example implementations of the described techniques on a variety of tasks. In particular, the example 600 compares the performance of two example versions of the described techniques, one example that uses a smaller sized sequence modeling neural network (“RobotCat (small)”) and another example that uses a relative larger sequence modeling neural network (“RoboCaf ’), with single-task baseline agents trained on the same data for each task. In particular, the example 600 shows performance for simulation tasks 610 and real -world tasks 620. As can be seen from the example 600, RoboCat performs better on the vast majority oftraining tasks, compared to the single-task baseline agents trained on the same data for the task. In the real world, where the system has limited data compared to simulation, RoboCat can take advantage of multi-task joint training on robotics data to perform significantly better than the baselines. The much smaller version (RoboCat (small)) is also evaluated on the simulation tasks, and demonstrates similar performance for stacking but much lower success for the more challenging cases.

[0086] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0087] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0088] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocolstack, a database management system, an operating system, or a combination of one or more of them.

[0089] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0090] In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0091] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0092] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0093] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0094] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0095] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and computeintensive parts of machine learning training or production, i.e., inference, workloads.

[0096] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework, a Microsoft Cognitive Toolkit framework, an Apache Singa framework, or an Apache MXNet framework.

[0097] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described inthis specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0098] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0099] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what can be claimed, but rather as descriptions of features that can be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features can be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination can be directed to a subcombination or variation of a subcombination.

[0100] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0101] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing can be advantageous.

[0102] Aspects of the present disclosure may be as set out in the following clauses:Clause 1. A method performed by one or more computers and for controlling an agent interacting with an environment, the method comprising: receiving a goal image that characterizes a goal state to be reached as a result of the agent performing a particular task in the environment; tokenizing the goal image to generate a plurality of visual tokens that represent the goal image, wherein each visual token is selected from a discrete vocabulary of tokens; at each of a plurality of time steps: obtaining one or more observation images characterizing a state of the environment at the time step; tokenizing each of the one or more observation images to generate a plurality of visual tokens selected from the discrete vocabulary of tokens that represent the one or more observation images; generating a sequence of input tokens that comprises the plurality of visual tokens that represent the goal image and the plurality of visual tokens that represent the one or more observation images; processing the sequence of input tokens using a sequence modeling neural network to generate an output sequence of output tokens from the discrete vocabulary of tokens that represents an action to be performed by the agent in response to the observation images; and causing the agent to perform the selected action.Clause 2. The method of clause 1, wherein the environment is a real -world environment and the agent is a robot.Clause 3. The method of clause 1 or 2, wherein: the discrete vocabulary of tokens comprises: a plurality of visual tokens; and a plurality of discretized tokens that each represent a discretized numerical value.Clause 4. The method of clause 3, wherein the output sequence represents the action as a sequence of discretized tokens.Clause 5. The method of clause 3 or 4, further comprising, at each time step: receiving a proprioceptive observation comprising a plurality of numerical values characterizing a state of the agent at the time step; generating a plurality of discretized tokens that represent the proprioceptive observation by discretizing each of the plurality of numerical values in the proprioceptive observation, wherein the input sequence further comprises the plurality of discretized tokens.Clause 6. The method of any preceding clause, wherein, for each time step after a first time step, the input sequence further comprises visual tokens representing the one or more observation images received at one or more preceding time steps.Clause 7. The method of any preceding clause, wherein the sequence modeling neural network is an auto-regressive neural network.Clause 8. The method of clause 7, wherein the sequence modeling neural network is a causal Transformer neural network.Clause 9. The method of any preceding clause, wherein each visual token in the vocabulary corresponds to a respective latent vector in a codebook of latent vectors, and wherein tokenizing the goal image to generate a plurality of visual tokens that represent the goal image comprises: processing the goal image using an image encoder neural network to generate an encoder output comprising a plurality of encoded vectors; for each encoded vector, identifying a latent vector from the codebook that is nearest to the encoded vector; and selecting, as the plurality of visual tokens, the visual tokens from the vocabulary that correspond to the nearest latent vector to each of the plurality of encoded vectors.Clause 10. The method of clause 10 or 11, wherein the sequence modeling neural network has been trained by performing operations comprising: obtaining first training data comprising a plurality of training input sequences, wherein: each training input sequence has been generated from interactions of a respective agent with a respective environment and corresponds to a respective task; each input sequence comprises, for each of a respective plurality of training time steps:(i) tokens representing a goal image and generated by tokenizing the goal image;(ii) tokens representing one or more observation images for the training time step and generated by tokenizing each of the observation image; and(iii) tokens representing an action performed at the training time step; and training the sequence modeling neural network on the first training data.Clause 11. The method of clause 10 when dependent on clause 9, wherein the image encoder neural network has been pre-trained prior to training the sequence modeling neural network and has been held fixed during the training of the sequence modeling neural network.Clause 12. The method of clause 11, wherein the image encoder neural network has been trained using a VQ-GAN framework.Clause 13. The method of any one of clauses 10-13, wherein training the sequence modeling neural network on the first training data comprises training the sequence modeling neural network to predict, for each training time step in each training sequence, the tokens representing the action performed at the training time step conditioned on at least the tokens representing the goal image and the tokens representing one or more observation images for the training time step.Clause 14. The method of clause 13, wherein training the sequence modeling neural network on the first training data comprises training the sequence modeling neural network to predict, for each of at least a subset of the training time steps in each of the training sequences, the tokens representing the one or more observation images at a future time step that is after the training time step in the training sequence.Clause 15. The method of clause 14, wherein the future time step is a time step that is k time steps after the training time step in the training sequence, wherein k is greater than 1.Clause 16. The method of any one of clauses 10-15, wherein each input sequence corresponds to a respective task episode, and wherein obtaining the first training data comprises: selecting, for a first training sequence, the goal image by selecting, as the goal image, an image of a last state in the corresponding task episode; and selecting, for a second training sequence, the goal image by: identifying a different task episode corresponding to a same task as the second training sequence; and selecting, as the goal image, an image of a last state in the different task episode.Clause 17. The method of any one of clauses 10-16, wherein: two or more of the training sequences in the first training data were generated from interactions of different agents.Clause 18. The method of clause 17, wherein the different agents include two or more different robots with different action spaces.Clause 19. The method of any one of clauses 10-18, wherein two or more of the training sequences in the first training data were generated from interactions of respective agents with different environments.Clause 20. The method of clause 19, wherein the different environments include a real- world environment and a simulated environment.Clause 21. The method of any one of clauses 10-20, wherein two or more of the training sequences in the first training data correspond to different tasks.Clause 22. The method of clause 21, wherein the different tasks include different robotic manipulation tasks.Clause 23. The method of clause 22, wherein the different robotic manipulation tasks require manipulating different types of objects.Clause 24. The method of any one of clauses 10-23, wherein each training input sequence has been generated from interactions of the respective agent while the respective agent has been controlled by a respective policy, and wherein two or more of the training sequences in the first training data were generated while the respective agents were controlled by different policies.Clause 25. The method of clause 24, wherein the different policies comprise one or more of: a policy neural network for the corresponding task trained through reinforcement learning; or a human-teleoperated policy.Clause 26. The method of clause 24 or clause 25, wherein the different policies comprise a policy defined by the sequence modeling neural network conditioned on a goal observation image for the corresponding task.Clause 27. The method of any one of clauses 10-26, the operations for the training of the sequence modeling neural network further comprising: after training the sequence modeling neural network on the first training data, obtaining second training data for the particular task; and training the sequence modeling neural network on the second training data.Clause 28. The method of clause 27, wherein obtaining second training data for the particular task comprises: controlling the agent to perform the particular task using the sequence modeling neural network.Clause 29. The method of any one of clauses 27 or 28, wherein the second training data comprises reward values that measure performance of the agent on the particular task, and wherein training the sequence modeling neural network on the second training data comprises training the sequence modeling neural network on the second training data through reinforcement learning using the reward values.Clause 30. The method of any preceding clause, further comprising: generating, using the input sequence for each time step and the action performed by the agent at each time step, additional training data; and training the sequence modeling neural network on the additional training data.Clause 31. The method of any preceding clause, wherein the input sequence further comprises one or more of:(i) one or more tokens representing a natural language instruction for the agent, or(ii) one or more tokens representing a video demonstration of the task being performed.Clause 32. The method of any preceding clause, wherein the agent is a robot and the one or more computers are on-board the robot.Clause 33. The method of any preceding clause, wherein the input sequence further comprises one or more tokens from the vocabulary that represent respective rewards for the particular task received at one or more preceding time stepsClause 34. A method of controlling a robot, the method comprising, at each of a plurality of time steps: obtaining, by a control system of the robot, one or more observation images of the environment at the time step; providing, by a control system of the robot, the one or more observation images to a policy system; obtaining, by the control system of the robot and from the policy system of the robot, data specifying a selected action, wherein the policy system selects the selected action in response to the one or more observation images by performing the operations of the respective method of any preceding clause; and causing, by the control system of the robot, the robot to perform the selected action.Clause 35. The method of clause 34, wherein the control system of the robot is on-board the robot.Clause 36. The method of clause 35, wherein the policy system is on-board the robot.Clause 37. The method of clause 35, wherein: the policy system is remote from the robot, providing the observation image comprises transmitting the observation image over a data communication network; and obtaining the data specifying the selected action comprises receiving the data specifying the selected action over the data communication network.Clause 38. A method for controlling a robot interacting with a real -world environment, the method comprising: while controlling the robot using a policy for a current task: obtaining an observation characterizing a current state of the environment at a current time step; determining, from the observation, that termination conditions for a first task are satisfied as of the current time step; identifying a second task that has initial conditions that overlap with the termination conditions for the first task; and controlling the robot using a policy for the second task starting from the current state of the environment.Clause 39. The method of clause 38, wherein the termination conditions for the first task specify that the first task has been successfully completed as of the current time step.Clause 40. The method of clause 39, wherein: determining, from the observation, that termination conditions for a first task are satisfied as of the current time step comprises: processing at least a portion of the observation using a success detector to generate a success score for the current task; and determining, based on the success score, that the current task has been successfully completed.Clause 41. The method of any one of clauses 38-40, wherein the first task is the current task.Clause 42. The method of any one of clauses 38-41, wherein the second task is different from the first task.Clause 43. The method of any one of clauses 38-42, wherein identifying a second task that has initial conditions that overlap with the termination conditions for the first task comprises: identifying a plurality of candidate tasks that each have respective initial conditions that overlap with the termination conditions for the first task; and selecting, as the second task, one of the candidate tasks.Clause 44. The method of clause 43, wherein selecting, as the second task, one of the candidate tasks comprises: selecting one of the candidate tasks at random.Clause 45. The method of any one of clauses 38-44, wherein the policy for the current task and the policy for the second task are defined by respective machine learning models trained to perform the corresponding task.Clause 46. The method of any one of clauses 38-44, wherein the policy for the current task and the policy for the second task are defined by a goal-conditioned policy that, for each of a plurality of tasks, is conditioned on a respective goal input for the task.Clause 47. The method of clause 46, wherein the goal input comprises a goal observation image for the task.Clause 48. The method of clause 47, wherein the goal-conditioned policy is the sequence modeling neural network of any one of clauses 1-33.Clause 49. The method of any one of clauses 38-48, wherein initial conditions for a first given task overlap with termination conditions for a second given task when a set of initial states for the first given task includes a set of termination states for the second given task, and wherein the set of termination states for the second given task includes a set of states in which the second given task is successfully completed.Clause 50. The method of any one of clauses 38-48, further comprising: while controlling the robot using the policy for the current task: obtaining an observation characterizing a particular state of the environment at a particular time step that is different from the current time step; determining, from the observation, that termination conditions for the first task are not satisfied as of the current time step; and in response, controlling the robot using the policy for the current task starting from the particular state of the environment.Clause 51. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform the operations of the respective method of any one of clauses 1-50.Clause 52. One or more computer storage media storing instructions that when executed by one or more computers cause the one more computers to perform the operations of the respective method of any one of clauses 1-50.

Claims

CLAIMS1. A method performed by one or more computers and for controlling an agent interacting with an environment, the method comprising: receiving a goal image that characterizes a goal state to be reached as a result of the agent performing a particular task in the environment; tokenizing the goal image to generate a plurality of visual tokens that represent the goal image, wherein each visual token is selected from a discrete vocabulary of tokens; at each of a plurality of time steps: obtaining one or more observation images characterizing a state of the environment at the time step; tokenizing each of the one or more observation images to generate a plurality of visual tokens selected from the discrete vocabulary of tokens that represent the one or more observation images; generating a sequence of input tokens that comprises the plurality of visual tokens that represent the goal image and the plurality of visual tokens that represent the one or more observation images; processing the sequence of input tokens using a sequence modeling neural network to generate an output sequence of output tokens from the discrete vocabulary of tokens that represents an action to be performed by the agent in response to the observation images; and causing the agent to perform the selected action.

2. The method of claim 1, wherein the environment is a real -world environment and the agent is a robot.

3. The method of claim 1 or 2, wherein: the discrete vocabulary of tokens comprises: a plurality of visual tokens; and a plurality of discretized tokens that each represent a respective discretized numerical value.

4. The method of claim 3, wherein the output sequence represents the action as a sequence of discretized tokens.

5. The method of claim 3 or 4, further comprising, at each time step: receiving a proprioceptive observation comprising a plurality of numerical values characterizing a state of the agent at the time step; generating a plurality of discretized tokens that represent the proprioceptive observation by discretizing each of the plurality of numerical values in the proprioceptive observation, wherein the input sequence further comprises the plurality of discretized tokens.

6. The method of any preceding claim, wherein, for each time step after a first time step, the input sequence further comprises visual tokens representing the one or more observation images received at one or more preceding time steps.

7. The method of any preceding claim, wherein the sequence modeling neural network is an auto-regressive neural network.

8. The method of claim 7, wherein the sequence modeling neural network is a causal Transformer neural network.

9. The method of any preceding claim, wherein each visual token in the vocabulary corresponds to a respective latent vector in a codebook of latent vectors, and wherein tokenizing the goal image to generate a plurality of visual tokens that represent the goal image comprises: processing the goal image using an image encoder neural network to generate an encoder output comprising a plurality of encoded vectors; for each encoded vector, identifying a latent vector from the codebook that is nearest to the encoded vector; and selecting, as the plurality of visual tokens, the visual tokens from the vocabulary that correspond to the nearest latent vector to each of the plurality of encoded vectors.

10. The method of any preceding claim, wherein the sequence modeling neural network has been trained by performing operations comprising: obtaining first training data comprising a plurality of training input sequences, wherein: each training input sequence has been generated from interactions of a respective agent with a respective environment and corresponds to a respective task; each input sequence comprises, for each of a respective plurality of training time steps:(i) tokens representing a goal image and generated by tokenizing the goal image;(ii) tokens representing one or more observation images for the training time step and generated by tokenizing each of the observation image; and(iii) tokens representing an action performed at the training time step; and training the sequence modeling neural network on the first training data.

11. The method of claim 10 when dependent on claim 9, wherein the image encoder neural network has been pre-trained prior to training the sequence modeling neural network and has been held fixed during the training of the sequence modeling neural network.

12. The method of claim 11, wherein the image encoder neural network has been trained using a VQ-GAN framework.

13. The method of any one of claims 10-12, wherein training the sequence modeling neural network on the first training data comprises training the sequence modeling neural network to predict, for each training time step in each training sequence, the tokens representing the action performed at the training time step conditioned on at least the tokens representing the goal image and the tokens representing one or more observation images for the training time step.

14. The method of claim 13, wherein training the sequence modeling neural network on the first training data comprises training the sequence modeling neural network to predict, for each of at least a subset of the training time steps in each of the training sequences, the tokens representing the one or more observation images at a future time step that is after the training time step in the training sequence.

15. The method of claim 14, wherein the future time step is a time step that is k time steps after the training time step in the training sequence, wherein k is greater than 1.

16. The method of any one of claims 10-15, wherein each input sequence corresponds to a respective task episode, and wherein obtaining the first training data comprises: selecting, for a first training sequence, the goal image by selecting, as the goal image, an image of a last state in the corresponding task episode; and selecting, for a second training sequence, the goal image by: identifying a different task episode corresponding to a same task as the second training sequence; and selecting, as the goal image, an image of a last state in the different task episode.

17. The method of any one of claims 10-16, wherein: two or more of the training sequences in the first training data were generated from interactions of different agents with a respective environment.

18. The method of claim 17, wherein the different agents include two or more different robots with different action spaces.

19. The method of any one of claims 10-18, wherein two or more of the training sequences in the first training data were generated from interactions of respective agents with different environments.

20. The method of claim 19, wherein the different environments include a real -world environment and a simulated environment.

21. The method of any one of claims 10-20, wherein two or more of the training sequences in the first training data correspond to different tasks.

22. The method of claim 21, wherein the different tasks include different robotic manipulation tasks.

23. The method of claim 22, wherein the different robotic manipulation tasks require manipulating different types of objects.

24. The method of any one of claims 10-23, wherein each training input sequence has been generated from interactions of the respective agent while the respective agent has been controlled by a respective policy, and wherein two or more of the training sequences in the first training data were generated while the respective agents were controlled by different policies.

25. The method of claim 24, wherein the different policies comprise one or more of: a policy neural network for the corresponding task trained through reinforcement learning; or a human-teleoperated policy.

26. The method of claim 24 or claim 25, wherein the different policies comprise a policy defined by the sequence modeling neural network conditioned on a goal observation image for the corresponding task.

27. The method of any one of claims 10-26, the operations for the training of the sequence modeling neural network further comprising: after training the sequence modeling neural network on the first training data, obtaining second training data for the particular task; and training the sequence modeling neural network on the second training data.

28. The method of claim 27, wherein obtaining second training data for the particular task comprises: controlling the agent to perform the particular task using the sequence modeling neural network.

29. The method of any one of claims 27 or 28, wherein the second training data comprises reward values that measure performance of the agent on the particular task, and wherein training the sequence modeling neural network on the second training data comprises training the sequence modeling neural network on the second training data through reinforcement learning using the reward values.

30. The method of any preceding claim, further comprising: generating, using the input sequence for each time step and the action performed by the agent at each time step, additional training data; and training the sequence modeling neural network on the additional training data.

31. The method of any preceding claim, wherein the input sequence further comprises one or more of:(i) one or more tokens representing a natural language instruction for the agent, or(ii) one or more tokens representing a video demonstration of the task being performed.

32. The method of any preceding claim, wherein the agent is a robot and the one or more computers are on-board the robot.

33. The method of any preceding claim, wherein the input sequence further comprises one or more tokens from the vocabulary that represent respective rewards for the particular task received at one or more preceding time steps.

34. A method of controlling a robot, the method comprising, at each of a plurality of time steps: obtaining, by a control system of the robot, one or more observation images of the environment at the time step; providing, by a control system of the robot, the one or more observation images to a policy system; obtaining, by the control system of the robot and from the policy system of the robot, data specifying a selected action, wherein the policy system selects the selected action in response to the one or more observation images by performing the operations of the respective method of any preceding claim; and causing, by the control system of the robot, the robot to perform the selected action.

35. The method of claim 34, wherein the control system of the robot is on-board the robot.

36. The method of claim 35, wherein the policy system is on-board the robot.

37. The method of claim 35, wherein: the policy system is remote from the robot, providing the observation image comprises transmitting the observation image over a data communication network; and obtaining the data specifying the selected action comprises receiving the data specifying the selected action over the data communication network.

38. A method for controlling a robot interacting with a real-world environment, the method comprising: while controlling the robot using a policy for a current task: obtaining an observation characterizing a current state of the environment at a current time step; determining, from the observation, that termination conditions for a first task are satisfied as of the current time step; identifying a second task that has initial conditions that overlap with the termination conditions for the first task; and controlling the robot using a policy for the second task starting from the current state of the environment.

39. The method of claim 38, wherein the termination conditions for the first task specify that the first task has been successfully completed as of the current time step.

40. The method of claim 39, wherein: determining, from the observation, that termination conditions for a first task are satisfied as of the current time step comprises: processing at least a portion of the observation using a success detector to generate a success score for the current task; and determining, based on the success score, that the current task has been successfully completed.

41. The method of any one of claims 38-40, wherein the first task is the current task.

42. The method of any one of claims 38-41, wherein the second task is different from the first task.

43. The method of any one of claims 38-42, wherein identifying a second task that has initial conditions that overlap with the termination conditions for the first task comprises: identifying a plurality of candidate tasks that each have respective initial conditions that overlap with the termination conditions for the first task; and selecting, as the second task, one of the candidate tasks.

44. The method of claim 43, wherein selecting, as the second task, one of the candidate tasks comprises: selecting one of the candidate tasks at random.

45. The method of any one of claims 38-44, wherein the policy for the current task and the policy for the second task are defined by respective machine learning models trained to perform the corresponding task.

46. The method of any one of claims 38-44, wherein the policy for the current task and the policy for the second task are defined by a goal-conditioned policy that, for each of a plurality of tasks, is conditioned on a respective goal input for the task.

47. The method of claim 46, wherein the goal input comprises a goal observation image for the task.

48. The method of claim 47, wherein the goal-conditioned policy is the sequence modeling neural network of any one of claims 1-33.

49. The method of any one of claims 38-48, wherein initial conditions for a first given task overlap with termination conditions for a second given task when a set of initial states for the first given task includes a set of termination states for the second given task, and wherein the set of termination states for the second given task includes a set of states in which the second given task is successfully completed.

50. The method of any one of claims 38-48, further comprising: while controlling the robot using the policy for the current task: obtaining an observation characterizing a particular state of the environment at a particular time step that is different from the current time step; determining, from the observation, that termination conditions for the first task are not satisfied as of the particular time step; and in response, controlling the robot using the policy for the current task starting from the particular state of the environment.

51. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform the operations of the respective method of any one of claims 1-50.

52. One or more computer storage media storing instructions that when executed by one or more computers cause the one more computers to perform the operations of the respective method of any one of claims 1-50.

Images