Robot control using trajectories

Abstract

Systems, methods, and computer program code for controlling a robot that is interacting with an environment to perform a particular task The technique involves generating a 2D trajectory image representing a 2D trajectory sketch. The 2D trajectory sketch indicates a desired trajectory for a part of the robot, e.g. an end effector, when performing the task. A neural network system uses the 2D trajectory image as deliberately underspecified guidance for how to perform the task. Techniques for training the neural network system are also described.

Description

[0001] DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application ROBOT CONTROL USING TRAJECTORIES CROSS-REFERENCE TO RELATED APPLICATION [1] This application claims priority to U.S. Provisional Application No.63 / 586,404, filed on September 28, 2023, and to U.S. Provisional Application No.63 / 541,699, filed on September 29, 2023. The disclosure of the prior applications is considered part of and is incorporated by reference in the disclosure of this application. BACKGROUND [2] This specification relates to controlling agents using neural networks. [3] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current value inputs of a respective set of parameters, e.g. weights. SUMMARY [4] This specification describes systems for controlling a robot that is interacting with an environment, e.g. a real-world environment, by selecting actions to be performed by the robot and then causing the robot to perform the actions. More particularly the specification describes techniques for instructing a (trained) neural network system to control a robot to perform a particular task. Techniques for training such a neural network system are also described. [5] In one aspect there is described a method and a corresponding system, implemented as computer programs on one or more computers in one or more locations, for using a (trained) neural network system to control a robot interacting with an environment to perform a task in the environment, in particular a user-directed task. [6] The method involves generating a trajectory image representing a trajectory sketch. [7] The trajectory sketch indicates a desired trajectory for a part of the robot when performing the task. The part of the robot may be, e.g., an end effector such as a gripper, or another part of an arm of the robot, or part of a body of the robot. Achieving the desired trajectory typically involves moving an arm (and end effector) of the robot, and the robot 37760703-1 DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application can remain stationary. However in some implementations achieving the desired trajectory can also or instead involve movement of the whole robot, i.e. of a body of the robot. [8] The trajectory image is a two-dimensional image that represents the desired trajectory as a line in the trajectory image. In general the trajectory sketch is also two- dimensional, i.e. it is defined as a line in a (2D) plane. In general the trajectory image contains the trajectory sketch. The neural network system uses the 2D trajectory sketch as guidance for how to perform the task; this guidance is deliberately underspecified, and is in general only approximate. [9] The trajectory sketch can be obtained from a user; the user can be a human or machine directing the task. The two-dimensional trajectory image can be generated from the trajectory sketch.

[0010] As one example, the trajectory sketch can be obtained from a user interface that allows the user to draw the trajectory sketch, or edit a draft trajectory sketch, e.g. by using a GUI (graphical user interface) or by moving their hand.

[0011] As another example, the trajectory sketch can be obtained from a user interface that allows the user to describe the desired trajectory, and / or actions to be performed by the robot from which a trajectory sketch with the desired trajectory can be obtained. For example, a set of waypoints can be generated from which a trajectory image containing the trajectory sketch can be obtained; or natural or computer language, text or audio input can be processed by a VLM (vision language model) to directly generate a trajectory image containing the trajectory sketch. Thus in some implementations, obtaining the trajectory sketch from the user and generating the two-dimensional trajectory image from the trajectory sketch can be performed in a single step.

[0012] The neural network system can have been trained, for example, using an imitation learning method such as behavior cloning, e.g. on training data comprising human or machine demonstrations of a plurality of different tasks (“expert demonstrations”); and / or in some other way, e.g. by means of reinforcement learning. The system can have been trained to perform a plurality of different tasks, in particular by training the system to perform example tasks whilst conditioned on, i.e. receiving as input, corresponding example trajectory sketches. In some implementations the neural network system includes a (trained) Transformer neural network.

[0013] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application

[0014] One problem with using a neural network system to control a robot to perform a particular task is that the system can have difficulty in generalizing to tasks that were not seen in the training data, particularly where the system was trained using imitation learning from a corpus of expert demonstrations of various tasks. However it is impractical to compile a training dataset that includes objects, scenes, and motions for every conceivable task.

[0015] Some previous approaches have trained a neural network system to control a robot using natural language to specify the task to be performed. However such approaches suffer from a lack of specificity and can find it difficult to generalize to new motions. For example there are many possible end states for an instruction such as “pick can”; and the system may not be able to perform a “fold towel” task even if the motion required is similar to a pick-and-place task. Some other approaches have used a goal image to define a task to be performed, but these can include distracting details and are also hard to learn from. Such approaches can be thought of as over-specified on the end state. Some other approaches have used a video to define a task to be performed, but these can be very hard to learn from as they specify an entire trajectory of states, and can also not generalize well. Generalization to novel motions is considered an open challenge in robotics.

[0016] Implementations of the described techniques use a 2D trajectory sketch to specify the desired motion of a part of a robot in 3D. This provides guidance in the context of visual observations of the environment of the robot, via the trajectory image, but is deliberately underspecified, and can allow the system to perform new tasks that it would otherwise be unable to perform, e.g. the previously described “fold towel” task.

[0017] Surprisingly, and by contrast with other approaches, the described techniques also enable the system to generalize to novel motions. For example implementations of the described techniques have been found to generalize well to diverse new types of task, new motions and new types of object, even when the neural network system has been trained only using imitation learning.

[0018] A further advantage of the described approach is that a trajectory sketch (or trajectory image) is easy for a human to generate and interpret. The sketch can be generated by drawing, or from video, or from waypoints, or using a VLM. That is, even though language-conditioned or video-conditioned policies can struggle to generalize, they can generalize when trajectory sketch as described is generated. The sketch need only approximately the desired trajectory, and it appears that using just a rough trajectory sketch can be positively beneficial. DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application

[0019] In general implementations of the described techniques provide controllability and can perform better than some previous approaches; in particular they can generalize to tasks and motions that are significantly beyond the training data. In some implementations the trajectory sketch can serve as a “prompt” for a vision-action model, in an analogous way to that in which a prompt can be used to guide the output of a large language model.

[0020] In general a task for the (trained) system to perform can be specified solely by a trajectory sketch. That is, the neural network system does not need a language input as well as a trajectory sketch although, optionally, the neural network system can also be conditioned on language (such as text or audio) that specifies a task when performing the task.

[0021] An additional advantage of the described techniques is that they make it easy to train the system. For example it is straightforward to obtain a 2D trajectory sketch from proprioceptive data of a robot (without any need for manual annotation), which makes it easy to re-use existing demonstration datasets for training, and facilitates use of new training datasets as they become available.

[0022] The nature of described techniques, i.e. based upon a trajectory sketch and 2D trajectory image, also facilitates better utilization of the training data because the training data reflects similarities between different motions of the robot, and in effect widens the scope of the tasks seen during training. Further, because motions generated by the described techniques are highly diverse they can even be used to generate new training data for further training of the system.

[0023] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] FIG.1 shows an example of an example of a neural network system for controlling a robot to perform a task.

[0025] FIG.2 shows details of an illustrative example of the system of FIG.1.

[0026] FIG.3 illustrates an example trajectory image.

[0027] FIG.4 illustrates obtaining a trajectory sketch using a graphical user interface. DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application

[0028] FIGS.5A and 5B shows, respectively, illustrates an example graphical user interface for the neural network system, and a trajectory image generated by the graphical user interface.

[0029] FIG.6 illustrates obtaining a trajectory sketch from a video of a user demonstrating a task.

[0030] FIG.7 illustrates obtaining a trajectory sketch using a foundation model.

[0031] FIGS.8A and 8B illustrate, respectively, detecting locations of objects in an image of an environment, and a trajectory sketch defined by waypoints determined from computer code generated by a language model based on the detected locations.

[0032] FIG.9 illustrates obtaining a trajectory sketch using a visual language model.

[0033] FIG.10 is a flow diagram of an example process for controlling a robot to perform a task.

[0034] FIG.11 shows an example system for training the neural network system of FIG.1.

[0035] FIG.12 is a flow diagram of an example process for training a neural network system to control a robot.

[0036] FIG.13 illustrates generation of a trajectory sketch.

[0037] FIGS.14A to 14E illustrate an example robot trajectory (top) and aspects of a corresponding trajectory sketch (bottom).

[0038] FIG.15 shows a video frame, with a trajectory sketch overlaid, of a robot performing a restock drawer task.

[0039] FIG.16 shows a video frame of a robot performing a modified version of a previously seen task.

[0040] FIG.17 shows a video frame of a robot performing a previously unseen task.

[0041] FIG.18 compares the performance of example implementations of the neural network system of FIG.1 with other systems.

[0042] FIG.19 shows a sequence of video frames, one with a trajectory sketch overlaid, showing a robot retrying a task with a different motion after an initial failure.

[0043] FIG.20 illustrates modifying an initial trajectory sketch to enable to successful performance of a task.

[0044] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application

[0045] FIG.1 shows an example of a neural network system 100 that may be implemented as one or more computer programs on one or more computers in one or more locations.

[0046] The neural network system 100 includes a neural network 110 and is used to control an agent, in particular a mechanical embodied agent i.e. a robot 102, interacting with an environment, in particular a real-world environment 104 to perform a task. The robot can be any type of mechanical embodied agent including, e.g., a robotic arm, a quadruped robot, a humanoid robot, a wheeled robot, and so forth. The task can be any type of task, e.g. a task that involves one or more of: locating or manipulating an object in the environment, or moving an object to a specified location in the environment, or navigating to a specified destination in the environment.

[0047] The neural network system 100 controls the agent 102 based on observations of the environment, i.e. data characterizing a state of the environment 104. The observations include at least an observation image 106, i.e. an image of the environment, e.g. of physical objects in the environment (which may include the robot, or part of the robot).

[0048] The observation at a time step can include data from a previous time step that may be beneficial in characterizing the environment, e.g. observation images from one or more previous time steps and / or features characterizing one or more previous actions taken by the robot. Merely as an example the observation at a time step may comprise six observation images, one from the current time step and five from previous time steps.

[0049] In example implementations the observation image 106 is obtained from an image sensor (not shown in FIG.1) sensing the real-world environment 104. As an example, the observation images can be images captured by an image sensor of the robot, e.g. an image sensor mounted on the robot 102 or an image sensor located in the environment 104. As used herein an image can be a still or moving image, and can be a depth image or a point cloud. The image sensor can be, e.g. a camera sensor such as an RGB camera, or an RGBD camera or other depth camera, or a LIDAR sensor.

[0050] In some implementations there may be multiple image sensors. In some implementations the image sensor can be mobile, e.g. where the image sensor is mounted on the robot, and the robot is mobile, a position of the image sensor can change. In some implementations the robot remains stationary, i.e. a body of the robot remains stationary; then the image sensor(s) can be stationary or mobile (e.g. mounted on an arm of the robot).

[0051] The (trained) neural network system 100, in particular the (trained) neural network 110, controls the robot 102 to accomplish a task by selecting actions 114 to be performed by the robot 102 at each of multiple time steps during the performance of an episode of the DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application task. In general an “episode” of a task is a sequence of interactions during which the robot attempts to perform an instance of the task. An episode can begin at some starting state of the environment. An episode can end with a terminal state that indicates whether (or not) the task was performed, e.g. the environment entering a state that has been designated as a terminal state, or after a specified number of action-selection time steps, or in general when some termination criterion is satisfied.

[0052] The neural network 110 is configured to process a neural network input 124 at a time step to generate a policy output 112 for selecting the action 114 to be performed by the robot 102 at the time step. In general an action may be discrete or continuous. An action may include multiple action dimensions; for example, each action dimension can correspond to a respective degree of freedom (DoF) of the robot 102.

[0053] The neural network 110 generates the policy output 112 by processing the neural network input 124 according to learned parameters, e.g. weights, of the neural network 110. In general the policy output may directly identify one or more actions to be performed, e.g. it may output a value that defines the action such as a speed or torque or position, or it may parameterize, i.e. output parameter values of, a distribution from which one or more actions to be performed are sampled or selected.

[0054] As an example the policy output 112 may comprise, for each of a plurality of action dimensions, a respective categorical distribution over possible values for the action dimension. Selecting an action to be performed by the agent using the policy output may comprise selecting a respective value for one or more of the action dimensions using the respective categorical distribution.

[0055] An example of such a neural network system is described in Brohan et al. “RT-1: Robotics Transformer for Real-World Control at Scale”, arXiv:2212.06817v1, 2023, in particular in sections 3 and 5.1, to which reference can be made. Another example is described in Driess et al. “PaLM-E: An Embodied Multimodal Language Model”, arXiv:2303.03378, 2023.

[0056] For example, when configured to control a robot with a base and an arm the output may have a first set of dimensions for controlling movement of the arm, a second set of dimensions for controlling movement of the base, and a discrete variable for switching between three modes: controlling the arm, controlling the base, or terminating an episode. Each dimension may define a value or it may be mapped to one of n bins that are uniformly distributed within the bounds of each variable that is controlled. DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application

[0057] In some implementations the policy output 112 is configured to output tokens, in particular a sequence of one or more output tokens, that define one or more actions, e.g. action dimensions (“tokenized actions”). In some implementations continuous actions, i.e. actions in a continuous action space, may be discretized. As previously described, multiple different actions (“action dimensions”), e.g. controlling different parts of the robot 102, may be performed at a single time step. That is, the sequence of one or more output tokens may define different action dimensions of a single, composite action to be performed at a time step.

[0058] In some implementations the policy output 112 comprises a sequence of output tokens, which may be text tokens, e.g. where the trained neural network system comprises a vision-(language)-action model (such as Brohan et al. “RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control”). To control the robot such text tokens can be de-tokenized, i.e. mapped, into robot actions, e.g. “A: 132114128525156” can de-tokenize to “Δ ^^ ൌ ^0.1,െ0.2,0^ Δ ^^ ൌ ^10^, 25^,െ7^^.

[0059] In some implementations the neural network 110 comprises a sequence modeling neural network in particular an auto-regressive neural network; more particularly in implementations the neural network 110 can comprise a (trained) Transformer neural network.

[0060] In general a Transformer neural network is characterized by having a succession of attention, e.g. self-attention neural network layers. A self-attention neural network layer has an attention layer input for each element of the input and is configured to apply an attention mechanism over the attention layer input to generate an attention layer output for each element of the input. There are many different attention mechanisms that may be used. For example a self-attention layer can be one that maps a query and a set of key-value pairs, each derived from an input to the self-attention layer (e.g. all vectors), to an output from which an output of the self-attention layer is derived. The output can be computed as a weighted sum of the values, weighted by a similarity function of the query to each respective key.

[0061] In general a Transformer neural network as described herein receives a sequence of input tokens as the neural network input 124, and is configured to apply the succession of attention neural network layers to the input sequence to generate a sequence of output tokens that comprises a transformed input element for each element of the sequence of input tokens. DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application

[0062] A “token” as used in this specification is a vector of numerical values having a specified dimensionality, i.e. the number of numerical values is constant across different tokens. A data item or an action may be tokenized by, for example, converting elements of the data item or action into respective tokens from a vocabulary comprising a discrete set of tokens, each token comprising a respective predetermined or learned embedding (an ordered collection of numerical values having a pre-determined dimensionality).

[0063] A Transformer neural network as described herein can may comprise an encoder neural network coupled to a decoder neutral network, each having a succession of (self- )attention neural network layers, or it may comprise a so-called decoder-only Transformer neural network. The Transformer neural network can be (but need not be) a causal Transformer neural network e.g., an encoder-decoder Transformer neural network with a causal Transformer decoder or a causal decoder-only Transformer neural network. An example of a Transformer neural network is described in Vaswani et al., 2017 arXiv:1706.03762); an example of a Transformer neural network controlling an agent is described in Reed et al.2022, arXiv:2205.06175v3.

[0064] In implementations in which the neural network 110 comprises a (trained) Transformer neural network a sequence of input tokens (image tokens) can be generated using any suitable image tokenizer, e.g. as described later, and processed using the trained Transformer neural network to generate the policy output 112. The policy output 112 can then comprise a sequence of output tokens that defines one or more actions, e.g. as previously described.

[0065] The system is configured to generate a trajectory image 122. The trajectory image 122 represents a desired trajectory for a part of the robot 102, e.g., of an end effector such as a gripper (e.g. of a defined point of the part, such as a center point of the end effector), or of another part of an arm of the robot, or of part of a body of the robot. Part or all of the robot 102 may be moved to achieve the desired trajectory.

[0066] In implementations the trajectory image 122 is a two-dimensional image that represents, or contains, the desired trajectory as a line in the trajectory image, e.g. on a blank or uniform background.

[0067] The line indicating the desired trajectory is referred to as a trajectory sketch; an example is illustrated in FIG.3, described later. In general the trajectory sketch is two- dimensional, i.e. it is defined as a line in a (2D) plane. In some implementations the trajectory sketch may indicate distance of the part of the robot from a reference level or DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application plane, e.g. by changing a property of the line such as color, but in such implementations the trajectory sketch itself remains two-dimensional.

[0068] In implementations of the described techniques the trajectory sketch is approximate; it indicates desired motion of the part of the robot but the system is not configured to follow the trajectory sketch exactly. In many cases following the trajectory sketch exactly would be undesirable as can be seen from the example trajectory sketches in, e.g. FIGS.4 to 8 described later. That is, the line in the trajectory sketch can indicate undesired types of motion. Instead the (trained) neural network system 100, and in particular the (trained) neural network 110, uses the trajectory sketch as an indication of the type of motion that, based on its training, it infers is desired.

[0069] Implementations of the neural network system 100 can also generalize, as illustrated later. That is, the neural network system 100 can infer new types of motion that are indicated in a trajectory sketch, even if these do not precisely match examples of motion on which the system was trained.

[0070] The trajectory sketch is obtained from a user, e.g. via a user interface 150. In general the user is a human user, but the user can also or instead be a machine. Some examples of how to generate the trajectory image 122 that represents the trajectory sketch are described later with reference to FIGS.4-8.

[0071] The neural network system 100 includes an image encoder 120; this can be implemented as an image encoder neural network. The image encoder 120 is configured to process the observation image 106 and the trajectory image 122, e.g. according to learned image encoder parameters such as weights, to generate a representation of a combination of the observation image and the trajectory image. The image encoder 120, e.g. the image encoder neural network, can be pre-trained, for example on a standard image training dataset such as ImageNet.

[0072] In implementations the image encoder 120 processes a combined image comprising a combination of the observation image 106, and optionally one or more previous observation images, and the trajectory image 122. For example the observation image 106 and the trajectory image 122 can be merged, concatenated, or otherwise combined. Where an observation at a time step comprises multiple observation images, e.g. a current image and images representing a recent history of the environment 104, each of the observation images can be combined with the trajectory image 122.

[0073] In some implementations the image encoder 120, in particular the image encoder neural network, includes one or more convolutional neural network layers to process the DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application observation image(s) 106 and the trajectory image 122. Instead of, or in addition to, one or more convolutional neural network layers other type of neural network layer can be used, e.g. feedforward layers, or attention layers (e.g. as used in a vision transformer, Dosovitskiy et al. arXiv:2010.11929).

[0074] Where the observation at a time step includes additional data to the observation image 106, such as proprioceptive data from the robot or other data captured by other sensors of the robot, this additional data can, e.g., be encoded jointly with the observation image by the image encoder neural network.

[0075] The neural network input 124 is generated from at least the representation of the combination of the observation image(s) 106 and the trajectory image 122. Thus the policy output 112 is generated using the trajectory sketch as context.

[0076] Thus in implementations the (trained) neural network 110 is conditioned on, i.e. processes a representation that depends on, a two-dimensional trajectory image 122 that depicts a two-dimensional trajectory sketch that deliberately underspecifies a task, rather than on, e.g. a 3D trajectory: Implementations of the described system do not construct a 3D motion trajectory to guide the robot (condition the neural network 110). In implementations, to perform a task in inference the neural network 110 is conditioned on, i.e. processes a representation that depends on, a single a two-dimensional trajectory image 122 that depicts a single two-dimensional trajectory sketch.

[0077] As an example, the observation image 106 and the trajectory image 122 can be concatenated to generate a concatenated image, and the concatenated image can be processed to generate a representation of the concatenated image as the representation of the combination of the observation image and the trajectory image. As some examples, the observation image 106 and the trajectory image 122 can be concatenated by joining the images horizontally, or vertically, by using the trajectory image 122 to provide pixel data for one or more extra color channels of the observation image 106, or can be concatenated along the feature dimension.

[0078] As an example of concatenation along the feature dimension, a color 2D image, e.g. an RGB image can have three dimensions (the color channels defining the third dimension), and two or more such images can be stacked in a fourth dimension. This 4D stack of images can provide an input to, e.g., a convolutional neural network layer that is a first layer of the image encoder neural network. During training as described later the weights of the additional input channels can be initialized, e.g., with all zeros. DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application

[0079] In this example, where an observation at a time step comprises multiple observation images, e.g. a current image and one or more previous observation images, each of the observation images can be concatenated with the trajectory image 122 along the feature dimension. The image encoder 120 can then process multiple combined images, e.g. six combined images if there are six observation images. Alternatively all the multiple observation images, and the trajectory image 122 may all be concatenated along the feature dimension.

[0080] In implementations where the neural network input 124 comprises a sequence of input tokens, and where each of the observation images is concatenated with the trajectory image 122, this sequence can be generated from (at least) the representation of the combined, e.g. concatenated, image. Optionally a sequence of image tokens for the combined, e.g. concatenated, image may be combined with one or more respective sequences of image tokens for one or more earlier combined, e.g. concatenated, images from one or more earlier time steps.

[0081] In some implementations a sequence of input tokens for a combined, e.g. concatenated, image can be generated by dividing the image into patches and processing each patch using one or more convolutional, feedforward, or attention-based neural network layers.

[0082] Optionally a position encoding can be applied to each input token in the sequence of input tokens for the combined or concatenated image and, where present, to each input token in the respective sequences of input tokens for the one or more earlier combined or concatenated images. The position encoding can, e.g., identify to which observation image each token of a sequence of tokens belongs and / or the position encoding can encode the position, e.g. index, of a token in a sequence. Any suitable position encoding may be used, e.g. an embedding of an index, a Fourier position encoding, a rotary position encoding, and so forth.

[0083] In some implementations the representation of the combined or concatenated image comprises a feature map that includes a respective feature vector for each of a plurality of regions in the combined or concatenated image. Where the neural network input 124 comprises a sequence of input tokens these feature vectors or feature maps may be converted into tokens.

[0084] Generating a sequence of input tokens for the concatenated image can involve generating an initial input sequence by flattening the feature map into a sequence of feature vectors. In some implementations the initial input sequence of feature vectors is processed DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application using a learned module that maps an input sequence to a reduced sequence that includes a smaller number of tokens.

[0085] FIG.2 shows details of an illustrative example of the system of FIG.1 that may be implemented as one or more computer programs on one or more computers in one or more locations

[0086] In FIG.2 the trajectory image 122 represents a trajectory sketch 130, described in more detail later. A set of observation images 106 from the current and previous time steps, and the trajectory image 122, are concatenated in a concatenation stage 121 of the image encoder 120. Merely as an illustration, the image encoder 120 can comprise a convolutional neural network front end, i.e. input portion, such as EfficientNet, e.g. variant B3, that has 300 x 300 resolution (Tan et al., arXiv:1905.11946v5, 2020), pre-trained on ImageNet. Each of six observation images at a time step (representing the current observation image and five previous observation images) can be concatenated with the trajectory image 122 along the feature dimension, to obtain a spatial feature map of size 9 x 9 x 512 from the final convolutional layer. The spatial feature map can then be flattened into 81 tokens for the neural network input 124.

[0087] Were additional language conditioning to be used this could be done by conditioning the image encoder 120 on an embedding of a language instruction, e.g. by incorporating FiLM (feature-wise Linear Modulation) layers (Perez et al., arXiv:1709.07871) into the image encoder as described in Brohan et al., “RT-1: Robotics Transformer for Real-World Control at Scale”, arXiv:2212.06817v1, 2023.

[0088] Optionally the, e.g.81, tokens can be subsampled to reduce the number of tokens, e.g. down to 8. This can be done using an element-wise attention module that learns to map a large number of tokens into a smaller number of tokens, such as that described in TokenLearner (Ryoo et al., 2021). This can soft-select tokens based on their information, passing only the important token combinations to the neural network input 124.

[0089] The, e.g.8, tokens for a combined image can then be concatenated with those of the other observation images in the history, forming, e.g.48 total tokens (with added position encoding), that provide the neural network input 124.

[0090] This sequence of input tokens is processed by the (trained) neural network 110 which in this example comprises a decoder-only Transformer with, e.g.8 self-attention layers, to generate the sequence of output tokens that define actions to be taken by the robot 102. Output tokens are generated until a “terminate” action is generated or until a preset DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application time limit is reached. The actions can be generated at real-time rates, e.g. in the range 1- 10Hz.

[0091] Merely as an example, a robot from Everyday Robots might have a 7 DoF arm, i.e. seven variables for the arm movement: x, y, z, roll, pitch, yaw, opening of the gripper; three variables for base movement: x, y, yaw (where the base is mobile); and a discrete variable to switch between three modes: controlling arm, base or terminating the episode. To tokenize the actions each of these action dimensions could be discretized into, say, 256 bins, which could be uniformly distributed within the bounds of each variable. Then the value of each of the action dimensions may correspond to one of the tokens in the vocabulary.

[0092] In general the neural network system 100, i.e. the neural network 110 and optionally also the image encoder 120, can be trained using training data comprising observation images and corresponding actions representing a plurality of demonstrated tasks (“demonstration tasks”). Trajectory images representing trajectory sketches for training the system can be generated from such training data.

[0093] As a particular example the neural network system 100 can be trained using an imitation learning method (i.e. using an imitation learning loss), such as behavioral cloning, inverse reinforcement learning, or Generative Adversarial Imitation Learning (GAIL. arXiv:1606.03476, Ho et al.). In some implementations the neural network system 100 is partially or wholly trained using training data from a simulation of the robot 102 performing simulated actions in a simulation of the real-world environment 104 to perform a task, before the system is used in the real-world. Training is described in more detail later.

[0094] FIG.3 illustrates an example trajectory image 122 that represents an example trajectory sketch 130. As illustrated the trajectory sketch is a 2D trajectory represented as the line in the two-dimensional trajectory image 122, the line representing a desired trajectory for a part of the robot. The example trajectory sketch 130 includes indications or “interaction markers” 132a,b of actions to be performed by the robot at respective points along the desired trajectory. Different actions, e.g. opening and closing of a gripper, may be distinguished, e.g. by using markers of different shapes or (in this example) colors.

[0095] The example trajectory sketch 130 also includes an indication of a desired height of the part of the robot along the desired trajectory, in this example by encoding the height into one of the color channels (e.g. the green channel). This can help disambiguate the desired motion of the robot for certain tasks. The example trajectory sketch 130 further includes an indication of a desired temporal progress of the part of the robot along the desired trajectory encoded into another of the color channels (e.g. the red channel). DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application

[0096] FIGS.4-8 illustrate some techniques for obtaining a trajectory sketch 130 from a user and generating a (2D) trajectory image 122 that represents the trajectory sketch. The user may be enabled to use more than one of these to generate a trajectory image representing the trajectory sketch.

[0097] In the example of FIG.4 the user interface 150 includes a graphical user interface. Obtaining the trajectory sketch involves capturing an image of the environment 104 where the task is to be performed, displaying the captured image on the graphical user interface, and enabling the user to draw a trajectory sketch 130 over the captured image to obtain the trajectory sketch, e.g. using a finger, pen, or mouse. The trajectory sketch 130 can be drawn on a blank or uniform canvas to generate the trajectory image 122.

[0098] The user interface 150 can also enable the user to input additional information for the trajectory sketch. For example the user interface may be configured to enable the user to annotate the trajectory sketch with height (vertical distance) values, or indications of one or more actions to be performed by the robot, e.g. by clicking with a mouse to indicate a gripper opening or closing, or a region of the trajectory where robot motion is constrained. The user interface may also capture an indication of desired relative temporal progress, e.g. speed of motion, of the robot.

[0099] FIG.5A illustrates an example GUI for the neural network system 100. In FIG.5A a camera image 500 is overlaid with a trajectory sketch 130 that is drawn by manual input from the user, and annotated with an interaction markers 132. The GUI also enables the user to add waypoints 134a-c corresponding to user-specified heights (entered via a menu). FIG.5B illustrates the trajectory image 122 generated by the GUI and representing the comprising trajectory sketch 130, which indicates the desired action and height of the part of the robot.

[0100] In general the trajectory sketch is a 2D trajectory represented as the line in the two- dimensional trajectory image and that is projected into, i.e. exists in, an image space that corresponds to (matches) a three-dimensional field of view of the image sensor, e.g. camera. That is the desired trajectory, represented as a 2D line in the 2D trajectory image is, in effect, projected into the field of view of the image sensor, e.g. camera.

[0101] The trajectory image 122 can be generated so that it matches a field of view or lateral extent of the observation image, e.g. so that it corresponds to the same region of physical space as the observation image 106. Thus in implementations, were one image to overlie the other the line depicting the desired trajectory in the trajectory image would indicate a desired motion of the part of the robot in real space as it would be seen by the DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application image sensor. In some implementations this can be facilitated by using a so-called geometric camera calibration (“camera resectioning”). However in general there is no need for a very precise camera calibration as the trajectory sketch is (by design) approximate.

[0102] Where the image sensor, e.g. camera, is mobile, e.g. where the robot is mobile and the camera is mounted on the robot, the representation of the trajectory sketch in the trajectory image may be modified to compensate for the motion. It is not necessary for the part of the robot, e.g. the end effector, always to be in view; for example the trajectory sketch can be snapped to the edge of the trajectory image if the end effector is out of view.

[0103] FIG.6 illustrates an example implementation in which the trajectory image 122 representing the trajectory sketch 130 is generated based on a video of the user demonstrating the desired motion.

[0104] This can involve capturing a (2D) video of the user using their hand to simulating performing the task in the environment. The captured video can then be processed to obtain a sequence of images each comprising a set of landmarks on the user’s hand. Any of a range of image processing tools can be used for this, e.g. Mediapipe, Lugaresi et al., 2019, arXiv:1906.08172. The trajectory sketch can then be determined from the sets of landmarks in the sequence of images.

[0105] As an example the set of landmarks in each image can be processed to extract landmarks on a finger and / or thumb of the hand, and the landmarks on the finger and / or thumb of the hand to identify one or more end effector actions, e.g. gripper actions, on the desired trajectory. A marker as previously described can then be added to the line representing the desired trajectory to indicate a location of each end effector action. For example, two points on the thumb and another two points on a finger, e.g. the index finger, can be extracted from the set of landmarks to represent a parallel gripper. These landmarks can then be used to identify the location of the end effector and / or a grip or drop action along the trajectory.

[0106] For example, the four 2D landmarks can be lifted into 3D given a depth map, e.g. from Mediapipe, a pose of the end effector can be determined from these four points. The estimated poses can then be converted to a 2D trajectory sketch, which can be manually annotated to indicate when the hand begins to grasp and release a target object.

[0107] FIG.7 illustrates two example implementations in each of which the trajectory image 122 representing the trajectory sketch 130 is generated based on natural language instructions from the user. Both of these example implementations use so-called foundation models to process the natural language instructions. DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application

[0108] In one of these obtaining the trajectory sketch 130, and generating the trajectory image 122, can involve obtaining a natural language instruction from the user to perform a task, and then converting the natural language instruction into a trajectory sketch represented in a trajectory image.

[0109] More specifically this can involve capturing an image of the environment 104 where the task is to be performed, obtaining a natural language input from the user comprising a natural language instruction to perform a task using an object, and processing the captured image and the natural language instruction using a first neural network system to generate a set of coordinates identifying a location of the object or of the part of the robot.

[0110] The first neural network system can comprise a visual language model configured, e.g. with an appropriate prompt (that can optionally specify the object or robot part), to perform an object detection task that generates the set of coordinates. In some implementations first neural network system can generate an object detection image comprising a version of the captured image including a bounding box for the object and / or robot part. As an example, FIG.8A illustrates detection of objects of interests, i.e. drawer handles, by a visual language model for an Open Drawer task.

[0111] Any suitable visual language model may be used such object detection. Some examples include ViLD, Gu et al. “Open-vocabulary object detection via vision and language knowledge distillation” arXiv:2104.13921, 2021, intended for the mobile robot domain, and MDETR, Kamath et al. “Mdetr-modulated detection for end-to-end multi- modal understanding”, ICCV, 2021, intended for the tabletop manipulation.

[0112] The set of coordinates and the natural language instruction using can then be processed using a second neural network system to generate coordinates for a set of waypoints along which to move the object or part of the robot to perform the task. In implementations the second neural network system comprises a so-called large language model (LLM), or VLM, in particular an LLM (or VLM) trained on code completion, i.e. an LLM that can write computer code. In some implementations the second neural network system may be the same as the first neural network system.

[0113] In more detail, such an LLM can be used to write robot policy code given natural language commands. That is, the LLM can write code that processes perception outputs from object detectors and parameterize robot control primitive APIs. When prompted with several example language commands (formatted as comments) followed by corresponding policy code LLMs can take in new commands and autonomously re-compose API calls to generate new policy code. DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application

[0114] Given below is an example of the LLM output that comprises code to generate trajectories given the task instructions ("open the top drawer") and object poses as well as object sizes: # There are two drawer handles, and the top-most should be compared along the z-axis. # First handle has z-value of 0.63, second has z-value of 0.12. # The positive z direction corresponds to up, so the first handle is the top-most. objects['drawer handle'][0]['centroid_pose']['position'] top_drawer_handle_orientation = objects['drawer handle'][0]['centroid_pose']['orientation'] top_drawer_handle_size = objects['drawer handle'][0]['size'] # The handle has a bounding box size of [0.00 , 0.12 , 0.06] in meters and is located on the front x-y plane of the cabinet. # The gripper has a max span of 10 cm, the size of the handle along the y-axis is 0.12 so it can only grasp the handle with fingers aligned along the x axis and z axis. # A bottom grasp is ruled out since the robot would collide with the cabinet. # A back grasp is unfeasible since the robot is in front of the cabinet and cannot go around it to make a back grasp. # A top grasp with fingers aligned with the x-axis or z-axis is feasible. # A front grasp with fingers aligned with the z-axis and a side grasp with fingers aligned with the x-axis are feasible too. # We choose the front grasp with fingers aligned with the z-axis for simplicity. # Get quaternion corresponding to [-90, 0, -90] roll,pitch and yaw for a front grasp with fingers aligned with the z-axis. # Calculate grasp position so object ends within gripper fingers. # The pregrasp pose is the pose right before the grasp. # Since this is a front grasp, this means the gripper is pointing towards the positive x axis, so the pregrasp_pose has a negative x delta over the grasp pose. # Calculate pregrasp pose accounting for object size and gripper size (0.1 m). + [- , 'orientation': # Open the gripper according to the z axis size of the handle plus a buffer of 2 cm. # handle pose is not valid anymore since we might have moved the base so we use the current arm pose. # The handle is at the front of the cabinet. Opening means moving the object (handle) away from their reference (cabinet) along the x axis. DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application # The cabinet is at x_cabinet = 1.12 and the handle is at x_handle = 0.69. # When the object coordinate is lower than its reference, to increase distance you need to substract a delta and to decrease distance you need to add a delta. # When the object coordinate is greater than its reference, to increase distance you need to add a delta and to decrease distance you need to substract a delta. # Since x_handle is lower than x_cabinet, it means the object coordinate is lower than its reference, so to increase the distance between the two we substract a positive delta to x_handle. current_arm_pose['position'] + [-0.25, 0, 0], 'orientation': current_arm_pose['orientation']} # Allow for base moves for after grasp moves since arm could be in a difficult position to execute the open. , calls, results in a sequence of target robot waypoints. For example the code, including the API calls, can generate a series of robot poses (originally intended to be executed using a motion planner such as an Inverse Kinematics motion planner), that can be used to obtain the target robot waypoints.

[0116] Alternatively the LLM or VLM can generate 3D coordinates for the target robot waypoints directly, i.e. without an intermediate code generation step.

[0117] The trajectory image 122 representing the trajectory sketch can then be determined from the set of waypoints.

[0118] FIG.8B shows, on the left, a frame from an example video of a policy rollout of a robot performing an open drawer task, and on the right the same frame with a trajectory image, generated as described above, overlaid. Although the trajectory sketch 130 in the trajectory image comprises precise, linear trajectories the system is nonetheless able to interpret these, and unlike motion planners can adapt to scene nuances such as object orientation.

[0119] This is described in detail described in Liang et al. “Code as Policies: Language Model Programs for Embodied Control”, arXiv:2209.07753, in particular in sections III-B, IV-B, and IV-C as well as in Appendix G, “Visual Language Models”, and H (which refers to waypoints), to which reference can be made.

[0120] In another of the example implementations illustrated in FIG.7 the trajectory image 122 representing the trajectory sketch 130 is generated directly from natural language instructions and an image of the environment 104, using a (fine-tuned) visual language model (VLM). DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application

[0121] Since the trajectory sketch 130 is represented as a trajectory image, text-guided image generation models can be used to generate the trajectory sketch when provided with an initial image and a language instruction which describes the task.

[0122] Thus obtaining the trajectory sketch 130 can involve capturing an image of the environment 104 where the task is to be performed, obtaining a natural language input from the user comprising a natural language instruction to perform a task, e.g. using an object. The captured image and the natural language instruction can then be processed using a visual language model neural network system to generate an image output i.e. the trajectory image 122, comprising the trajectory sketch 130.

[0123] FIG.9 shows, on the left, an example image output i.e. a trajectory image 122 comprising a trajectory sketch 130, generated in this way, overlaid on the image of the environment 104. The input to the VLM comprises the image of the environment 104 and the natural language instruction “pick orange can from top drawer and place on counter”. FIG.9 shows, on the right, a frame from a video of the robot performing the task when conditioned on the trajectory sketch. Although the trajectory sketch in the trajectory image is noisy, and in this example looks different to the trajectory sketches used during training (described later), the system is nonetheless able to interpret the trajectory sketch.

[0124] There are many visual language model (VLM) neural network systems that can be used. As one particular example the visual language model neural network system can be based on PaLM-E, Driess et al. “PaLM-E: An Embodied Multimodal Language Model”, arXiv:2303.03378, which can be used to generate image tokens that represent the trajectory image. In this example the image tokens can comprise vector-quantized image tokens derived from a ViT-VQGAN, Yu et al., “Vector-Quantized Image Modeling with Improved VQGAN” arXiv:2110.04627, which uses a Transformer neural network to predict image tokens autoregressively, discrete image tokens being produced from ViT-VQGAN image quantizers. The image tokens can then be detokenized to generate the trajectory image 122; or, depending on the representation used, the image tokens can be detokenized into 2D pixel coordinates for drawing the trajectory sketch.

[0125] In general a so-called foundation model VLM can generate a trajectory image from an image and a language instruction. As one example, a VLM can be fine-tuned (i.e. a trained VLM can be further trained), on a trajectory image generation task to that it directly outputs (RGB) pixels of a trajectory image representing a trajectory sketch of a desired format given the image and the language instruction. The VLM can be trained, e.g. using trajectories obtained using one or more of the previously described techniques to generate DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application training data. As another trajectory examples and a description of the trajectory format can be included in a prompt for the VLM so that the VLM generates a trajectory image representing a trajectory sketch of a desired format.

[0126] A trajectory image representing a trajectory sketch can also be obtained in other ways.

[0127] As one further example, obtaining the trajectory sketch can involve obtaining proprioceptive data from proprioceptive sensors of the robot as the user guides the (part of the) robot through the desired trajectory.

[0128] In general proprioceptive data comprises data obtained from sensors of the robot that sense a state, in particular an internal state, or a configuration of the robot, such as characteristics related to position, movement or actions of the robot or of parts of the robot, such as data relating to position, orientation, joint angle, force or torque, linear or angular speed or acceleration, load, electrical power, current or voltage.

[0129] The proprioceptive data can be processed, e.g. as described later, to generate the trajectory sketch 130 and trajectory image 122.

[0130] FIG.10 is a flow diagram of an example process for using a neural network system including a (trained) neural network to control a robot interacting with an environment to perform a task. The process can be performed by a system of one or more computers located in one or more locations, and for convenience will be described as performed by the neural network system 100 of FIG.1.

[0131] At step 1000 the process generates the trajectory image 122 representing the trajectory sketch 130. The trajectory sketch 130 can obtained directly (e.g. by drawing or similar), or indirectly (e.g. by text or audio natural language instruction), from a user, e.g. as described above.

[0132] An observation image 106 is obtained and processed at each of a plurality of time steps, to select an action 114 to be performed by the robot 102 at the time step.

[0133] More particularly this involves obtaining, from the real-world environment 104, an observation image 106 characterizing a state of the environment at a current time step (step 1002).

[0134] The observation image and the trajectory image are processed using the image encoder 120 to generate a representation of a combination of the observation image and the trajectory image to generate the neural network input 124 (step 1004).

[0135] The neural network input 124 is processed by the (trained) neural network 110 to generate the policy output 112 (step 1006). In some implementations this involves DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application generating a sequence of input tokens (image tokens) from at least the representation of the combination of the observation image and the trajectory image.

[0136] The policy output 112 is then used to select at least one action to be performed by the robot 102 to perform, and the robot is caused to perform the selected action (step 1008).

[0137] In some implementations of the process obtaining the trajectory sketch involves obtaining from the user a relative speed of motion along the desired trajectory for the part of the robot when performing the user-directed task. Then generating the trajectory image can include depicting the speed of motion on the line representing the desired trajectory, e.g. using a color component of the line.

[0138] In some implementations obtaining the trajectory sketch involves obtaining from the user an indication of relative temporal progress along the desired trajectory for the part of the robot when performing the user-directed task, e.g. by capturing the desired trajectory in combination with a series of time stamps. Then generating the trajectory image can include depicting the relative temporal progress on the line representing the desired trajectory, e.g. using a color component of the line or by changing the appearance of the line in some other way.

[0139] In some implementations obtaining the trajectory sketch involves obtaining an indication of a vertical distance, e.g. height, of the part of the robot from a reference level at one or more points along the desired trajectory, e.g. a distance along a z-axis that defines a vertical direction of motion of the part. The vertical distance can then also be determined at additional points along the desired trajectory from the vertical distance at the one or more points, e.g. by interpolation or extrapolation, e.g. to provide vertical distance information for the full trajectory. Generating the trajectory image can then include depicting the vertical distance at the one or more points and at the additional points, on the line representing the desired trajectory, e.g. using a color component of the line or by changing the appearance of the line in some other way.

[0140] In some implementations obtaining the trajectory sketch can also involve obtaining from the user an indication of one or more particular actions to be performed by the robot at one or more respective points along the desired trajectory, such as opening and closing of a gripper. Generating the trajectory image can then include depicting at the respective points on the line representing the desired trajectory, one or more markers representing the one or more particular actions to be performed. Different particular actions, e.g. gripper opening and gripper closing, may be distinguished by using markers of different shapes or colors, or in some other way. DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application

[0141] In some implementations obtaining the trajectory sketch can also involve obtaining from the user an indication of a region along the desired trajectory where motion of the part of the robot is constrained. Generating the trajectory image can then include depicting the region on the line representing the region of the desired trajectory where motion is constrained.

[0142] In general in implementations of the process of FIG.10 the environment is a real- world environment and the method involves causing the (real-world) robot to perform the selected action in the real-world environment. However the process may also be performed using a simulated robot in a simulation of a real-world environment, e.g. before using the robot in the real world. This can, for example, be used to check that the robot is behaving safely before deploying the system in the real world environment, or it can be used to generate training data based on the trajectory sketches, that can in turn be used to train the or another neural network system.

[0143] The process uses the trajectory sketch 130 to guide the robot to perform a task, but in principle this form of guidance may be combined with other forms of guidance such as natural language guidance, although this is not necessary.

[0144] As an example a natural language text sequence that characterizes the task can be received and encoded, and the observation image and the trajectory image can be processed using the image encoder neural network, conditioned on the encoded representation of the natural language text sequence, to generate the representation of the (combination of the) observation image and the trajectory image. This can be done by providing the image encoder neural network with one or more conditioning layers, each configured to receive an intermediate output of an intermediate layer of the image encoder neural network output and the encoded representation of the natural language instruction and to update the respective intermediate output using the encoded representation and provide the updated output as input to a subsequent intermediate layer of the image encoder neural network. The conditioning layers may be FiLM layers; the intermediate layer and / or the subsequent intermediate layer may be a convolutional neural network layer.

[0145] FIG.11 shows an example system for training the neural network system 100 of FIG.1. The neural network system 100 can be trained on a corpus of training data 160 (demonstration data) that comprises a plurality of demonstration tasks under control of a training engine 170.

[0146] Each demonstration task may comprise a sequence of observations and actions that move an object along a demonstration trajectory for the demonstration task. The object can DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application be an object that is manipulated, e.g. grasped and / or released by a robot, or an object that is part of a robot such as an end effector of the robot. That is, the actions can be actions of a robot. The robot may be, but need not be, the same as the robot 102 that the system will be used to control after training. The demonstration tasks can be performed by a simulation of the robot in a simulation of a real-world environment of the robot, or by the robot in the real-world environment, or both.

[0147] In implementations the observations include image observations of the environment and proprioceptive observations, i.e. proprioceptive data from the robot. Trajectory images representing trajectory sketches for the tasks can be obtained from the training data, in hindsight; such trajectory sketches may be referred to as hindsight trajectory sketches. In implementations the hindsight trajectory sketches can be obtained from the proprioceptive data 162, e.g. by the training engine 170.

[0148] In general training data can be obtained by capturing observations images and proprioceptive data from a robot performing a variety of tasks. The robot can be manually guided and / or instructed to perform tasks using natural language (e.g. as described in Brohan et al. and Driess et al., ibid). As one particular example using a robot from Everyday Robots a dataset of about 73K real-world robot demonstrations across 542 seen tasks were collected by manual teleoperation. The tasks involved a set of 17 household kitchen items and 8 different manipulation skills, i.e. tasks; examples of the tasks are below: Skill Description Example Pick Object Lift the object off the surface pick iced tea can Move Object Near Object Move the first object near the move pepsi can near rxbar second blueberry Place Object Upright Place an elongated object place water bottle upright upright Knock Object Over Knock an elongated object knock redbull can over over Open Drawer Open any of the cabinet open the top drawer drawers Close Drawer Close any of the cabinet close the middle drawer drawers Place Object into Receptacle Place an object into a place brown chip bag into receptacle white bowl DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application Pick Object from Receptacle Pick an object up from a pick green jalapeno chip bag and Place on the Counter location and then place it on from paper bowl and place on the counter counter

[0149] The neural network system 100 can be trained, e.g., using a supervised learning objective based on the training data, i.e. based on demonstration observations and demonstration actions sampled from the training data. The supervised learning may based on a log likelihood of a selected action, and may include a regularization term. As another example the neural network system may be trained to optimize an objective function that depends on a difference between a distribution of actions defined by the policy output and a distribution of actions defined by the actions in the training data.

[0150] As a particular example, given an episode, ^^, comprising a sequence of pairs of observation images ^^௧, e.g. RGB images from a head camera of the robot, and actions ^^௧and a hindsight trajectory sketch, ^^௧^^^, the neural network 110 can be trained to minimize the negative log likelihood of the actions ^^௧given the observation images ^^௧and the hindsight trajectory sketch, ^^௧^^^.

[0151] FIG.12 is a flow diagram of an example process for training a neural network system, such as the neural network system 100 of FIG.1, to control a robot to perform a task. The process can be performed by a system of one or more computers located in one or more locations, e.g. by the system of FIG.11.

[0152] The process involves obtaining a corpus of training data that comprises a plurality of demonstration tasks (step 1200). Each demonstration task comprises a sequence of demonstration observation images 106 and corresponding demonstration actions 114 that move (at least) a part of the robot along a demonstration trajectory for the demonstration task.

[0153] A demonstration task can be a demonstration of a task by an expert, such as a human or another machine learning system. The demonstration observation images 106 characterize states of a demonstration environment in which a demonstrating agent, e.g. a robot, performs the task and the demonstration actions 114 comprise actions the demonstrating agent takes to perform the task. The demonstration environment may be the same as or different to the environment in which the agent, i.e. the robot 102, will later act. For example the demonstration environment may be a simulation of a real environment and the agent may later act in the real environment. The demonstrating agent or “expert” may DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application be a human, or another machine learning system, or both. Training in simulation can reduce unnecessary wear and tear on and damage to a real-world robot.

[0154] The training can involve generating, for each of the demonstration tasks, a trajectory image 122 representing the demonstration trajectory as a line in the trajectory image, i.e. a trajectory image representing a hindsight trajectory sketch 130 (step 1202). The line in the trajectory image can optionally depict speed of motion (of the part), relative temporal progress, vertical distance, one or more markers, and / or a motion-constrained region. An example of generating the hindsight trajectory sketch is described later.

[0155] In implementations the training involves, for each demonstration task, and for each observation image 106 and corresponding action 114 in the sequence, processing the (current) observation image and the trajectory image using the neural network system, in particular using the image encoder 120 of the neural network system. In implementations this involves combining each of the observation images 106 with the trajectory image 122 to generate a combined image and processing the combined image using the image encoder 120 of the neural network system.

[0156] In some implementations, in particular where the neural network 110 is configured to process a sequence of input tokens, processing the (current) observation image 106 and the trajectory image 122 can involve processing one or more previous observation images and / or features characterizing one or more previous actions taken.

[0157] Thus the observation image 106 and the trajectory image 122 are processed using the neural network system to generate the policy output (step 1204). In particular is done by processing the observation image 106 and the trajectory image 122 using the image encoder 120 to generate the neural network input 124, and processing the neural network input 124 using the neural network system 110 to generate the policy output 112.

[0158] The policy output 112 is used for selecting an action 114 to be performed by, i.e. that would be performed by, the robot (step 1206); there is no need for the robot actually to perform the action. In some implementations the policy output 112 can directly specify the action 114; in some implementations the action 114 is selected from a distribution specified by the policy output 112.

[0159] The neural network system 100 is then trained to encourage the actions selected using the policy output to match the corresponding actions in the training data (step 1208). This can be done using an objective function, e.g. loss function, that depends on a measure of a match or difference between an action selected using the policy output and a corresponding action in the training data. DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application

[0160] In general training the neural network system 100 involves backpropagating gradients of the objective function, e.g. a loss function, to update learnable parameters, e.g. weights, of the neural network system. This may use any appropriate gradient descent optimization algorithm, e.g. Adam or another optimization algorithm.

[0161] In implementations the neural network system is trained by an imitation learning method, i.e. using an imitation learning objective. There are various imitation learning techniques, as previously indicated. As one example the neural network system can be trained a behavioral cloning technique, e.g. using a supervised learning objective dependent on the training data, e.g. based on the log likelihood of a selected action.

[0162] In some implementations the image encoder 120 is trained jointly with the neural network system 100; in some implementations a pre-trained image encoder 120 can be used. In some implementations, prior to the joint training, the image encoder neural network may have been pre-trained e.g. on an image classification task.

[0163] In some implementations, as previously described, a pre-trained image encoder neural network is used and an input to the image encoder is expanded from an input size that corresponds to the observation image to an input size that corresponds to a concatenation of the observation image and the trajectory image. Then weights for additional inputs for the trajectory image may be initialized, e.g. to zero, and the image encoder neural network thereafter trained jointly with the neural network system.

[0164] In one example implementation of the above described system generating the sequence of input tokens from at least the representation of the concatenation of the observation image and the trajectory image is performed by an EfficientNet (Tan et al., arXiv:1905.11946), e.g. a B3 EfficientNet, e.g. pre-trained using an ImageNet. Then additional input channels used for processing the trajectory image part of the concatenated image can have weights that are initialized, e.g. to all zeros.

[0165] In some implementations some or all of the training data may be obtained by capturing examples of a human performing a particular task in a real or simulated environment; an example of generating data in this way is described in arXiv:2112.03762, Abramson et al., “Creating Multimodal Interactive Agents with Imitation and Self- Supervised Learning”, to which reference may be made for details. Then trajectory sketches for the demonstrated tasks may be obtained, e.g., by manual annotation.

[0166] Hindsight trajectory sketches.

[0167] FIG.13 illustrates generation of a hindsight trajectory sketch from proprioceptive information. In particular FIG.13 shows motion of a robot end effector, and an example of DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application a trajectory image 122 representing a hindsight trajectory sketch 130, generated from proprioceptive information from the robot.

[0168] As previously described, in some implementations the training data is obtained from a real or simulated robot. This can involve obtaining from the robot, for each of the demonstration tasks, proprioceptive information corresponding to motion of the part of the robot along the demonstration trajectory, and generating the trajectory image representing the trajectory sketch, here a hindsight trajectory sketch, from the proprioceptive information.

[0169] In implementations each of the observation images 106 in the training data has been captured using an image sensor e.g. a camera (or using a simulation of an image sensor, e.g. camera, if the training data is obtained from a simulated robot). Training the neural network system, in particular obtaining the training data can involve capturing each of the observation images using the image sensor.

[0170] In some cases a 3D demonstration trajectory of part of the robot, e.g. an end effector of the robot, may be available directly from the robot. In some cases a 3D demonstration trajectory can be determined from the proprioceptive information, e.g. from a 3D position of part of the robot, e.g. an end effector of the robot. The 3D demonstration trajectory can then be projected into an image space of the image sensor used for capturing of the observation images to obtain a 2D trajectory image 122 that comprises a line, e.g. on a blank background, that depicts the demonstration trajectory in 2D in the image space of the image sensor, i.e. a trajectory sketch 130.

[0171] Optionally to simplify the projection (but not necessarily), one or both of a base of the robot and the image sensor, e.g. camera, may be help stationary whilst a demonstration trajectory is collected. In some implementations vertical distance (height) information as previously described is also obtained from the proprioceptive information and represented on the line in the trajectory image.

[0172] Projecting the 3D demonstration trajectory (or coordinate) into an image space of the image sensor can be performed based on extrinsic and intrinsic parameters of the image sensor, e.g. camera, i.e. based on the trajectory reference frame and image sensor intrinsic parameters. This can be done by the robot system software; alternatively there are many standard software packages that can be used for such projection based on the robot system and camera parameters, e.g. Open CV (a library of computer vision functions at opencv.org) or ROS (Robot Operating System, open source robotics middleware at ros.org). DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application

[0173] In general the extrinsic parameters of the image sensor can define a location and orientation of the image sensor with respect to a world coordinate frame. More particularly the extrinsic parameters can define a mapping from the world coordinate frame to an image sensor coordinate frame, e.g. as an extrinsic transformation matrix.

[0174] In general the intrinsic parameters define a mapping from the image sensor coordinate frame to pixel coordinates in an image captured by the image sensor. For example the intrinsic parameters can define an intrinsic transformation matrix that transforms points from the image sensor coordinate frame to a pixel coordinate frame. Examples of intrinsic parameters include focal length, aperture, field-of-view, and resolution.

[0175] As a particular example, given the proprioceptive information recorded in an episode the 3D position of the robot end-effector center defined in the robot base frame at each time step is obtained and projected it to the camera space given the known camera extrinsic and intrinsic parameters, assuming that the robot base and camera do not move within the episode. Then given a 2D trajectory, e.g. as a sequence of pixel positions, a curve can be drawn on a blank image, by connecting 2D robot end-effector center points at adjacent time steps by straight lines, to obtain the trajectory image representing the trajectory sketch.

[0176] The line on the trajectory image can be augmented with additional information from the proprioceptive information or other information obtained from the robot, e.g. to add information representing speed of motion (of at least a part of the robot), relative temporal progress, vertical distance, one or more markers representing one or more actions, e.g. end effector actions, as previously described.

[0177] FIGS.14A to 14E illustrate an example robot trajectory (top) and different aspects of a corresponding example hindsight trajectory sketch (bottom) augmented with additional information. In particular FIG.14A illustrates interaction markers; FIG.14B illustrates temporal progress along 2D end effector waypoints; FIG.14C illustrates end effector height; FIG 14D illustrates a combination of the trajectory sketches of FIGS.14A and 14B; and FIG.14E illustrates a combination of the trajectory sketches of FIGS.14A, 14B, and 14C.

[0178] The interaction markers 132 generally indicate end effector actions and can, e.g., mark the time steps when a gripper begins to grasp, and to release, an object. There are many ways of obtaining interaction markers; merely by way of illustration one way is as follows: A determination is made of whether the gripper has contact with an object, e.g. by determining a difference ^^௧ൌ ^̂^௧െ ^^௧between sensed ( ^^௧) and target ( ^̂^௧) gripper joint DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application positions. If ^^௧^ 0 and ^̂^௧^ ^^, where ^^ is a threshold of a closing action (in this example ^^௧increases as the gripper closes), this can be taken as an indication that the gripper has closed and grasped the object. A status change can be determined, e.g., as ^^௧^ 0⋁^̂^௧^ ^^ and ^^௧ା^^ 0⋀^̂^௧ା^^ ^^ and the time step ^^ can then be considered as the time step for the closing action. A time step for an opening action can be found in a corresponding manner. The trajectory sketch can then be annotated with suitable markers, e.g. using different colored circles, at a center point of the end effector and at the identified time step(s).

[0179] As an example of expressing temporal progress, e.g. to encode speed along a trajectory, a color channel of the trajectory sketch can be used to indicate a normalized time step,௧ା^, where ^^ is a current time step and ^^ is a total episode length (in times steps). As an example of expressing height information, another color channel of the trajectory sketch can be used to indicate normalized height relative to, e.g., the robot base as ^^ೌ^ି^^^^.

[0180] Some implementations of the training generate augmented training data from the corpus of training data, or from using the neural network system to control a robot interacting with an environment to perform a user-directed task in the environment, or both.

[0181] In some implementations the trained system can guided by a trajectory sketch to perform a user-directed task, in the real-world or in simulation, and observation and action data from this can then be used for further training of the neural network system (using an imitation learning technique).

[0182] In some implementations additional training data can be generated from the corpus of training data. This can involve determining a modified trajectory image for some of the observation images, e.g. for randomly selected observation images, in particular where the combination of the observation image and the trajectory image is processed using the image encoder

[0183] This can be done by replacing the trajectory image with an image that depicts a limited part of the demonstration trajectory around a point in the demonstration trajectory that corresponds to the observation image, e.g. part of the demonstration trajectory defined by a window including or centered on a time step of the observation image, where the window has a width defined by a number of time steps. In implementations where one or more earlier combined, e.g. concatenated, images from one or more earlier time steps are processed the window can be similarly centered on the current (i.e. most recent) time step. This can increase a diversity of the trajectory conditioning. DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application

[0184] FIG.15 shows an example of a robot trained and instructed using the techniques described herein. The figure shows a video frame, with a trajectory sketch overlaid, of the robot performing a restock drawer task that requires the robot to place snacks into the drawer precisely into an empty slot.

[0185] FIG.16 shows an example of a robot trained and instructed using the techniques described herein, performing the task of moving an object within a drawer having only been trained on those motions at height of the tabletop.

[0186] FIG.17 shows an example of a robot trained and instructed using the techniques described herein performing a previously unseen task (i.e. the task was absent from the training data) of cleaning a table. Given a 2D trajectory sketch the robot successfully plans and executes a wiping trajectory.

[0187] FIG.18 compares the performance of example systems as described herein that use a 2D trajectory sketch (“RT-Traj (2D)”) and a 2D trajectory sketch with height information (“RT-Traj (2.5D”) with the performance of other systems, in particular RT-1 (ibid, a language-conditioned policy trained on the same training data), RT-2 (ibid, a language- conditioned policy trained on the same training data and internet scale VQA data), and RT- 1-goal (a goal-conditioned policy trained on the same training data). The illustrated tasks require combining seen motions in novel ways or generalizing to new motions; it can be seen that implementations overall perform significantly better than the other techniques.

[0188] In general implementations of the described techniques are robust to out-of- distribution visual distribution shifts and can generalize to new backgrounds, lighting conditions, objects, layouts, and environment geometries, and can successfully perform a variety of tasks requiring novel motion.

[0189] Implementations of the neural network system 100 as described above can show emergent behavior. For example the system can exhibit retry behavior in which, if the robot experiences difficulty in a step of the task, it can try the step again.

[0190] FIG.19 illustrates retry behavior when the neural network 110 conditioned on a trajectory sketch generated using the “Code As Policies” approach described above (left frame). The system first tries, and fails to grasp the drawer by its handle (middle frame), then tries, successfully, to open the drawer by grasping its edge (right frame). This behavior would not be possible in a system that was specifically required to follow a 3D motion trajectory.

[0191] The use of a trajectory sketch as described above can also facilitate user adaption of a trajectory sketch to accomplish a task, reminiscent of language model prompt engineering. DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application That is, where a sub-optimal trajectory sketch results in task failure the user can adapt the trajectory sketch so that the task succeeds.

[0192] FIG.20 illustrates user adaption of a trajectory sketch, showing failure of a task (to place an apple on the top stage) on the top, and success on the bottom. The trajectory sketches are shown on the left; the successful trajectory placed the apple from above. It will be appreciated that neither of the 2D trajectory sketches dictates how the end effector of the robot should move in 3D; the 3D motion is inferred by the trained neural network system from the 2D trajectory sketch.

[0193] The one or more computers used to implement the techniques described herein may be on-board the robot 102, remote from the robot, or partly on-board and partly remote. For example in some implementations the observation images 106 and trajectory sketch 130 or trajectory image 122 may be transmitted to a remote computer system over a data communication network, and the data specifying the selected action may be received back over the data communication network.

[0194] The robot 102 can include a control system to provide control signals to control the robot in accordance with the selected actions, e.g. to provide control signals to electromechanical devices to control movement or locomotion of the robot in a real-world environment. Such a control system may be configured to receive data in any format that defines one or more actions to be performed by the robot.

[0195] The robot 102 may be any sort of static or mobile robot interacting with the environment to accomplish any specific task, e.g., to locate an object of interest in the environment, or to manipulate a specified object, or to move a specified object to a specified location in the environment, or to navigate to a specified destination in the environment. The object, location, or destination may be specified by the trajectory sketch.

[0196] An observation image 106 may be obtained, for example from a camera or other image sensor e.g. a LIDAR sensor. As used herein an “image” includes a point cloud; an image may be a still or moving image. The camera or other image sensor may be mounted on the robot or located separately from the robot in the environment.

[0197] Other observations may also be processed, e.g. from other sensor data such as object position data, data from a distance or position sensor, data from an actuator, or sensed electronic signals such as motor current or a temperature signal.

[0198] Proprioceptive data from a robot may include data characterizing the current state of the robot, e.g., one or more of: joint position, joint velocity, joint force, torque or DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application acceleration, e.g., gravity-compensated torque feedback, and global or relative pose of an item held by the robot or of one or more parts of the agent.

[0199] The observation at any given time step may include data from a previous time step that may be beneficial in characterizing the environment.

[0200] The actions can generally comprise control signals to control the robot e.g., torques for the joints of the robot or torques to a control surface or other control elements e.g. steering control elements of a vehicle, or higher-level control commands. The control signals can include for example, position, velocity, or force, torque, or acceleration data for one or more joints of a robot or parts of another mechanical agent. The control signals may also or instead include electronic control data such as motor control data or, for a vehicle, signal to control navigation, e.g., steering, and movement, e.g., braking and / or acceleration.

[0201] In the case of a simulated environment the observations may include simulated versions of one or more of the previously described observations or types of observations and the actions may include simulated versions of one or more of the previously described actions or types of actions.

[0202] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0203] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0204] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0205] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0206] In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently.

[0207] Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0208] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0209] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0210] Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

[0211] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0212] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

[0213] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a Jax framework.

[0214] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0215] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0216] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0217] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0218] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application CLAIMS 1. A computer-implemented method of using a trained neural network system including a trained neural network, to control a robot interacting with an environment to perform a task in the environment, the method comprising: generating a trajectory image representing a trajectory sketch that indicates a desired trajectory for a part of the robot when performing the task, wherein the trajectory image is a two-dimensional image that represents the desired trajectory as a line in the trajectory image; and, at each of a plurality of time steps: obtaining an observation image characterizing a state of the environment at the time step; processing the observation image and the trajectory image using at least one image encoder to generate a representation of a combination of the observation image and the trajectory image; generating, from at least the representation of the combination of the observation image and the trajectory image, a neural network input; processing the neural network input using the trained neural network to generate a policy output for selecting at least one action to be performed by the robot in response to the observation image; selecting at least one action to be performed by the robot using the policy output; and causing the robot to perform the selected at least one action.

2. The method of claim 1, wherein trained neural network comprises a trained Transformer neural network; wherein the processing the observation image and the trajectory image using at least one image encoder to generate the representation of the combination of the observation image and the trajectory image comprises processing a combination of the observation image and the trajectory image using the at least one image encoder to generate the representation of the combination of the observation image and the trajectory image; wherein the neural network input comprises a sequence of input tokens, the method comprising generating the sequence of input tokens from at least the representation of the combination of the observation image and the trajectory image; andDeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application wherein processing the neural network input using the trained neural network to generate the policy output comprises processing the sequence of input tokens using the trained Transformer neural network to generate the policy output.

3. The method of claim 1 or 2, wherein obtaining the observation image comprises capturing an image of the environment with an image sensor; and wherein the task is a user- directed task; the method further comprising: obtaining the trajectory sketch from the user; and generating the two-dimensional trajectory image from the trajectory sketch, wherein the trajectory sketch is a 2D trajectory represented as the line in the two-dimensional trajectory image and that is projected into an image space that corresponds to a three- dimensional field of view of the image sensor.

4. The method of any one of claims 1-3, wherein obtaining the trajectory sketch further comprises obtaining, from a user, a relative speed of motion along the desired trajectory for the part of the robot when performing the task, and wherein generating the trajectory image includes depicting the speed of motion on the line representing the desired trajectory.

5. The method of any one of claims 1-4, wherein obtaining the trajectory sketch further comprises obtaining, from a user, an indication of relative temporal progress along the desired trajectory for the part of the robot when performing the task, and wherein generating the trajectory image includes depicting the relative temporal progress on the line representing the desired trajectory.

6. The method of any one of claims 1-5, wherein the trajectory image is a two- dimensional trajectory image, wherein obtaining the trajectory sketch further comprises obtaining an indication of a vertical distance of the part of the robot from a reference level at one or more points along the desired trajectory, the method further comprising determining the vertical distance at additional points along the desired trajectory from the vertical distance at the one or more points, and wherein generating the trajectory image includes depicting the vertical distance at the one or more points and at the additional points, on the line representing the desired trajectory.DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application 7. The method of any one of claims 1-6, wherein obtaining the trajectory sketch further comprises obtaining, from the user, an indication of one or more actions to be performed by the robot at one or more respective points along the desired trajectory; and wherein generating the trajectory image includes adding one or more markers to the line representing the one or more actions to be performed at the respective points along the desired trajectory.

8. The method of any one of claims 1-7, wherein obtaining the trajectory sketch further comprises obtaining, from the user, an indication of a region along the desired trajectory where motion of the part of the robot is constrained; and wherein generating the trajectory image includes depicting the region on the line representing the desired trajectory.

9. The method of any preceding claim, wherein processing the combination of the observation image and the trajectory image using the image encoder comprises concatenating the observation image and the trajectory image to generate a concatenated image, and processing the concatenated image to generate a representation of the concatenated image as the representation of the combination of the observation image and the trajectory image.

10. The method of claim 9 when dependent upon claim 2, further comprising: generating, the sequence of input tokens from at least the representation of the concatenated image by combining a sequence of image tokens for the concatenated image with respective sequences of image tokens for one or more earlier concatenated images from one or more earlier time steps.

11. The method of claim 10, wherein combining the sequence of image tokens for the concatenated image with respective initial sequences of image tokens for one or more earlier concatenated images from one or more earlier time steps comprises applying a position encoding to each input token in the sequence of input tokens for the concatenated image and in the respective sequences of input tokens for the one or more earlier concatenated images.DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application 12. The method of any one of claims 9-11, wherein the representation of the concatenated image comprises a feature map that includes a respective feature vector for each of a plurality of regions in the concatenated image, and wherein generating, from the representation of the concatenated image, a sequence of input tokens for the concatenated image comprises: generating an initial input sequence by flattening the feature map into a sequence of feature vectors.

13. The method of claim 12, wherein generating, from the representation of the concatenated image, a sequence of input tokens for the concatenated image comprises: processing the initial input sequence of feature vectors using a learned module that maps an input sequence to a reduced sequence that includes a smaller number of tokens.

14. The method of any one of claims 1-13, comprising obtaining the trajectory sketch from the user by: capturing an image of the environment; displaying the captured image on a graphical user interface; enabling the user to draw the trajectory sketch on the captured image to obtain the trajectory sketch.

15. The method of any one of claims 1-13, comprising obtaining the trajectory sketch from the user by: capturing a video of the user using their hand to simulating performing the task in the environment; processing the captured video to obtain a sequence of images each comprising a set of landmarks on the user’s hand; determining the trajectory sketch from the sets of landmarks in the sequence of images.

16. The method of claim 15, wherein the part of the robot is an end effector of the robot, the method further comprising: processing the set of landmarks in each image to extract landmarks on a finger and thumb of the hand; andDeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application processing the landmarks on the finger and thumb of the hand to identify one or more end effector actions on the desired trajectory; and adding a marker to the line representing the desired trajectory to indicate a location of each end effector action.

17. The method of any one of claims 1-13, comprising obtaining the trajectory sketch from the user by: capturing an image of the environment; obtaining a natural language input from the user comprising a natural language instruction to perform a task using an object; processing the captured image and the natural language instruction using a first neural network system to generate a set of coordinates identifying a location of the object or of the part of the robot; processing the set of coordinates and the natural language instruction using a second neural network system to generate coordinates for a set of waypoints along which to move the object or part of the robot to perform the task; determining the trajectory sketch from the set of waypoints.

18. The method of any one of claims 1-13, comprising obtaining the trajectory sketch from the user by: capturing an image of the environment; obtaining a natural language input from the user comprising a natural language instruction to perform a task using an object; processing the captured image and the natural language instruction using a visual language model neural network system to generate an image output comprising the trajectory sketch.

19. The method of any preceding claim, wherein the trained neural network system has been trained using an imitation learning method on a corpus of training data that comprises a plurality of demonstration tasks each comprising a sequence of observations and actions that move an object along a demonstration trajectory for the demonstration task.

20. The method of any preceding claim, wherein the environment is a real-world environment.DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application 21. The method of any preceding claim, wherein the image encoder comprises an image encoder neural network, the method further comprising: receiving a natural language text sequence that characterizes the task; and generating an encoded representation of the natural language text sequence; and wherein processing the observation image and the trajectory image using the image encoder neural network to generate the representation of the combination of the observation image and the trajectory image comprises: processing the observation image and the trajectory image using the image encoder neural network when conditioned on the encoded representation of the natural language text sequence, to generate the representation of the combination of the observation image and the trajectory image.

22. The method of claim 21, wherein the image encoder neural network comprises one or more conditioning layers that are each configured to receive a respective intermediate output of a respective intermediate layer of the image encoder neural network output and the encoded representation of the natural language instruction and to (i) update the respective intermediate output of the image encoder neural network using the encoded representation of the natural language instruction and (ii) provide the updated respective intermediate output as input to a respective subsequent intermediate layer of the image encoder neural network.

23. A computer-implemented method of training the neural network system of any one of claims 1-22, comprising: obtaining a corpus of training data that comprises a plurality of demonstration tasks each comprising a sequence of observation images and corresponding actions that move a part of the robot along a demonstration trajectory for the demonstration task; generating, for each of the demonstration tasks, a trajectory image representing the demonstration trajectory as a line in the trajectory image; for each demonstration task, and for each observation image and corresponding action in the sequence, processing the observation image and the trajectory image using the neural network system to generate the policy output for selecting an action to be performed by the robot; andDeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application training the neural network system to encourage the actions selected using the policy output to match the corresponding actions in the training data.

24. The method of claim 23, further comprising: obtaining from the robot, for each of the demonstration tasks, proprioceptive information corresponding to motion of the part of the robot along the demonstration trajectory; and generating the trajectory image from the proprioceptive information.

25. The method of claim 24, further comprising: capturing each of the observation images using an image sensor; and determining a 3D demonstration trajectory from the proprioceptive information; and projecting the 3D demonstration trajectory into an image space of the image sensor to obtain the trajectory image, wherein the trajectory image comprises a line depicting the demonstration trajectory in the image space of the image sensor.

26. The method of any one of claims 23-25, further comprising: generating augmented training data from the corpus of training data, or from using the neural network system to control a robot interacting with an environment to perform a user-directed task in the environment, or both; and training the neural network system on the augmented training data.

27. The method of claim 26, wherein generating additional training data from the corpus of training data comprises: determining a modified trajectory image for some of the observation images, wherein determining the modified trajectory image for an observation image comprises replacing the trajectory image with an image that depicts a limited part of the demonstration trajectory around a point in the demonstration trajectory that corresponds to the observation image.

28. One or more computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the operations of the respective method of any one of claims 1-27.DeepMind Technologies Limited F&R Ref.: 45288-0377WO1 PCT Application 29. A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the respective method of any one of claims 1-27.

Images