Real-World Robot Control Using Transformer Neural Networks

Abstract

A method, system, and apparatus, including a computer program encoded on a computer storage medium, for controlling an agent interacting with an environment. In one aspect, the method includes receiving a natural language text sequence characterizing a task to be performed by the agent in the environment, generating an encoded representation of the natural language text sequence, obtaining, at each of a plurality of time steps, an observation image characterizing a state of the environment at the time step, processing the observation image to generate the encoded representation of the observation image, generating a sequence of input tokens, processing the sequence of input tokens to generate a policy output defining an action to be performed by the agent in response to the observation image, using the policy output to select an action to be performed by the agent, and causing the agent to perform the selected action.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 432,373, filed December 13, 2022. The disclosure of the prior application is considered part of the disclosure of this application and is incorporated by reference into the disclosure of this application. [Background technology]

[0002] This specification relates to controlling agents using neural networks.

[0003] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the input of the current value of its respective parameter set. Summary of the Invention

[0004] This specification describes a system implemented as a computer program on one or more computers in one or more locations that controls agents, e.g., robots, interacting in an environment by selecting actions to be performed by the agents and causing the agents to perform the actions.

[0005] The subject matter described herein can be implemented in particular embodiments to achieve one or more of the following advantages: The policy system described herein is a neural network system that implements a multi-task neural network backbone that can control an agent, e.g., a robot, to perform any of a wide variety of real-world robot control tasks. By leveraging open-ended, task-independent training that extends beyond robot control, a high-capacity, data-efficient architecture that can learn knowledge present in large datasets, or both, the policy system can solve specific downstream robot control tasks with high levels of performance in zero-shot or with relatively small, task-specific datasets.

[0006] Additionally, the policy system can control the agent with reduced latency, e.g., generate policy outputs specifying actions to be performed by the agent in response to observed images at a frequency of 3 Hz or greater. This reduced latency ensures that the policy system can control the agent to move in a more natural and fluid manner, resulting in greater precision in agent movement, which can be key to successful task accomplishment.

[0007] Furthermore, once trained, the policy system can control a robot in new tasks (including new environments, new objects, or both) not previously seen during the training of the policy system. This generalization ability of the policy system is advantageous for extending the applicability of the described policy system to complex robot control tasks, e.g., dexterous tasks, long-term tasks, etc., for which obtaining demonstration data is difficult or computationally expensive.

[0008] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0009] [Figure 1] 1 illustrates an exemplary policy system and an exemplary control system. [Figure 2] FIG. 1 is an architectural diagram of an exemplary policy system. [Figure 3] FIG. 2 is a diagram of operations performed by an exemplary policy system. [Figure 4] FIG. 1 is a flow diagram of an exemplary process for controlling an agent interacting with an environment. [Figure 5] FIG. 1 is a flow diagram of an exemplary process for training a set of neural networks included in a policy system. [Figure 6] FIG. 1 is a diagram of training a set of neural networks on a mixture training dataset. [Figure 7] 10 shows a quantitative example of the performance improvement that can be achieved by using the policy system described herein. DETAILED DESCRIPTION OF THE INVENTION

[0010] Like reference symbols and designations in the various drawings indicate like elements.

[0011] 1 illustrates an example policy system 100 and an example control system 101. Policy system 100 and control system 101 are examples of systems implemented as computer programs on one or more computers at one or more locations in which the systems, components, and techniques described below are implemented.

[0012] The policy system 100 and the control system 101 may control an agent 102, e.g., a robot, to accomplish any of a wide variety of tasks in an environment 104. To control an agent 102 interacting within the environment 104 to accomplish a task, the policy system 100 selects an action 144 to be performed by the agent 102, and the control system 101 then causes the agent 102 to perform the selected action 144.

[0013] As general examples, tasks may include, for example, one or more of: navigating to a specified location within environment 104, identifying a particular object within environment 104, manipulating a particular object in a specified manner, controlling an item of equipment to meet a criterion, etc. To accomplish such tasks, agent 102 moves within environment 104, e.g., navigates and / or changes its configuration.

[0014] Typically, the control system 101 is local to the agent 102. For example, the control system 101 can be on-board the agent 102, and can be implemented on one or more computers, local workstations, or local servers having relatively little processing and memory resources, for example, on-board the agent 102.

[0015] In some embodiments, the policy system 100 is local to the agent 102. For example, like the control system 101, the policy system 100 can also be on-board the agent 102. Furthermore, in some of these embodiments, the policy system 100 may be part of the control system 101 that causes the agent 102 to perform the action 144.

[0016] In other embodiments, the policy system 100 is remote from the agent 102. For example, unlike the control system 101, the policy system 100 can be hosted in a data center, which can be a distributed computing system having hundreds or thousands of computers in one or more locations. That is, the control system 101 can receive data identifying actions 144 from an external source, rather than generating such data on-board the agent 102, for example.

[0017] In these embodiments, the policy system 100 and the control system 101 may be connected by a data communications network, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof.

[0018] In these embodiments, the control system 101 of the agent 102 interacts with a remote policy system 100 hosted in a data center that has significantly more computing and other resources than are available on-board the agent 102 in order to reduce latency in selecting an action 144, reduce consumption of the agent's 102's limited power supply in selecting an action 144, or both.

[0019] In some implementations, the policy system 100, the control system 101, or both may expose one or more application programming interfaces (APIs) or other data interfaces that facilitate control of the agent 102. For example, a user of the agent 102 may use an API made available by the action selection system 100 to provide a natural language text sequence 108 that characterizes a task to be performed by the agent. As another example, the policy system 100 and the control system 101 may interact via an API between the two systems; for example, the control system 101 may use the API to provide the observed image 106 to the policy system 100, and the policy system 100 may use the API to provide the control system 101 with data specifying the selected action 144.

[0020] In particular, the policy system 100 and control system 101 control the agent 102 based on policy outputs generated by a set of neural networks configured through training to control the agent 102 in response to observed images 106 characterizing the environment 104 and natural language text sequences 108 characterizing the task to be performed by the agent 102.

[0021] For example, the observed image 106 may be an image captured by a camera sensor on the agent 102 or by a camera sensor located in the environment 104. The camera sensor may be, for example, a still camera or a video camera.

[0022] The natural language text sequence 108 may be received from another agent in the environment 104 or from the control system 101 of the agent 102. For example, another agent in the environment 104 can speak an instruction, which the control system 101 or other system can transcribe into a natural language text sequence 108 and then provide the transcription to the policy system 100. As another example, the control system 101 may receive an instruction entered by a user specifying the natural language text sequence 108, e.g., a text-based input, a selection-based input, or an audio-based input, and then provide the instruction to the policy system 100.

[0023] More specifically, the policy system 100 receives a natural language text sequence 108 that characterizes a task to be performed by an agent 102 within an environment 104, and processes the natural language text sequence 108 using a text encoder neural network 110 to generate an encoded representation 112 of the natural language text sequence (or "encoded text" for short).

[0024] The encoded representation 112 may be or include an embedding of the text sequence 108. As used herein, an "embedding" is a sequence of vectors of one or more numerical values, e.g., floating-point or other values, where each vector has a pre-determined number of dimensions.

[0025] The text encoder neural network 110 may have any suitable neural network architecture that enables the neural network to map natural language text sequences 108 to embeddings of the text sequences 108. Particular exemplary architectures are described further below, but more generally, the text encoder neural network 110 may include any suitable number (e.g., 2, or 5, or 10) of any suitable type of neural network layers (e.g., embedding layers, fully connected layers, etc.), connected in any suitable configuration (e.g., as a directed graph of layers).

[0026] At each of a plurality of time steps, the policy system 100 acquires an observation image 106 that characterizes the state of the environment 104 at the time step. The observation image 106 may be acquired from a camera sensor (e.g., a camera sensor of the agent 102 or a camera sensor located in the environment 104) or from the control system 101 of the agent 102. For example, the control system 101 of the agent 102 acquires the observation image 106 of the environment 104 at the time step from the camera sensor and then provides the observation image 106 to the policy system 100.

[0027] In embodiments in which the policy system 100 is remote from the agent 102, providing the observed image 106 to the policy system 100 may include, for example, transmitting data representing the observed image 106 over a data communications network connecting the policy system 100 and the control system 101. As another example, providing the observed image 106 to the policy system 100 may include providing data to the policy system 100 specifying the name or network location (e.g., a uniform resource locator (URL)) of a server from which the policy system 100 can obtain the observed image 106.

[0028] The policy system 100 processes inputs including an observed image 106 and, optionally, an encoded representation 112 of a natural language text sequence using an image encoder neural network 120 to generate an encoded representation 122 of the observed image (or "encoded image" for short).

[0029] Although observations are generally described herein as being images, in some cases, the observations may include additional data in addition to images, such as proprioceptive data characterizing the agent or other data captured by other sensors of the agent. In these cases, the other data may be jointly encoded with the observed image 106 by the image encoder neural network 120.

[0030] If the coded representation 112 of the natural language text sequence is also provided by the policy system 100 as part of the input to the image encoder neural network 120, the image encoder neural network 120 generates a coded representation 122 of the observed image conditioned by the coded representation 112 of the natural language text sequence. That is, the image encoder neural network 120 uses the coded representation 112 of the natural language text sequence as a context when generating the coded representation 122 of the observed image; i.e., different text sequences may result in different representations being generated for the same observed image.

[0031] Specific exemplary architectures are described further below, but more generally, image encoder neural network 120 may include any suitable number (e.g., 5, or 10, or 50) of any suitable type of neural network layers (e.g., convolutional layers, conditioning layers, attention layers, etc.), connected in any suitable configuration (e.g., as a directed graph of layers).

[0032] The policy system 100 generates a sequence of input tokens 132 based on the coded representation 122 of the observed image. The sequence of input tokens 132 may include a sequence of image tokens for the observed image 106. As used herein, a "token" is a vector or other ordered set of numeric values ​​that has a fixed number of dimensions, i.e., the number of values ​​in the ordered set is constant across different tokens.

[0033] The encoded representation 122 may include a feature map that includes a respective feature vector for each of a plurality of regions in the observed image 106. The policy system 100 may thus generate the sequence of input tokens 132 by using the feature vectors included in the feature map, or data derived from these feature vectors, as the image tokens included in the sequence of input tokens 132.

[0034] In some implementations, the policy system 100 generates the sequence of input tokens 132 by applying a learned module to map feature vectors included in the encoded representation 122 of the observed image to a smaller number of feature tokens, and then using the feature vectors generated as a result of applying the learned module as image tokens included in the sequence of input tokens 132.

[0035] The trained module may be, but need not be, a neural network. In the example of FIG. 1, the trained module is implemented using a neural network (“token neural network 130”). When configured as a neural network, the trained module may include any suitable number (e.g., 1 layer, or 5 layers, or 10 layers) of any suitable type of neural network layers (e.g., convolutional layers, fully connected layers, attention layers, pooling layers, etc.) and may be connected in any suitable configuration (e.g., as a directed graph of layers).

[0036] Specific exemplary architectures of token neural network 130 are described further below. As another example, token neural network 130 may have a Vision Transformer (ViT) architecture that includes one or more attention layers. As another example, token neural network 130 may have a convolutional neural network architecture that includes one or more convolutional layers.

[0037] In yet another example, the learned module may include any data values ​​that define a learned mapping that reduces the number of feature vectors provided as input to the learned module, i.e., that maps a larger number of feature vectors to a smaller number of feature vectors. By "learned," we mean that these data values ​​are adjusted during the joint training of a set of neural networks included in policy system 100.

[0038] After generating the sequence of input tokens 132, the policy system 100 then processes the sequence of input tokens 132 using a Transformer neural network 140 to generate a policy output 142 that defines an action to be taken by the agent 102 in response to the observed image 106 received at the time step. The Transformer neural network 140 receives the sequence of input tokens 132 and generates, for example, in an autoregressive manner, the policy output 142, which is comprised of a plurality of data values.

[0039] Specific exemplary architectures are described further below, but more generally, the Transformer neural network 140 may be any suitable Transformer-based architecture, such as, for example, Ashish Vaswani, et al., “Attention is all you need,” Advances in neural information processing systems, 30, 2017; Peter J. Liu, et al., “Generating Wikipedia by summarizing long sequence,” arXiv preprint arXiv:1801.10198 (2018); Daniel Adiwardana, et al., “Towards a human-like open-domain chatbot,” CoRR, abs / 2001.09977, 2020; Tom B. Brown, et al., “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165, 2020; and Aakanksha Chowdhery, et al., “Palm: Scaling language modeling with pathways,” arXiv preprint It can have one of the architectures described in arXiv:2204.02311(2022).

[0040] The policy system 100 uses the policy outputs 142 to select an action 144 to be performed by the agent 102. Examples of policy outputs 142 and how the policy system 100 uses such policy outputs 142 to make this action selection are further described below in FIG.

[0041] After selecting an action 144 to be performed by agent 102 at a time step, policy system 100 provides data identifying the selected action 144 to control system 101. In embodiments where policy system 100 is remote from agent 102, providing the data identifying the selected action 144 may include, for example, transmitting the data identifying the selected action 144 over a data communications network connecting policy system 100 and control system 101.

[0042] The control system 101 then causes the agent 102 to perform the selected action 144. For example, the control system 101 may do this by generating instructions for the agent 102 that, when executed, cause the agent 102 to perform the selected action 144, by sending control inputs directly to appropriate controls of the agent, or by using other suitable control techniques.

[0043] In some implementations, the environment 104 is a real-world environment and the agent 102 is a mechanical agent that interacts with the real-world environment. For example, the agent may be a robot that interacts with the environment to achieve a goal, such as to position an object of interest within the environment, to move an object of interest to a specified location within the environment, to physically manipulate an object of interest within the environment in a specified manner, or to navigate to a specified destination within the environment, or the agent may be an autonomous or semi-autonomous land, air, or water vehicle that navigates through the environment to a specified destination within the environment.

[0044] Actions 144 may be control inputs for controlling a robot, such as torques or higher-level control commands for the joints of the robot, or control inputs for controlling an autonomous or semi-autonomous land, air, or water vehicle, such as torques or higher-level control commands for the control surfaces or other control elements of the vehicle.

[0045] In other words, actions 144 may include, for example, position, velocity, or force / torque / acceleration data for one or more joints of a robot or other mechanical agent part. Actions may additionally or alternatively include electronic control data, such as motor control data, or more generally, data for controlling one or more electronic devices in the environment, the control of which affects the observed state of the environment. For example, in the case of an autonomous or semi-autonomous land, air, or water vehicle, actions may include actions that control navigation, e.g., steering, and actions that control movement, e.g., braking and / or accelerating the vehicle.

[0046] In some implementations, the environment 104 is a simulated environment and the agent 102 is implemented as one or more computer programs that interact with the simulated environment. For example, the environment can be a computer simulation of a real-world environment and the agent can be a simulated machine agent that navigates through the computer simulation.

[0047] For example, the simulated environment may be a mobile simulation environment, such as a driving or flight simulation, and the agent may be a simulated vehicle that navigates through the mobile simulation. In these implementations, the actions 144 may be control inputs for controlling a simulated user or a simulated vehicle. As another example, the simulated environment may be a computer simulation of a real-world environment, and the agent may be a simulated robot that interacts with the computer simulation.

[0048] In general, when environment 104 is a simulated environment, actions 144 may include simulated versions of one or more of the actions or types of actions previously described.

[0049] In some implementations, environment 104 is a suitable execution environment, e.g., a runtime environment or operating system environment, implemented on one or more computing devices, such as a smartphone, a tablet computer, a wearable device, an automotive system, a standalone personal assistant device, etc., and agent 102 is a virtual agent (also known as an “automated assistant” or “mobile assistant”) that can be interacted with by a user through the computing device. The virtual agent can receive input (e.g., typed or spoken natural language input) from the user and respond with responsive content (e.g., visual and / or audible natural language output). The virtual agent can provide a wide range of functionality through interactions with various local and / or third-party applications, websites, or other agents. In these implementations, action 144 can include any activity or operation that can be performed or initiated by a user on the computing device, for example, within application software installed on the computing device.

[0050] In some cases, policy system 100 can be used to control an agent's interactions with a simulated environment, and policy system 100 (or another training system) can train a set of neural networks used to control agent 102 based on the agent's (or another agent's) interactions with the simulated environment to determine trained values ​​for the parameters of the set of neural networks. Training a set of neural networks is described in more detail below with reference to FIGS. 5-6.

[0051] After the set of neural networks has been trained based on the interactions of agent 102 (or other agents) with the simulated environment, the trained neural networks can be used by policy system 100 to control the interactions of real-world agents with the real-world environment, i.e., to control the agents that were simulated in the simulated environment.

[0052] Training a neural network based on the agent's interactions with a simulated environment (i.e., instead of a real-world environment) can avoid wear on the agent and reduce the possibility that the agent may damage itself or aspects of its environment by performing poorly chosen actions.

[0053] FIG. 2 is a diagram of the architecture of an example policy system 200 .

[0054] The policy system 200 receives a natural language text sequence 208. The natural language text sequence 208 characterizes a task to be performed by an agent in an environment. The natural language text sequence 208 may have an instruction format. For example, Figure 2 shows that the natural language text sequence 208 is a natural language instruction describing the task "take an apple from the top drawer and place it on the counter."

[0055] The policy system 200 uses a text encoder neural network 210 to process a natural language text sequence 208 and generate an encoded representation 212 of the natural language text sequence.

[0056] In the example of Figure 2, the text encoder neural network 210 has a Universal Sentence Encoder architecture and generates a 512-dimensional embedding, i.e., a vector containing 512 entries, each of which is a numeric value, e.g., a floating-point value. The Universal Sentence Encoder is described in more detail in Daniel Cer, et al., "Universal sentence encoder." arXiv preprint arXiv:1803.11175, 2018.

[0057] In other examples, the text encoder neural network 210 may have a different architecture and may generate embeddings with smaller or larger dimensions. Additionally, in other examples, the text encoder neural network 210 may generate embeddings that include a sequence of multiple embedding vectors.

[0058] At each of a plurality of time steps, policy system 200 acquires an observed image 206 that characterizes the state of the environment at the time step. In the example of Figure 2, the agent performs a single action in response to each observed image 206, e.g., a new observed image is acquired by policy system 200 after each action performed by the agent.

[0059] The policy system 100 uses an image encoder neural network 220 to generate an encoded representation 222 of the observed image 206. The encoded representation 222 may include a feature map that includes a respective feature vector for each of a plurality of regions in the observed image 206.

[0060] For example, FIG. 2 shows that the image encoder neural network 220 generates a feature map that includes 81 feature vectors, each having 512 dimensions and corresponding to 81 regions (e.g., 81 non-overlapping pixel subsets) arranged along the horizontal and vertical dimensions in the observed image 206.

[0061] The image encoder neural network 220 may generally be configured as a convolutional neural network including one or more convolution layers. As a specific example of this, FIG. 2 shows that the image encoder neural network 220 includes a stack of 26 inverted residual blocks (“MBConv blocks”). The inverted residual blocks are described in more detail in Mingxing Tan, et al., “EfficientNet: Rethinking model scaling for convolutional neural networks.” In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6105-6114. PMLR, 09-15 Jun 2019. URL: https: / / proceedings.mlr.press / v97 / tan19a.html.

[0062] 2, image encoder neural network 220 generates coded representation 222 of the observed image conditioned on coded representation 212 of the natural language text sequence. That is, image encoder neural network 220 receives coded representation 212 of the natural language text sequence and observed image 206 as inputs, and processes the inputs to generate coded representation 222 of the observed image as output.

[0063] The image encoder neural network 220 uses the coded representation 212 of the natural language text sequence as context when generating the coded representation 222 of the observed image; i.e., different text sequences may result in different representations being generated for the same observed image.

[0064] To that end, the image encoder neural network 220 also includes one or more conditioning layers, which may be inserted between other intermediate layers of the image encoder neural network 220, such as convolutional layers, e.g., depthwise convolutional layers.

[0065] Each conditioning layer receives as input (i) a respective intermediate output of a respective hidden layer of the image encoder neural network and (ii) an encoded representation 212 of a natural language text sequence, processes the inputs such that (i) it updates a respective intermediate output of the image encoder neural network using the encoded representation 212 of the natural language instructions, and (ii) it provides the respective updated intermediate output as an input to a respective subsequent hidden layer of the image encoder neural network.

[0066] As a specific example of this, Figure 2 shows that an image encoder neural network 220 includes 26 feature-wise linear modulation (FiLM) layers inserted between stacks of 26 inverted residual blocks. The FiLM layers learn functions f and h and output γ and β as functions of the input x. γ=f(x);β=h(x) Here, γ and β modulate the respective intermediate outputs F of the respective hidden layers through an affine transformation for each feature. FiLM(F|γ,β)=γF+β

[0067] The functions f and h may, but need not, be implemented as neural networks.

[0068] The FiLM layer is described in more detail in Ethan Perez, et al. Film: Visual reasoning with a general conditioning layer. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), April 2018. doi:10.1609 / aaai.v32i1.11671.

[0069] For example, as shown, the image encoder neural network 220 includes a FiLM layer positioned between the first inverted residual block and the second inverted residual block in the stack. The FiLM layer receives as input (i) the respective intermediate outputs of the first inverted residual block and (ii) the encoded representation of the natural language text sequence 212, processes the inputs such that (i) it updates the respective intermediate outputs of the first inverted residual block using the encoded representation of the natural language instruction 212, and (ii) it provides the respective updated intermediate outputs as inputs to the second inverted residual block. Thus, instead of receiving as inputs the respective intermediate outputs of the first inverted residual block, the second inverted residual block receives the respective updated intermediate outputs updated by the FiLM layer using the encoded representation of the natural language text sequence 212.

[0070] The policy system 200 generates a sequence of input tokens 232 based on the coded representation 222 of the observed image. The sequence of input tokens 232 may include a sequence of image tokens.

[0071] As mentioned above, the encoded representation 222 of the observed image may include a feature map that includes a respective feature vector for each of a number of regions in the observed image 206 .

[0072] 2, policy system 200 generates an initial input sequence by flattening the feature map into a sequence of feature vectors and then processing the initial input sequence of feature vectors using token neural network 230 to map the initial input sequence to a reduced input sequence containing fewer feature tokens. The feature tokens contained in the reduced input sequence are then used by policy system 200 as image tokens contained in sequence of input tokens 232.

[0073] As a specific example of this, Figure 2 shows that the token neural network 230 has a TokenLearner architecture and processes an initial input sequence of 81 feature vectors (generated based on flattening a feature map containing 81 feature vectors) based on applying a spatial attention mechanism to generate a reduced input sequence containing 8 feature tokens, each with 512 dimensions, for observed image 206 acquired at time step 206. TokenLearner is described in more detail in Michael Ryoo, et al., Tokenlearner: Adaptive space-time tokenization for videos. Advances in Neural Information Processing Systems, 34:12786-12797, 2021.

[0074] In some embodiments, the sequence of input tokens 232 includes only image tokens of the observed image 206 acquired at the current time step, i.e., only feature tokens included in the reduced input sequence generated for the observed image 206 acquired at the current time step.

[0075] In another embodiment, the sequence of input tokens 232 includes not only image tokens of the observed image 206 acquired at the current time step, but also image tokens of one or more previous observed images acquired at one or more previous (or earlier) time steps, i.e., feature tokens included in the reduced input sequence generated for one or more previous observed images acquired at one or more previous time steps.

[0076] In these implementations, to speed up inference by avoiding redundant computation, policy system 200 may store feature tokens contained in the reduced input sequence generated at each time step and reuse them at later time steps.

[0077] 2, the sequence of input tokens 232 includes a sequence of image tokens that is a combination, e.g., a concatenation, of (i) feature tokens included in the reduced input sequence generated for the observed image 206 acquired at the current time step, and (ii) feature tokens included in the reduced input sequences generated for five previous observed images acquired at five previous time steps preceding the current time step. Specifically, FIG. 2 shows that the sequence of input tokens 232 includes a concatenation of 48 feature tokens.

[0078] 2 further illustrates that policy system 200 employs a positional encoding scheme. Specifically, policy system 200 adds a respective positional encoding 233 to (i) each image token in the sequence of image tokens of the observed image acquired at the current time step and (ii) each image token in the sequence of image tokens of one or more previous observed images acquired at one or more previous time steps.

[0079] The positional coding 233 may be determined for each image token to uniquely identify the respective time step among the multiple time steps at which the image token is generated, for example according to a sinusoidal positional coding scheme or other coding scheme.

[0080] Policy system 200 then provides sequence of input tokens 232 as input to Transformer neural network 240. As a specific example, Figure 2 shows that Transformer neural network 240 has a decoder-only Transformer neural network architecture including eight self-attention layers. In other examples, Transformer neural network 240 can have a different Transformer-based architecture, e.g., an encoder-decoder Transformer neural network architecture, including more or fewer layers, each with the same or different attention mechanisms.

[0081] In some embodiments, each possible action that can be performed by an agent is defined by a respective value for each of a plurality of action dimensions. In these embodiments, for each of the plurality of action dimensions, policy output 242 may define a respective distribution over the possible values ​​of the action dimension.

[0082] In the example of FIG. 2, the agent is a robot having a base and one or more arms, at least one of which has an end effector (e.g., a gripper or other tool) attached to its end. In this example, the multiple action dimensions include seven action dimensions for arm movement: x, y, z, roll, pitch, yaw, and end effector status (e.g., gripper open / closed status). The multiple action dimensions also include three action dimensions for base movement: x, y, and yaw. The multiple action dimensions further include an action dimension for mode switching (e.g., for switching between controlling the robot's arm, controlling the robot's base, or ending the episode).

[0083] In other examples, the agent may be a different type of robot, or as described above, a vehicle or other type of agent, and therefore each possible action that can be performed by the agent may be characterized by a different set of action dimensions.

[0084] In any example, the possible values ​​of the action dimension may be discretized into a fixed number of bins, and policy output 242 may include data values ​​that define a distribution over the fixed number of bins of the action dimension. The distribution may be a categorical distribution (respective discrete probability distribution) that assigns a respective probability score to each bin in the fixed number of bins of the action dimension.

[0085] In some of these embodiments, for each action dimension, a fixed number of bins may correspond to an approximately equal number of possible values ​​of the action dimension. For example, the possible values ​​of a roll (or, analogously, pitch or yaw) action dimension have a range of 0 to 360 degrees and are divided into 32 bins (each bin corresponding to a range spanning 11.25 degrees), 128 bins (each bin corresponding to a range spanning approximately 2.81 degrees), 256 bins (each bin corresponding to a range spanning approximately 1.41 degrees), etc. As another example, the possible values ​​of a mode-switch action dimension are 0 (control the robot's arm), 1 (control the robot's base), and 2 (end the episode) and are divided into 3 bins (each bin corresponding to a respective value), 256 bins (approximately 85 bins each corresponding to the same respective value), etc.

[0086] In some other implementations, policy output 242 may have a different format that defines or otherwise specifies an action. For example, policy output 242 may be a natural language description of an action to be performed by the agent, such as "move arm to position (x, y, z)," "move arm to pose (x, y, z, roll, pitch, yaw)," or "open gripper." As another example, policy output 242 may be a sequence of data elements that represent an action to be performed by the agent, such as an identifier of the action.

[0087] To select an action to be performed by the agent at a time step, policy system 200 then, for each of one or more action dimensions, selects a respective value among the possible values ​​of the action dimension using the respective distribution. For example, policy system 200 can greedily select the highest-scoring bin, or can sample a bin from each distribution defined by policy output 242 for the action dimension, e.g., using nucleus sampling or other sampling techniques, and then select the value that corresponds to, e.g., falls within, the selected bin as the selected value of the action dimension.

[0088] Figure 3 is a diagram of operations performed by an example policy system 300. Policy system 300 may be the same as or similar to policy system 200 of Figure 2. Policy system 300 may control an agent to perform a task in an environment by repeatedly performing one iteration of those operations at each of multiple time steps to select actions to be performed by the agent in response to obtaining captured observational images in or of the environment at a time step.

[0089] At a high level, at each of a plurality of time steps, the operations include receiving as input data derived from a sequence of observed images and a sequence of natural language text, and processing the data to generate a policy output that defines an action to be taken by an agent at the time step.

[0090] In particular, when configured with the architecture illustrated in Figure 2, the policy system 300 can repeatedly perform these operations at a frequency of 3 Hz or higher, i.e., the policy system can select three or more actions to be performed by the agent per second (in response to obtaining three or more observation images), which is comparable to the control frequency that would be achieved, for example, by a human skilled in the task being performed by the agent.

[0091] As shown, the policy system 300 receives a natural language text sequence 308 and processes the natural language text sequence 308 using a text encoder neural network to generate an encoded representation of the natural language text sequence 308. The natural language text sequence 308 may have an instruction format that defines tasks to be performed by agents in an environment.

[0092] At each of a number of time steps, the policy system 300 obtains an observed image 306 that characterizes the state of the environment at the time step, and then provides the observed image 306 to a set of neural networks included in the policy system 300 for further processing to select an action to be performed by the agent in response to the observed image 306.

[0093] The set of neural networks includes an image encoder neural network 320. The image encoder neural network 320 processes both the observed image 306 and the encoded representation of the natural language text sequence to generate an encoded representation of the observed image. The encoded representation includes a feature map containing a respective feature vector for each of a plurality of regions in the observed image 306.

[0094] The set of neural networks also includes a token neural network 330. The policy system 300 processes the initial input sequence of feature vectors using the token neural network 330 to flatten the feature map into a sequence of feature vectors and generate a reduced input sequence containing a smaller number of feature tokens.

[0095] The policy system 300 generates a sequence of input tokens from feature tokens generated for a current observed image 306 acquired at a current time step and, optionally, feature tokens generated for one or more previous observed images acquired at one or more previous time steps.

[0096] The set of neural networks further includes a Transformer neural network 340. The Transformer neural network 340 processes the sequence of input tokens to generate a policy output 342 that defines the actions to be taken by the agent in response to the observed image 306.

[0097] The policy system 300 uses the policy output to select an action to be performed by the agent, and then causes the agent to perform the selected action.

[0098] The policy system 300 may repeat these operations at a frequency of 3 Hz or greater until some termination condition is met, for example, until it generates a particular policy output 342 that defines a termination action, until a predetermined number of iterations of these operations have been performed, or until a predetermined length of time has elapsed.

[0099] 4 is a flow diagram of an exemplary process 400 for controlling an agent interacting with an environment. For convenience, process 400 is described as being performed by one or more computer systems located at one or more locations. For example, a policy system, such as policy system 100 of FIG. 1 , suitably programmed in accordance with this specification, may perform process 400.

[0100] The system controls the agent to perform tasks in the environment by repeatedly performing iterations of process 400 at each of a number of time steps (hereafter referred to as the "current" time step).

[0101] The tasks performed by the agents are characterized by natural language text sequences. For example, prior to a first iteration of process 400, the system receives the natural language text sequences that characterize the tasks performed by the agents in the environment and generates an encoded representation of the natural language text sequences. For example, the encoded representation may include an embedding of the natural language text sequences, and the system may generate the embedding by processing the natural language text sequences using a text encoder neural network.

[0102] The system acquires an observational image that characterizes the state of the environment at the current time step, step 402. For example, the observational image can be captured by a camera sensor on the agent or by a camera sensor located in the environment.

[0103] The system processes the observed image using an image encoder neural network conditioned on the coded representation of the natural language text sequence to generate a coded representation of the observed image (step 404). That is, the image encoder neural network receives as input the coded representation of the natural language text sequence and the observed image, and processes the input to generate as output the coded representation of the observed image.

[0104] The system generates a sequence of input tokens from at least the coded representation of the observed image (step 406). The sequence of input tokens may include a sequence of image tokens for a current observed image acquired at a current time step. Optionally, the sequence of input tokens may also include a sequence of image tokens for each of one or more previous observed images acquired at one or more previous time steps.

[0105] For example, the encoded representation may include a feature map including a respective feature vector for each of a plurality of regions in the observed image. In this example, the system may generate the initial input sequence by flattening the feature map into a sequence of feature vectors and then processing the initial input sequence of feature vectors using a trained module that maps the initial input sequence to a reduced input sequence including a fewer number of feature tokens. The feature tokens included in the reduced input sequence may then be used as the image tokens included in the sequence of input tokens.

[0106] The system processes the sequence of input tokens using a Transformer neural network to generate policy outputs that define actions to be taken by the agent in response to the observed image acquired at the current time step (step 408). Each possible action that may be taken by the agent is defined by a respective value for each of a plurality of action dimensions. To define the actions to be taken by the agent, the policy outputs generated by the Transformer neural network include, for each of a plurality of action dimensions, a respective categorical distribution over the possible values ​​of the action dimension.

[0107] The system uses the policy output to select an action to be performed by the agent (step 410). This selection may be made by selecting respective values ​​for one or more of a plurality of action dimensions using respective categorical distributions defined by the policy output of the Transformer neural network.

[0108] The system causes the agent to perform the selected action (step 412), for example, by sending a control input directly to the agent, or by sending, for example, via a data communications network, instructions or other data to the agent's control system that causes the agent to perform the selected action.

[0109] Process 400 may be performed when controlling an agent to perform a task where the action to be taken is unknown, e.g., the action that will result in progress toward completing the task. Process 400 may also be performed as part of selecting an action to be performed by an agent based on processing observed images derived from a set of training data sets, i.e., observed images for which the action to be performed by the agent in response is known, to train a set of neural networks to determine trained values ​​for the parameters of the neural networks.

[0110] 5 is a flow diagram of an exemplary process 500 for training a set of neural networks included in a policy system. For convenience, process 500 is described as being performed by one or more computer systems located at one or more locations. For example, a policy system, such as policy system 100 of FIG. 1, or other training system suitably programmed in accordance with this specification, can perform process 500.

[0111] To train the set of neural networks, the system obtains a set of training data (step 502). The set of training data may include training data generated based on interactions of the agent (or other agents) with the environment.

[0112] In one example, the training dataset is N training examples

number

[0113] For example, the training data set may include training examples collected when the agent 102 of FIG. 1, or other agent, performs one or more of the following tasks listed in Table 1 below: [Table 1]

[0114] In some cases, the training dataset includes expert interaction data characterizing interactions of one or more expert agents with a corresponding environment. An expert agent may be any agent that selects actions in response to observed images according to an action selection policy that enables the expert agent to effectively progress toward accomplishing a task. For example, an expert agent may be an agent controlled by another already trained policy system, a human being skilled in the task performed by the agent, etc.

[0115] In some of these cases, the expert interaction data includes simulation data in which a simulated expert agent performs one or more tasks in a simulated environment. In other of these cases, the expert interaction data includes real-world data in which a real-world expert agent performs one or more tasks in a real-world environment. In still other cases, the expert interaction data includes both simulation data and real-world data.

[0116] In some cases, the training dataset includes training examples generated from one or more robots of the same model (and therefore having identical physical characteristics), e.g., when they were performing the same or different tasks, e.g., one of the tasks mentioned in Table 1 above, or other tasks.

[0117] In other cases, the training dataset may be a mixed training dataset that includes training examples generated from multiple robots that are not the same model, located at the same site, or manufactured by the same manufacturer. For example, the mixed training data can be generated from tens or hundreds of different robots that have different physical characteristics and are different models. In addition, the mixed training data need not be generated from a physical robot. For example, the mixed training data can include data generated from a simulation of a physical robot.

[0118] 6 is a diagram illustrating training a set of neural networks on a mixed training dataset. FIG. 6 shows a system obtaining a first set of training examples generated from a first robot performing a first task and a second set of training examples generated from a second robot performing a second task. The first and second robots have different physical characteristics and are manufactured by different manufacturers. The set of neural networks can then be trained to select actions to be performed by the first robot to perform both the first task and the second task.

[0119] In general, the more diverse the training dataset, the better a system, when trained on the training dataset, can generalize to unseen robot control tasks, e.g., tasks characterized by unseen commands, tasks that involve selecting actions in response to observational images collected in or about unseen environments, tasks that involve unseen objects, etc. Increasing the diversity of the training dataset may also improve the robustness of the system to possible disturbances that may arise in tasks previously seen during training, e.g., new obstacles, new background scenes, etc.

[0120] The system trains a set of neural networks on the set of training data (step 504). The set of neural networks may include the text encoder neural network 110, the image encoder neural network 120, the token neural network 130, and the Transformer neural network 140 of FIG.

[0121] To train the set of neural networks, the system selects training examples from the training dataset and, for each training example selected from the training dataset, generates a training policy output that defines actions to be performed by the agent based on processing the natural language text sequences and observed images contained in the training example. For example, step 502 may include performing multiple iterations of process 400.

[0122] The system updates the values ​​of the parameters of the neural network based on machine learning training techniques, e.g., using gradient descent training techniques that use backpropagation using a suitable optimizer, e.g., stochastic gradient descent, RMSprop, or Adam optimizer, to optimize the objective function.

[0123] For example, the objective function may be a cross-entropy objective function or other objective function that evaluates, for each training example selected from the training dataset, the difference between (i) the training policy outputs generated by the set of neural networks from processing the natural language text sequences and observed images included in the training examples, and (ii) the ground truth policy outputs that define the actions included in the training examples.

[0124] During training, the system may incorporate any number of techniques to improve the speed, effectiveness, or both of the training process.

[0125] For example, rather than training each of a set of neural networks from scratch, e.g., from initial parameter values, the system can start with pre-trained parameter values ​​for some of the neural networks and train the set of neural networks.

[0126] In some cases, a text encoder neural network may be pre-trained on a text processing task, e.g., a text representation learning task, prior to the joint training of a set of neural networks, e.g., as part of a larger text processing neural network. In some of these cases, the text encoder neural network is fine-tuned during the joint training; in others of these cases, the text encoder neural network remains frozen during the joint training, i.e., the joint training of the neural networks on the training dataset does not adjust the pre-trained parameter values ​​of the pre-trained text encoder neural network.

[0127] In some cases, an image encoder neural network may be pre-trained on an image processing task, e.g., an image classification or segmentation task, e.g., as part of a larger image processing neural network, and then fine-tuned during joint training of a set of neural networks on a training dataset.

[0128] In some of these cases, the image encoder neural network does not include one or more conditioning layers for pre-training. For example, the image encoder neural network can be trained as part of a neural network that is trained to classify images into a set of categories without conditioning on any natural language text sequences as context.

[0129] Furthermore, in some of these cases, inserting a conditioning layer as a new layer into such a pre-trained image encoder neural network may cause confusion in the intermediate outputs of the neural network, so before joint training, the system initializes each conditioning layer to act as an identity transformation for its corresponding respective intermediate output. This can be done, for example, by setting at least some of the parameter values ​​associated with each conditioning layer to zero.

[0130] As another example, the joint training of a set of neural networks may include imitation learning. For example, if the training data set includes expert interaction data, the system can train a set of neural networks through behavioral cloning on the expert interaction data to generate training policy outputs from which actions can be selected that closely mimic those performed by the expert agent.

[0131] Figure 7 shows a quantitative example of the performance gains that can be achieved by using the policy system described herein. Specifically, Figure 7 shows the overall performance (in terms of success rate) across seen tasks, generalization ability to unseen tasks, and robustness to distractors and background for an agent controlled using policy 100 ("RT-1") of Figure 1 and an agent controlled using a baseline system.

[0132] The baseline systems include the Gato system (described in Reed, Scott, et al. "A generalist agent" arXiv preprint arXiv:2205.06175 (2022)), the BC-Z system (described in Jang, Eric, et al. "Bc-z: Zero-shot task generalization with robotic imitation learning" Conference on Robot Learning. PMLR, 2022), and the BC-Z XL system (a BC-Z system with a larger number of parameters).

[0133] In Figure 7, "seen tasks" are tasks seen during training, i.e., the tasks on which the policy system was trained; "unseen tasks" are tasks in which instructions and object(s) were seen separately in the training dataset but combined in a new way; "obstructions" are tasks that include obstruction objects not previously seen during training; and "background" are tasks that include a background not previously seen, e.g., an environment with a different scene or a background with different lighting.

[0134] It can be seen that RT-1 significantly outperforms these baseline systems in all of these tasks. In particular, RT-1 has high general performance in the seen-before task (97% success rate compared to 72% for BC-Z, the highest success rate of all baseline systems), as well as remarkable generalization ability (76% success rate compared to 52% for Gato) and robustness (83% success rate for distractors compared to 47% for BC-Z, and 59% success rate for background compared to 41% for BC-Z).

[0135] This specification uses the term "configured" in the context of systems and computer program components. To say that one or more computer systems are configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when running, causes the system to perform the operation or action. To say that one or more computer programs are configured to perform a particular operation or action means that the one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0136] Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, such as the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or controlling the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random-access or serial-access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be encoded in an artificially generated propagated signal, such as a mechanically generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a receiving device suitable for execution by the data processing apparatus.

[0137] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or include special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus may optionally include code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0138] A computer program, which may be called or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted, declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, for example, one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple cooperating files, for example, files that store one or more modules, subprograms, or code portions. A computer program can be deployed to be executed on one computer or to be executed on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communications network.

[0139] As used herein, the term "database" is used broadly to refer to any collection of data. The data need not be structured in any particular way, or even at all, and may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed in a different way.

[0140] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines may be installed and run on the same computer or computers.

[0141] The processes and logic flows described herein may be implemented by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by special purpose logic circuitry, such as an FPGA or an ASIC, or a combination of special purpose logic circuitry and one or more programmed computers.

[0142] A computer suitable for running a computer program may be based on a general-purpose or special-purpose microprocessor, or both, or on other types of central processing units. Generally, the central processing unit receives instructions and data from a read-only memory, a random-access memory, or both. The basic components of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Generally, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operatively connected to receive data from or transmit data to them, or both. However, a computer need not have such devices. Furthermore, a computer may be incorporated in other devices, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0143] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0144] To provide for interaction with a user, embodiments of the subject matter described herein may be implemented in a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices may also be used to interact with the user. For example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. Furthermore, a computer may interact with a user by sending and receiving documents to a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. A computer may also interact with a user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and then receiving a reply message from the user.

[0145] A data processing apparatus for implementing machine learning models may also include dedicated hardware accelerator units for handling, for example, the general and numerically intensive parts of the machine learning training or machine learning production, i.e., inference, workload.

[0146] The machine learning model may be implemented and deployed using a machine learning framework, for example, the TensorFlow framework or the Jax framework.

[0147] Embodiments of the subject matter described herein may be implemented in a computing system that includes a back-end component, e.g., a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app that allows a user to interact with an embodiment of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), e.g., the Internet.

[0148] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data (e.g., HTML pages) to a user device, for example, to display the data to and receive user input from a user interacting with the device functioning as a client. Data generated at the user device, e.g., results of user interaction, can be received from the device by the server.

[0149] While the specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or the scope of patentable subject matter, but rather as descriptions of features that may be unique to particular embodiments of a particular invention. Certain features described herein as separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented in multiple embodiments, either separately or in any suitable subcombination. Furthermore, even if features may be described above as functioning in a particular combination and originally claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to subcombinations or variations of the subcombinations.

[0150] Similarly, while operations are illustrated in the figures and described in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order or sequential order shown, or that all of the operations shown be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0151] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still produce desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. 1. A method executed by one or more computers for controlling an agent that interacts with an environment, the method comprising: receiving a natural language text sequence characterizing a task to be performed by the agent in the environment; generating an encoded representation of the natural language text sequence; At each of a plurality of time steps, acquiring an observed image characterizing a state of the environment at the time step; processing the observed image using an image encoder neural network conditioned on the encoded representation of the natural language text sequence to generate an encoded representation of the observed image; generating a sequence of input tokens from the coded representation of at least the observed image; processing the sequence of input tokens using a Transformer neural network to generate a policy output that defines an action to be taken by the agent in response to the observed image; using the policy output to select an action to be performed by the agent; causing the agent to perform the selected action.

2. The method of claim 1 , wherein the environment is a real-world environment and the agent is a robot.

3. Generating a sequence of input tokens from the coded representation of at least the observed image includes: The method of claim 1 or 2, comprising generating a sequence of image tokens of the observed image from the encoded representation of the observed image.

4. Generating a sequence of input tokens from the coded representation of at least the observed image includes:

4. The method of claim 3, comprising generating the sequence of input tokens by combining the sequence of image tokens of the observed image with sequences of image tokens of each of one or more previous observed images received at one or more previous time steps.

5. 5. The method of claim 4, wherein combining the sequence of image tokens of the observed image with a respective sequence of image tokens of each of one or more previous observed images received at one or more previous time steps comprises applying positional coding to each image token in the sequence of image tokens of the observed image and to each image token in the respective sequence of image tokens of the one or more previous observed images.

6. The encoded representation includes a feature map including a respective feature vector for each of a plurality of regions in the observed image, and generating a sequence of image tokens for the observed image from the encoded representation includes: A method according to any one of claims 3 to 5, comprising generating an initial input sequence by flattening the feature map into a sequence of feature vectors.

7. generating a sequence of image tokens for the observed image from the encoded representation of the observed image, 7. The method of claim 6, comprising processing the initial input sequence of feature vectors using a trained module that maps the initial input sequence to a reduced input sequence that includes a smaller number of feature tokens.

8. 8. The method of claim 1, wherein the image encoder neural network includes one or more conditioning layers, each of the one or more conditioning layers being configured to receive a respective intermediate output of a respective hidden layer of the image encoder neural network and the encoded representation of the natural language instruction, (i) update the respective intermediate output of the image encoder neural network using the encoded representation of the natural language instruction, and (ii) provide the updated respective intermediate output as an input to a respective subsequent hidden layer of the image encoder neural network.

9. The method of claim 8 , wherein the one or more conditioning layers are feature-by-feature linear modulation (FiLM) layers.

10. 10. The method of claim 8 or 9, wherein the image encoder neural network is a convolutional neural network, and wherein each of the intermediate layers, each of the subsequent layers, or both, are convolutional layers.

11. The method of any one of claims 1 to 10, wherein the Transformer is a decoder-only Transformer.

12. The method of any preceding claim, wherein the policy output comprises, for each of a plurality of action dimensions, a respective categorical distribution over the possible values ​​of the action dimension.

13. 13. The method of claim 12, wherein using the policy output to select an action to be performed by the agent comprises using the respective categorical distributions to select a respective value for one or more of the action dimensions.

14. The method of any one of claims 1 to 13, wherein the image encoder neural network and the transformer neural network are jointly trained on a set of training data.

15. The method of claim 14 , wherein prior to the joint training, the image encoder neural network is pre-trained on an image classification task.

16. The method of claim 15 when dependent on claim 8, wherein the image encoder neural network does not include the one or more conditioning layers for the pre-training.

17. The method of any one of claims 14 to 16 when dependent on claim 8, wherein each conditioning layer is initialized to act as an identity transformation on the corresponding respective intermediate output prior to the joint training.

18. The method of any one of claims 14 to 17 when dependent on claim 7, wherein the learned module is further trained as part of the joint training.

19. 19. The method of any one of claims 1 to 18 when dependent on claim 14, wherein the collaborative training comprises training through imitation learning, and the training data comprises expert interaction data characterizing interactions of one or more expert agents with a corresponding environment.

20. The method of claim 19 , wherein the expert interaction data comprises simulation data.

21. The method of claim 19 , wherein the expert interaction data comprises real-world data.

22. 22. The method of claim 20 or 21, wherein the expert interaction data includes both simulation data and real-world data.

23. 23. The method of any one of claims 1 to 22, wherein generating an encoded representation of the natural language text sequence comprises processing the natural language text sequence using a text encoder neural network to generate an embedding of the natural language text sequence.

24. 23. The method of claim 22, wherein the text encoder neural network is pre-trained on a text representation learning task.

25. The method of claim 23 when dependent on claim 14, wherein the text encoder neural network is fine-tuned during the joint training.

26. 24. The method of claim 23 when dependent on claim 14, wherein the text encoder neural network remains frozen during the joint training.

27. The method of any one of claims 1 to 26, wherein the agent is a robot and the one or more computers are on-board the robot.

28. 1. A method of controlling a robot, the method comprising, at each of a plurality of time steps: acquiring, by a control system of the robot, observed images of the environment at the time steps; providing the observed image to a policy system by a control system of the robot; obtaining, by the control system of the robot, data specifying a selected action from the policy system of the robot, the policy system selecting the selected action in response to the observed image by performing respective method operations of any one of claims 1 to 27; and causing the robot to perform the selected action by the control system of the robot.

29. 30. The method of claim 28, wherein the control system of the robot is on-board the robot.

30. The method of claim 29 , wherein the policy system is onboard the robot.

31. the policy system is remote from the robot; providing the observed image includes transmitting the observed image over a data communications network; 30. The method of claim 29, wherein obtaining the data specifying the selected action comprises receiving the data specifying the selected action over the data communications network.

32. 32. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the respective method of any one of claims 1 to 31.

33. 32. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the respective method operations of any one of claims 1 to 31.

Images