Controlling a Robot Using Multimodal Language Models

Abstract

A method, system, and apparatus, including a computer program encoded on a computer storage medium, for controlling an agent that interacts with an environment. In one aspect, the method includes receiving one or more observations of an environment, receiving an input text sequence describing a task to be performed by a robot in the environment, generating an encoded representation of the input text sequence in an embedding space, generating a corresponding encoded representation of each of the one or more observations in the embedding space, generating a sequence of input tokens including the encoded representation of the input text sequence and the corresponding encoded representation of each observation, processing the sequence of input tokens using a language model neural network to generate an output text sequence including high-level natural language instructions, and determining one or more actions to be performed by the robot from the high-level natural language instructions.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 443,002, filed February 2, 2023. The disclosure of the prior application is considered part of the disclosure of this application and is incorporated by reference into the disclosure of this application. [Background technology]

[0002] This specification relates to robotics planning using neural networks.

[0003] Robotics planning refers to scheduling the physical actions of a robot to perform a task. For example, a service robot (e.g., a home or domestic robot) that performs household chores may be configured to first retrieve a household object for a human by locating the drawer where the object is stored, opening the drawer, retrieving the object, and delivering the object to the human. Each of these actions may itself involve multiple individual low-level actions by the robot's motors and actuators.

[0004] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the input of the current value of its respective parameter set. Summary of the Invention [Means for solving the problem]

[0005] Described herein is a system implemented as a computer program on one or more computers in one or more locations that generates a plan for performing a task by an agent, e.g., a robot or other machine agent, at each of a plurality of time steps. The plan can specify a sequence of two or more actions, each action selected from a space of possible actions. The sequence of two or more actions can include a first action followed by a second action. At each time step, the system then causes the agent to perform the actions one after the other in the order defined in the plan, thereby controlling the agent to interact with the environment and perform the task.

[0006] The subject matter described herein can be implemented in particular embodiments to realize one or more of the following advantages.

[0007] The robotics planning system described herein is a multimodal robot control system that can generate plans for a robot to perform a specific task by processing system inputs across multiple data modalities. The described robotics planning system combines rich semantic knowledge learned by a language model neural network, e.g., a pre-trained language model neural network, with multimodal prompts that link text and images to enable the generation of high-quality plans from a textual task description, observed images, and possibly additional data from different sensor modalities. The robotics planning system allows a user to control a robot using natural language input without being limited to specific predefined command words or phrases in the input.

[0008] Robotics planning systems are broadly applicable for generating plans for a wide range of tasks that robots can be configured to perform. Advantageously, because each system output is a plan that includes a sequence of multiple actions, controlling a robot using the described robotics planning system requires fewer control iterations than conventional systems that control a robot, thereby enabling more efficient use of computational resources (e.g., memory and computing power) to perform individually selected actions one after the other over multiple control iterations.

[0009] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0010] [Figure 1] 1 illustrates an exemplary policy system and an exemplary control system. [Figure 2] FIG. 1 illustrates an exemplary diagram of using a policy system to control agents. [Figure 3] FIG. 1 is a flow diagram of an exemplary process for controlling an agent interacting with an environment. [Figure 4] FIG. 1 is a flow diagram of an exemplary process for training a learnable module included in a policy system. [Figure 5A] FIG. 2 is an exemplary diagram of tasks that may be performed by an encoder subsystem and a language model neural network. [Figure 5B] FIG. 10 is an exemplary diagram of additional tasks that may be performed by the encoder subsystem and the language model neural network. [Figure 5C] FIG. 1 is an exemplary diagram of a task in the medical domain that can be performed by an encoder subsystem and a language model neural network. [Figure 5D]FIG. 1 is an exemplary diagram of the performance of a medical visual question answering task by an encoder subsystem and a language model neural network. DETAILED DESCRIPTION OF THE INVENTION

[0011] Like reference symbols and designations in the various drawings indicate like elements.

[0012] 1 illustrates an example policy system 100 and an example control system 101. Policy system 100 and control system 101 are examples of systems implemented as computer programs on one or more computers at one or more locations in which the systems, components, and techniques described below are implemented.

[0013] The policy system 100 and the control system 101 may control an agent 102, e.g., a robot, to accomplish any of a wide variety of tasks in an environment 104. To control an agent 102 interacting within the environment 104 to accomplish a task, the policy system 100 selects an action 144 to be performed by the agent 102, and the control system 101 then causes the agent 102 to perform the selected action 144.

[0014] As general examples, tasks may include one or more of, for example, having the agent navigate to different locations within the environment and avoid obstacles along the way, having the agent locate different objects, having the agent pick up different objects, or having different objects move to one or more specified locations, etc. To accomplish such tasks, the agent 102 moves within the environment 104, e.g., navigates and / or changes its configuration.

[0015] Typically, the control system 101 is local to the agent 102. For example, the control system 101 can be on-board the agent 102 and can be implemented on one or more computers, local workstations, or local servers that have relatively small processing and memory resources, e.g., have limited processing power and / or constrained memory space, that are on-board the agent 102.

[0016] In some embodiments, the policy system 100 is local to the agent 102. For example, like the control system 101, the policy system 100 can also be on-board the agent 102. Furthermore, in some of these embodiments, the policy system 100 may be part of the control system 101 that causes the agent 102 to perform the action 144.

[0017] In other implementations, policy system 100 is remote from agent 102. For example, unlike control system 101, policy system 100 can be hosted in a data center, which can be a distributed computing system having hundreds or thousands of computers in one or more locations. That is, control system 101 can receive data identifying actions 144 from an external source, rather than, for example, generating such data on-board agent 102.

[0018] In these embodiments, the policy system 100 and the control system 101 may be connected by a data communications network, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof.

[0019] In these embodiments, the control system 101 of the agent 102 interacts with a remote policy system 100 hosted in a data center that has significantly more computing and other resources than are available on-board the agent 102 in order to reduce latency in selecting an action 144, reduce consumption of the agent's 102's limited power supply in selecting an action 144, or both.

[0020] In some implementations, the policy system 100, the control system 101, or both may expose one or more application programming interfaces (APIs) or other data interfaces that facilitate control of the agent 102. For example, a user of the agent 102 may use an API made available by the action selection system 100 to provide a natural language text sequence 108 that characterizes a task to be performed by the agent. As another example, the policy system 100 and the control system 101 may interact via an API between the two systems; for example, the control system 101 may use the API to provide observations 106 to the policy system 100, and the policy system 100 may use the API to provide data specifying a determined action 144 to the control system 101.

[0021] In particular, at each of a plurality of time steps, the policy system 100 and control system 101 control the agent based on a policy output 142 for the time step generated by a set of learnable modules configured through training to control the agent 102 in response to one or more observations 106 characterizing the environment 104 and natural language text sequences 108 characterizing the task performed by the agent 102 received by the policy system 100 at the time step.

[0022] In some cases, the one or more observations 106 include one observation image or multiple observation images. For example, the observation image may include an image captured by a visual sensor of the agent 102 or by a visual sensor located in the environment 104. The visual sensor may be, for example, a camera, such as a still or video camera, a depth camera, or a lidar sensor.

[0023] In other cases, the one or more observations 106 include low-dimensional data generated from robot hardware or from data collected from other types of sensors that make observations as the robot interacts with its environment. These sensors may include force sensors, electrical connection sensors, acceleration sensors, audio sensors, gyros, contact sensors, radar sensors, and proximity sensors, such as infrared proximity sensors, capacitive proximity sensors, or inductive proximity sensors, to name just a few. The robot hardware may include actuators, motors, drivers, grippers, etc., to name just a few.

[0024] In these cases, the low-dimensional data may be in the form of a vector ("state vector"), an array, or any other suitable variable-sized data structure, where the data structure represents state information for each of one or more entities in the environment 104. For example, the low-dimensional data may include a corresponding state vector for each object in the environment 104, representing one or more of the object's pose, size, or color. For example, the pose may be a global or relative pose of the object held by the robot. As another example, the low-dimensional data may include one or more state vectors of the agent 102, e.g., a state vector representing the pose (position and orientation) of the robot, a state vector representing the velocity of each joint of the robot, a state vector representing the torque of each joint of the robot, e.g., gravity-compensated torque feedback, a state vector representing the acceleration of each joint of the robot, or a combined state vector representing some combination of these of the robot.

[0025] In still other cases, one or more observations 106 include both observed images and low-dimensional data.

[0026] In any of these cases, the one or more observations 106 may be obtained by the policy system 100 directly from sensors or robot hardware, or through the control system 101 of the agent 102. For example, the control system 101 of the agent 102 obtains one or more observations 106 of the environment 104 at a time step from sensors or robot hardware and then provides the one or more observations 106 to the policy system 100.

[0027] In embodiments in which the policy system 100 is remote from the agent 102, providing the one or more observations 106 to the policy system 100 may include, for example, transmitting data representing the one or more observations 106 over a data communications network connecting the policy system 100 and the control system 101. As another example, providing the one or more observations 106 to the policy system 100 may include providing data to the policy system 100 specifying the name or network location (e.g., a uniform resource locator (URL)) of a server from which the policy system 100 can obtain the one or more observations 106.

[0028] The natural language text sequence 108 may be or include a natural language query that explicitly or implicitly defines a task, for example, by asking a question or describing a goal state of the environment. The natural language text sequence 108 may reference one or more objects in the environment. The natural language text sequence 108 may be received from other agents in the environment 104 or from the control system 101 of the agent 102 at each of multiple time steps.

[0029] For example, other agents in the environment 104 can speak commands, which the control system 101 or other system can transcribe into natural language text sequences 108 and then provide the transcriptions at each time step to the policy system 100.

[0030] As another example, the control system 101 may receive instructions entered by a user via an input device specifying a natural language text sequence 108, e.g., text-based input, selection-based input, or audio-based input, and then provide the instructions at each time step to the policy system 100. Examples of input devices include a touchscreen, a mouse, a keyboard, a voice response system, a video camera, a microphone, or any other type of device through which a user or another agent in the environment can submit a query.

[0031] More specifically, at each of a plurality of time steps, the policy system 100 obtains one or more observations 106 and natural language text sequences 108 for the time step and processes the one or more observations 106 and natural language text sequences 108 using the encoder subsystem 110 to generate an encoded representation of each of the one or more observations (or, for short, "encoded observations") and an encoded representation of the natural language text sequence (or, for short, "encoded text").

[0032] In general, the policy system 100 acquires different observations 106 over multiple time steps. For example, the observations 106 may include different observation images, state vectors, or both, at different time steps.

[0033] In some cases, the policy system 100 captures the same natural language text sequence 108 across multiple time steps. For example, the natural language text sequence 108 may describe a long-term goal for an entire episode, such as "sort objects into corners by color," "center all objects," or "remove blue objects from the line." An episode is generally a period during which an agent attempts to perform a specified task. A time step may be defined by a specific or threshold number of time steps and / or may continue until other termination criteria are met (e.g., until a termination signal is received indicating the task was successfully performed).

[0034] In other cases, policy system 100 obtains different natural language text sequences 108 across multiple time steps. For example, control system 101 or another system can iteratively update the natural language text sequence, i.e., generate an updated natural language text sequence at each of multiple time steps based, for example, on previous actions performed by the agent in the previous time step, previous states of the environment, or both, and provide the updated natural language text sequence to policy system 100. In this example, natural language text sequence 108 can describe an immediate goal, such as "move robot forward," "reach target position (x,y)," etc.

[0035] As another example, a user may provide an updated natural language text sequence after an episode begins in response to the user providing an initial natural language text sequence, and thus policy system 100 obtains the initial natural language text sequence at each of some of the multiple time steps and the updated natural language text sequence at each of others of the multiple time steps.

[0036] For each of one or more observations 106, the encoded observation may be an embedding of the observation 106. As used herein, an "embedding" is a sequence of vectors of one or more numerical values, e.g., floating-point values ​​or other values, each vector having a predetermined number of dimensions. The space of possible vectors having a predetermined number of dimensions is referred to as the "embedding space."

[0037] Like the encoded observations, the encoded text may be an embedding of the natural language text sequence 108, i.e., it may include a sequence of one or more vectors of numerical, e.g., floating-point, or other values, each vector having a predetermined dimensionality. In some implementations, the encoded observations and the encoded text reside in the same embedding space, i.e., the constituent vectors of an embedding of the encoded observations and the encoded text all have the same dimensionality.

[0038] The encoder subsystem 110 includes any combination of trained modules that enable the encoder subsystem 110 to generate, at each time step, an encoded representation and encoded text from one or more observations 106 and natural language text sequences 108 received at that time step. The trained modules may be, but need not be, neural networks. "Trained" means that the parameters of these modules are tuned during the training of the policy system 100.

[0039] 1, encoder subsystem 110 includes one or more encoder neural networks 114, a projector neural network 116, and an embedding matrix 118. When one or more observations 106 include state vectors or other low-dimensional data, one or more encoder neural networks 114 may include a multi-layer perceptron (MLP) that receives as input the state vectors included in the observations 106, processes the input, and generates as output an embedding for each state vector. When one or more observations 106 include observed images, one or more encoder neural networks 114 may include an image encoder neural network that receives as input the observed images, processes the input, and generates as output an embedding of the observed images.

[0040] In some cases, the one or more encoder neural networks 114 include an image encoder neural network having a convolutional neural network architecture that includes one or more convolutional layers.

[0041] In other cases, the one or more encoder neural networks 114 include an image encoder neural network with a transformer-based architecture including one or more attention layers. For example, the image encoder neural network may have a 3D-aware Object Scene Representation Transformer (OSRT) architecture described in Sajjadi, MSM, et al. Object Scene Representation Transformer NeurIPS, 2022. The OSRT neural network includes a slot-mixer decoder that computes an output by applying an attention mechanism across multiple slots, each representing a respective portion of the observed image, using one or more query rays with predetermined directions.

[0042] In still other cases, the one or more encoder neural networks 114 include an image neural network with a Vision Transformer (ViT) architecture. For example, the neural network of the image encoder can have one of the ViT architectures described in Dosovitskiy, A., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020; Chen, X., et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794, 2022; and Ryoo, MS, et al. Tokenlearner: What can 8 learned tokens do for images and videos? arXiv preprint arXiv:2106.11297, 2021.

[0043] In these cases, the image encoder neural network may be configured to receive as input the observed image and the object instance mask, process the input, and generate as output an embedding of the observed image. The object instance mask, which may be generated by another trained module of the policy system 100, e.g., an image segmentation neural network, is a digital representation of the observed image that identifies which portions of the observed image each include a corresponding target object, e.g., an object referenced in the natural language text sequence 108, and which portions of the observed image do not include any target object, e.g., instead include only one or more background objects, e.g., the ground or other background / static objects.

[0044] If the outputs of these encoder neural networks have different dimensionality than the inputs of the language model neural network 130 of the policy system 100, the embeddings may need to be projected to have the same dimensionality as the inputs of the language model neural network 130 for subsequent processing by the language model neural network 130. Thus, Figure 1 shows that the encoder subsystem 110 uses a projector neural network 116 to process, or project, the embeddings produced as output by the encoder neural network 114. The projected embeddings are then used as encoded representations of one or more observations 106.

[0045] For example, the projector neural network 116 can be configured as a multi-layer perceptron (MLP) or other suitable network architecture, e.g., having a single linear projection layer, receiving as input the embeddings being produced as output by the encoder neural network 114, processing the input, and producing as output an encoded representation of each of the one or more observations 106.

[0046] Alternatively, if the embeddings produced by the encoder neural network 114 all have the same dimensionality, the embeddings can be used directly as the encoded observations for one or more observations 106 (and thus the encoder subsystem 110 does not need to include a projector neural network 116).

[0047] In some cases, to generate the encoded text, the encoder subsystem 110 generates a one-hot encoded representation of the natural language text sequence 108. That is, it represents the natural language text sequence 108 as a sequence of one-hot encoded vectors and then maps each one-hot encoded vector to a corresponding numeric value according to a predefined mapping. The sequence of one-hot encoded vectors may include a corresponding one-hot encoded vector for each subword included in the natural language text sequence 108. A subword may be an entire word, or alternatively, a phoneme, a syllable, a portion of a syllable, or any other such portion of a word including one or more characters.

[0048] 1, the predefined mapping is represented as an embedding matrix 118 having learned values, and thus, to generate the encoded text, the encoder subsystem 110 determines the product of each one-hot encoded vector in the one-hot encoded representation with the embedding matrix 118. For example, the embedding matrix 118 may have a size of k×|W|, where k is the dimensionality of the embedding space and |W| is the size of the subword vocabulary (the total number of subwords in the vocabulary).

[0049] In other examples, the predefined mappings can be represented as different trained modules. For example, the different trained modules can be text encoder neural networks. The text encoder neural networks can have any suitable neural network architecture, such as a feed-forward architecture, an encoder-specific transformer neural network, or a recurrent architecture, that enables the neural network to map natural language text sequences 108 to encoded text.

[0050] At each of a plurality of time steps, the policy system 100 generates a sequence of input tokens 112 for the time step from at least the encoded observations and the encoded text. The sequence of input tokens 112 includes at least a sequence of observation tokens from the encoded observations and a sequence of text tokens from the encoded text. Thus, the sequence of input tokens 112 is generated from data from two or more modalities. As used herein, a "token" (e.g., an image token or a text token) is an ordered set of vectors or other numerical values ​​with a fixed number of dimensions, i.e., the number of values ​​in the ordered set is constant across different tokens.

[0051] In some cases, the sequence of input tokens 112 may include additional tokens, which, if included, may be prepended to, appended to, or interleaved within the sequence of image tokens and the sequence of text tokens.

[0052] For example, the additional tokens 112 may include template tokens generated from one or more natural language words or clauses selected from a predetermined template (the words "given" and "Q:" in FIG. 2, described below, are examples of such words). Some implementations of the policy system 100 use these template tokens to convert the sequence of input tokens 112 into a format that more closely resembles the format of a training input sequence for a task, such as a language modeling task or a visual question answering task, on which the language model neural network 130 was pre-trained, thereby stabilizing the performance of the language model neural network 130.

[0053] As another example, the additional tokens 112 may include object reference tokens. To generate the object reference tokens, the policy system 100 may assign an index to each of one or more objects included in the environment 104, and then generate, as the object reference token, an encoded representation of the mapping between the one or more objects and the assigned index. For example, the mapping may be in the following format: "Object 1 is,<obj_1> , ..., object j is<obj_j> ". Some implementations of the policy system 100 use these object reference tokens in situations where the environment 104 contains multiple objects that may not be easily identifiable in natural language with a limited number of words, for example, when there are multiple blocks in a table with the same color but in different positions. This allows the language model neural network 130 to generate high-level plans that accurately reference one or more of these objects.

[0054] After generating a sequence of input tokens 112 at each of a number of time steps, the policy system 100 processes the sequence of input tokens 112 using a language model neural network 130 to autoregressively generate an output text sequence 132 containing instructions to the agent in natural language.

[0055] The language model neural network 130 can have any suitable transformer-based architecture, such as an encoder-only transformer architecture, an encoder-decoder transformer architecture, a decoder-only transformer architecture, or other attention-based architecture, and the language model neural network 130 includes one or more attention layers. As a particular example, the language model neural network 130 can be one of the neural networks described in Chowdhery, A., et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022, Kojima, T., et al. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916, 2022, and Alayrac, J.-B., et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198, 2022.

[0056] In particular, the natural language instructions may define or otherwise specify a high-level plan that includes a sequence of two or more actions to be performed by the agent 102 at a time step. Generally, the high-level plan defines the actions to be performed by the agent and the sequential order in which the actions should be performed. Each of these actions may itself include multiple lower-level actions that may be performed by the agent 102.

[0057] For example, output text sequence 132 may be "open the drawer and take the ball out of the drawer." In this example, output text sequence 132 defines a first action of opening the drawer, followed by a second action of taking the ball out of the drawer. In another example, output text sequence 132 may define a different sequence of more or fewer actions.

[0058] The plans specified by the natural language instructions contained in the output text sequence 132 are referred to as "high-level" because they are not directly used to control the agent 102. Instead, after generating the output text sequence 132 by using the language model neural network 130 at each of multiple time steps, the policy system 100 uses other trained modules to map the output text sequence 132 to one or more low-level actions (or "actions" for short) 144, and then controls the agent based on the actions 144.

[0059] 1, another trained module is policy neural network 140. Policy neural network 140 is a neural network that receives as input, output text sequence 132, processes the input, and generates as output, one or more policy outputs 142 that can be used to determine one or more actions 144 to be taken by agent 104 at a time step, e.g., in response to observation 106.

[0060] For example, the policy neural network 140 can be configured as a goal-conditioned policy neural network as described in Lynch, C., et al. Language conditioned imitation learning over unstructured data. arXiv preprint, arXiv:2005.07648, 2020, or as a transformer neural network as described in Brohan, A., et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint, arXiv:2212.06817, 2022, the entire contents of which are incorporated herein by reference in their entirety.

[0061] In other examples, the other trained modules may not be neural networks: for example, the trained modules may be regression tree / forest models, decision tree / forest models, support vector machines, or other machine learning models that can map high-level plans to low-level actions.

[0062] When configured as a policy neural network, the policy output 142 may specify one or more actions 144 in any suitable manner. For example, each policy output 142 may include a respective numerical probability value for each action in a set of possible actions that may be performed by the agent. In this example, the policy system 100 may determine the action 144 to be performed by the agent 102, for example, by sampling the actions according to their probability values ​​or by selecting the action with the highest probability value.

[0063] Similarly, each policy output 142 may assign a respective numerical value to each action dimension in a set of action dimensions, e.g., a set of end effector movement action dimensions, a set of arm motion action dimensions, a set of primitive movement action dimensions, or some combination thereof, and policy system 100 may determine an action 144 to be performed by agent 102 from the respective numerical values ​​for the set of action dimensions. The numerical values ​​may be assigned deterministically or probabilistically by policy output 142, for example, where policy output 142 parameterizes a distribution for each action dimension from which the numerical values ​​of the action dimension are sampled.

[0064] As another example, each policy output 142 may directly define an action to be performed by the robotic agent, for example, by defining torque values ​​to be applied to the agent's joints.

[0065] Continuing with the example above, the policy neural network 140 can generate one or more corresponding policy outputs 142 from the output text sequence 132 (“open the drawer and take the ball out of the drawer”), and the policy system 100 can determine at least three actions 144 from the corresponding policy outputs 142: a first action that moves the robot arm to a position (x, y, z) that is the location of the handle of the drawer, followed by a second action that uses the end effector to hold on to the handle, followed by a third action that moves the robot arm outward with the handle within the grasp of the robot arm.

[0066] After determining one or more actions 144 to be performed by the agent 102 at a time step, the policy system 100 provides data identifying the determined actions 144 to the control system 101. In embodiments where the policy system 100 is remote from the agent 102, providing the data identifying the determined actions 144 may include, for example, transmitting the data identifying the determined actions 144 over a data communications network connecting the policy system 100 and the control system 101.

[0067] The control system 101 then causes the agent 102 to perform the determined action 144 at the time step. For example, the control system 101 may do this by generating instructions for the agent 102 that, when executed, cause the agent 102 to perform the determined action 144, by sending one or more control inputs directly to appropriate controls of the agent, or by using other suitable control techniques.

[0068] In some implementations, the environment 104 is a real-world environment and the agent 102 is a machine agent that interacts with the real-world environment. For example, the agent may be a robot that interacts with the environment to achieve a goal, such as to position an object of interest within the environment, to move an object of interest to a specified location within the environment, to physically manipulate an object of interest within the environment in a specified manner, or to navigate to a specified destination within the environment, or the agent may be an autonomous or semi-autonomous land, air, or water vehicle that navigates through the environment to a specified destination within the environment.

[0069] Actions 144 may be control inputs for controlling a robot, such as torques or higher-level control commands for the joints of the robot, or control inputs for controlling an autonomous or semi-autonomous land, air, or water vehicle, such as torques or higher-level control commands for the control surfaces or other control elements of the vehicle.

[0070] In other words, actions 144 may include, for example, position, velocity, or force / torque / acceleration data for one or more joints of a robot or other mechanical agent part. Actions may additionally or alternatively include electronic control data, such as motor control data, or more generally, data for controlling one or more electronic devices in the environment, the control of which affects the observed state of the environment. For example, in the case of an autonomous or semi-autonomous land, air, or water vehicle, actions may include actions that control navigation, e.g., steering, and actions that control movement, e.g., braking and / or accelerating the vehicle.

[0071] In some implementations, the environment 104 is a simulated environment and the agent 102 is implemented as one or more computer programs that interact with the simulated environment. For example, the environment can be a computer simulation of a real-world environment and the agent can be a simulated machine agent that navigates through the computer simulation.

[0072] For example, the simulated environment may be a mobile simulation environment, such as a driving or flight simulation, and the agent may be a simulated vehicle that navigates through the mobile simulation. In these implementations, the actions 144 may be control inputs for controlling a simulated user or a simulated vehicle. As another example, the simulated environment may be a computer simulation of a real-world environment, and the agent may be a simulated robot that interacts with the computer simulation.

[0073] In general, when environment 104 is a simulated environment, actions 144 may include simulated versions of one or more of the actions or types of actions previously described.

[0074] In some implementations, environment 104 is a suitable execution environment, e.g., a runtime environment or operating system environment, implemented on one or more computing devices, such as a smartphone, a tablet computer, a wearable device, an automotive system, a standalone personal assistant device, etc., and agent 102 is a virtual agent (also known as an “automated assistant” or “mobile assistant”) that can be interacted with by a user through the computing device. The virtual agent can receive input (e.g., typed or spoken natural language input) from the user and respond with responsive content (e.g., visual and / or audible natural language output). The virtual agent can provide a wide range of functionality through interactions with various local and / or third-party applications, websites, or other agents. In these implementations, action 144 can include any activity or operation that can be performed or initiated by a user on the computing device, for example, within application software installed on the computing device.

[0075] In some cases, policy system 100 can be used to control an agent's interactions with a simulated environment, and policy system 100 (or other training system) can train a set of learnable modules used to control agent 102 based on the agent's (or other agent's) interactions with the simulated environment to determine trained values ​​for parameters of the set of learnable modules. Training a set of learnable modules is described in more detail below with reference to FIG. 4.

[0076] After the set of learnable modules has been trained based on the interaction of agent 102 (or other agents) with the simulated environment, the trained modules can be used by policy system 100 to control the interaction of the real-world agent with the real-world environment, i.e., to control the agent that was simulated in the simulated environment.

[0077] Training a learnable module based on an agent's interactions with a simulated environment (i.e., instead of a real-world environment) can avoid agent wear and reduce the possibility that the agent may damage itself or aspects of its environment by performing poorly chosen actions.

[0078] 2 is an exemplary diagram of controlling an agent 202 by using a policy system 200 and a control system 201. In FIG. 2, the agent 202 is a robot having a base, one or more arms, and an end effector. The policy system 200 and the control system 201 may be the same as or similar to the policy system 100 and the control system 101, respectively, of FIG.

[0079] The policy system 200 and the control system 201 can control the agent 202 to accomplish a task within an environment by repeatedly performing one iteration of these operations at each of multiple time steps to determine one or more actions to be performed by the agent 202, and then cause the agent 202 to perform the one or more determined actions.

[0080] At a high level, at each of a plurality of time steps, the operations performed by the policy system 200 include obtaining one or more observations 206 and a natural language text sequence 208 that describes a task to be performed by an agent 202 in an environment, and processing the one or more observations 206 and the natural language text sequence 208 to generate an output text sequence 232 that includes natural language instructions for the agent 202 at the time step.

[0081] The natural language instructions may define or otherwise specify a high-level plan that includes a sequence of two or more actions to be performed by the agent 102 starting from a time step. Each of these actions may itself include multiple lower-level actions by the agent 202.

[0082] Specifically, when configured with the architecture illustrated in FIG. 1 , which does not rely on human interaction (e.g., human-in-the-loop interaction) or other supervisory signals, policy system 200 can repeatedly execute these actions at a frequency of 1 Hz. That is, policy system 200 can generate 10 potentially different plans (specified by 10 output text sequences) per second for agent 202 to follow, which is higher than the control frequency achieved by many existing robotic control systems. Such a high frequency allows agent 202 to react in a more natural and fluid manner, resulting in greater movement accuracy, shorter cycle times, and greater reliability in completing tasks.

[0083] As shown, at each of multiple time steps, policy system 200 obtains a natural language text sequence 208 that implicitly characterizes the task of grasping an object by referencing the object in the question form, "How do I grasp the blue block?". Policy system 200 processes natural language text sequence 208 using an embedding matrix to generate an encoded representation of the natural language text sequence ("encoded text") in the embedding space.

[0084] Further, at each of multiple time steps, policy system 200 takes one or more observations 206 and processes each of the one or more observations 206 using an encoder neural network (configured as a ViT neural network in the example of FIG. 2), and in some cases, using a projector neural network, to generate a corresponding encoded representation of each of the one or more observations in the embedding space (an “encoded observation”).

[0085] At each of a plurality of time steps, the policy system 200 generates a sequence of input tokens 212 for the time step from the encoded observations and the encoded text. The sequence of input tokens 212 includes a sequence of observation tokens from the encoded observations and a sequence of text tokens from the encoded text at each of a plurality of input positions in the input order.

[0086] 2, the policy system 200 interleaves the observation tokens of the encoded observations with the text tokens of the encoded text. Thus, the sequence of input tokens 212 includes multiple segments of the encoded text separated by encoded observations: <text1> <obs1> … <obsn> <textn> Here, <obs1>" represents a first encoded observation, e.g., an encoded representation of low-dimensional observation data, and " <obsn>" represents the nth encoded observation, e.g., the encoded representation of the observed image, and " <text1>" represents the first encoded text corresponding to the first part of the input text sequence ("Given"), and " <textn>" represents the first encoded text corresponding to the nth part of the input text sequence ("Q: How do I grab the blue block?").

[0087] In other examples, the sequence of input tokens 212 may include a sequence of observation tokens from the encoded observations and a sequence of text tokens from the encoded text in different orders, e.g., non-interleaved sequential order, at each of multiple input locations. For example, the sequence of input tokens 212 may include an image token followed by a text token, or vice versa. The sequence of input tokens 212 may also include additional tokens, e.g., template tokens, object reference tokens, or both, as described above.

[0088] The policy system 200 then processes the sequence of input tokens 212 using a language model neural network 230 to autoregressively generate an output text sequence 232 containing natural language instructions for the agent 202. The natural language instructions may specify a high-level plan containing a sequence of two or more actions to be performed by the agent 202 at a time step. Each of these actions may itself contain multiple lower-level actions by the agent 202.

[0089] The policy system 200 processes the output text sequence 232 to generate one or more policy outputs 242 that can be used to determine one or more actions to be performed by the agent 202, and then causes the agent 202 to perform the determined actions, for example, by providing data identifying the determined actions to the control system 201.

[0090] In the example of Figure 2, the natural language instruction is "A: First, grab the yellow block..." and the actions that can be determined from the policy output include (i) a first action that moves the robot arm to position (x, y, z), which is a position that is a distance d above the yellow block; (ii) a second action that opens the gripper; (iii) a third action that moves the robot arm to position (x, y, zd), which is the position of the yellow block; and (iv) a fourth action that closes the gripper and grabs the yellow block.

[0091] The policy system 200 may repeat these actions at a frequency of 1 Hz or greater until some termination condition is met, for example, until it generates a particular output text sequence containing a plan that includes the termination actions, until a predetermined number of iterations of these actions have been performed, or until a predetermined amount of time has passed.

[0092] 3 is a flow diagram of an exemplary process 300 for controlling an agent interacting with an environment. For convenience, process 300 is described as being performed by one or more computer systems located at one or more locations. For example, a policy system, such as policy system 100 of FIG. 1 , suitably programmed in accordance with this specification, may perform process 300.

[0093] The system controls the agent to perform tasks in the environment by repeatedly performing iterations of process 300 at each of a number of time steps (hereafter referred to as the "current" time step).

[0094] The system acquires one or more observations of an environment including one or more objects (step 302). In some cases, the one or more observations include an observation image that characterizes the state of the environment at the current time step. For example, the observation image may be captured by a visual sensor of the agent or by a visual sensor located in the environment.

[0095] In other cases, the one or more observations include low-dimensional data generated from the robot hardware or from data collected from other types of sensors that make observations as the robot interacts with its environment. For example, the low-dimensional data may include a first state vector representing one or more of a pose, a size, or a color of objects contained within the environment, and a second state vector representing a pose of the agent.

[0096] In still other cases, the one or more observations include both observed images and low-dimensional data.

[0097] Typically, the system acquires different observations over multiple time steps, e.g., the observations may contain different observation images, state vectors, or both, at different time steps.

[0098] The system obtains a natural language text sequence that explicitly or implicitly describes a task to be performed by an agent in the environment, for example, by asking a question or describing a goal state of the environment (step 304). To describe the task, the natural language text sequence may reference one or more objects in the environment.

[0099] In some cases, the system obtains the same natural language text sequence across multiple time steps, such as when the natural language text sequence describes a long-term goal for an entire episode. In other cases, the system obtains different natural language text sequences across multiple time steps.

[0100] The system generates an encoded representation of the natural language text sequence in the embedding space (step 306). To generate the encoded representation of the input text sequence, the system may represent the natural language text sequence as a sequence of one-hot encoded vectors and then map each one-hot encoded vector to a corresponding numeric value according to a predefined mapping. The sequence of one-hot encoded vectors may include a corresponding one-hot encoded vector for each subword included in the natural language text sequence. For example, the predefined mapping may be represented as an embedding matrix.

[0101] The system generates a corresponding encoded representation of each of the one or more observations in the embedding space (step 308). To generate the encoded representation of the one or more observations, the system can process each of the one or more observations using an encoder neural network and, in some cases, a projector neural network. For example, the encoder neural network can be an image encoder neural network configured as a convolutional neural network, a transformer neural network, or a vision transformer (ViT) neural network, and the projector neural network can be a multilayer perceptron (MLP).

[0102] The system generates a sequence of input tokens (step 310) that includes text tokens from the encoded representation of the natural language text sequence and observation tokens from the corresponding encoded representation of each observation. In some cases, the system generates the sequence of input tokens by interleaving the observation tokens with the text tokens. In other cases, the system generates the sequence of input tokens by prepending the observation tokens to the text tokens, by appending the observation tokens to the text tokens, or by concatenating the sequence of input tokens and the sequence of text tokens in some other order. In some cases, the sequence of input tokens can also include additional tokens, such as template tokens, object reference tokens, or both.

[0103] The system processes the sequence of input tokens using a language model neural network to autoregressively generate an output text sequence containing natural language instructions for the agent (step 312). The natural language instructions may define or otherwise specify a high-level plan containing a sequence of two or more actions to be performed by the agent in the current time step. Generally, the high-level plan defines the actions to be performed by the agent and the sequential order in which the actions should be performed. Each of these actions may itself contain multiple lower-level actions.

[0104] The system determines from the natural language instructions one or more actions to be performed by the agent at the current time step (step 314). To map the high-level plan defined by the output text sequence to low-level actions, the system may process the output text sequence using a policy neural network to generate one or more policy network outputs, and then determine one or more actions to be performed by the agent according to the one or more policy network outputs.

[0105] The system then causes the agent to perform the determined action, for example, by sending control inputs directly to the agent, or by sending instructions or other data, for example, via a data communications network, to the agent's control system that causes the agent to perform the determined action.

[0106] Process 300 may be performed when controlling an agent to perform a task where the action to be taken is unknown, e.g., the action that will result in progress toward completing the task. Process 300 may also be performed as part of selecting an action to be performed by an agent based on processing observations and natural language text sequences, e.g., observed images, derived from a set of training datasets, for which the action to be performed by the agent in response is known, to train a set of learnable modules to determine trained values ​​for the parameters of the learnable modules.

[0107] 4 is a flow diagram of an exemplary process 400 for training a learnable module included in a policy system. For convenience, process 400 is described as being performed by one or more computer systems located at one or more locations. For example, a policy system, such as policy system 100 of FIG. 1, or other training system suitably programmed in accordance with this specification, may perform process 400.

[0108] For example, the system may perform process 400 to train one or more of the encoder neural network 114, projector neural network 116, embedding matrix 118, and possibly the language model neural network 130, policy neural network 140, or both, included in the policy system 100 of Figure 1. In some cases, the system trains one or more of the encoder neural network 114 and projector neural network 116 from scratch. For example, the system may train the neural networks 114, 116 from randomly initialized parameters, while the system fine-tunes the embedding matrix 118 and language model neural network 130 from their pre-trained parameter values.

[0109] To train these learnable modules, the system obtains a set of training data (step 402), which may include training data generated based on the interaction of the agent (or other agents) with the environment.

[0110] In one example, the training dataset is N training examples

number

number

[0111] The system trains the learnable modules included in the policy system on the training dataset (step 404). To train the learnable modules, the system selects training examples i from the training dataset and, for each training example i selected from the training dataset, uses an encoder neural network, a projector neural network, and a language model neural network to generate u i Individual observations and natural language text sequences

number

number

[0112] The system updates the values ​​of the parameters of the neural network based on machine learning training techniques, such as gradient descent using a backpropagation training technique that uses an appropriate optimizer, such as stochastic gradient descent, RMSprop, or Adam optimizer, to optimize an objective function (e.g., a cross-entropy objective function specific to the next token prediction task).

[0113] For each training example i selected from the training dataset, the next token prediction task is

number

number

[0114] In some cases, the system trains the learnable modules from scratch, e.g., from randomly initialized parameters, while in other cases, the system trains at least some of the learnable modules from pre-trained parameter values. For example, one of the encoder neural networks may be pre-trained on object-centric 3D scene representation learning as described in Sajjadi, MSM, et al. Object Scene Representation Transformer NeurIPS, 2022.

[0115] As another example, the policy neural network may be pre-trained based on optimizing a language-conditioned imitation learning objective using unstructured and / or unlabeled demonstration data (i.e., no language or task labels) as described in Lynch, C., et al. Language conditioned imitation learning over unstructured data. arXiv preprint, arXiv:2005.07648, 2020.

[0116] As yet another example, the language model neural network may be pre-trained on one or more of the language modeling tasks described in Chowdhery, A., et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. In this example, pre-training the language model neural network may also include learning pre-trained values ​​of the embedding matrix.

[0117] Additionally, during training, the system may incorporate any number of techniques to improve the speed, effectiveness, or both of the training process. For example, instead of updating the parameters of all learnable modules, the system updates only some of the parameters of the learnable modules and keeps other parameters of the learnable modules fixed during training. For example, if the language model neural network has been pre-trained on a language modeling task, the system may keep the pre-trained parameter values ​​of the language model neural network and the policy neural network fixed and only update the parameter values ​​of one or more of the encoder neural network and the projector neural network.

[0118] As another example, in addition to training a learnable module on an agent control task, the system also trains the learnable module on a related task that also requires multi-model input, but that does not involve agent control. For example, the set of training data can also include training data specific to an embodied reasoning task, a visual question answering task, an image captioning task, etc. Cross-task co-training on a wide variety of training data can improve the system's performance on individual tasks, for example, by transferring knowledge from the visual-linguistic domain to the agent control domain.

[0119] Although the tasks are generally described herein as agent control tasks involving controlling an agent in an environment, the language model neural networks, encoder neural networks, and projection neural networks can similarly be configured to perform any of a broader range of multimodal machine learning tasks, which are tasks that require a neural network to process inputs that include data from two or more modalities in order to generate an output for the task.

[0120] Specifically, in some cases, a sequence of input tokens from a user may be a question about the environment or other query that does not require the robot to take any action. In these cases, rather than outputting a text sequence that defines a high-level plan, the output text sequence generated by the language model neural network may be a response to the query. Examples of such queries include queries about open vocabulary image classification, open vocabulary object detection, image captioning, text-based image retrieval, image-based search, visual question answering, etc.

[0121] In some cases, the sequence of input tokens from the user may represent a sequence of health data, and the output text sequence may include a sequence of predicted treatments. The input tokens may then represent any aspect of the patient's health, such as data from the patient's blood and other medical tests and / or other patient data, and the output text sequence may represent diagnostic information, for example, related to the patient's medical condition and / or regarding suggested treatments for the patient and / or regarding the likelihood of an adverse health event for the patient.

[0122] Figure 5A is a diagram of example tasks that may be performed by encoder subsystem 510 and language model neural network 530. Figure 5B is a diagram of additional example tasks that may be performed by encoder subsystem 510 and language model neural network 530. Encoder subsystem 510 and language model neural network 530 may be the same as or similar to encoder subsystem 110 and language model neural network 130, respectively, of Figure 1.

[0123] For example, FIG. 5B shows that the encoder subsystem 510 and language model neural network 530 can be configured to perform robotic vision, interaction, and planning tasks, where the observations include images, the natural language text sequence includes a description of a task to be performed by the robot by interacting with the environment ("Use all of the visible ingredients to make the batter for the cake"), and the output text sequence specifies a high-level plan, including a sequence of two or more actions that can be performed by the robot to accomplish the task ("1. Crack the eggs. 2. Put the eggs in the bowl...").

[0124] As another example, FIG. 5B shows that the encoder subsystem 510 and language model neural network 530 can be configured to perform a robot-directed task, where the observations include images, the natural language text sequence includes a question about the environment characterized by the images ("If a robot wanted to be helpful here, what steps should the robot take?"), and the output text sequence specifies a high-level plan including a sequence of two or more actions that can be performed by the robot in the environment ("Clean the table...Pick up the trash...").

[0125] As another example, FIG. 5B shows that the encoder subsystem 510 and language model neural network 530 can be configured to perform a physics prediction task, where the observations include images characterizing an object, the natural language text sequence includes a question regarding the physical state or condition of the object characterized by the image ("What will the robot do next?"), and the output text sequence includes a response to the question ("Falling").

[0126] As another example, FIG. 5B shows that the encoder subsystem 510 and language model neural network 530 can be configured to perform a multi-image relational inference task, where the observations include two or more images featuring different objects (a coffee pot and a stove), the natural language text sequence includes a question regarding the relationship, e.g., spatial, temporal, or semantic relationship, between the different objects characterized by the two or more images, and the output text sequence includes a response to the question ("the pot is on the stove").

[0127] 5B illustrates that the encoder subsystem 510 and language model neural network 530 have the generalization ability to adapt to tasks, such as multimodal thought chain tasks and multi-image relational reasoning tasks, involving novel objects not previously seen during system training, e.g., via one-shot or zero-shot learning, thereby avoiding the need to expend additional computing resources, power resources, or both, to train separate neural networks corresponding to different tasks.

[0128] 5C is an example diagram of a task in the medical domain that may be performed by encoder subsystem 510 and language model neural network 530. Encoder subsystem 510 and language model neural network 530 may be the same as or similar to encoder subsystem 110 and language model neural network 130, respectively, of FIG.

[0129] For example, the encoder subsystem 510 and the language model neural network 530 may be configured to process medical images depicting regions of a patient's body and generate a final classification output that classifies the medical images into each of a plurality of classes.

[0130] In this example, the patient may be a human or an animal. The region of the patient's body depicted by the medical image may be, for example, an eye, breast, prostate, brain, or tissue throughout the body. The medical image may be a two-dimensional (2D) image (e.g., represented as a 2D array of pixels), a three-dimensional (3D) image (e.g., represented as a 3D array of voxels), or a higher-dimensional image. The medical image may be acquired by a medical image scanner of any modality, such as, for example, an optical coherence tomography (OCT) scanner, a magnetic resonance imaging (MRI) scanner, an X-ray scanner, a computed tomography (CT) scanner, an ultrasound (US) scanner, or a photographic scanner.

[0131] In some implementations, each class of the plurality of classes may represent a particular type and / or variant of a medical condition. For example, each class of the plurality of classes may represent a particular dermatological condition (e.g., eczema, lupus, melasma, etc.). In other examples, the plurality of classes may correspond to various other medical conditions that may be represented by and / or diagnosed based on, for example, chest x-rays, brain computed tomography (CT) scans, fundus images, and / or other types of representations of various anatomical sites.

[0132] In some embodiments, each class of the plurality of classes may represent a respective reference score for each of a plurality of possible clinical referral decisions for the patient. Each reference score may represent a predicted likelihood, conditioned on the segmentation map of the patient's medical images, that the corresponding reference decision is the most appropriate reference decision for the patient. The clinical referral decision may reflect the urgency with which the patient needs further medical attention (e.g., by a specialist). Examples of clinical referral decisions include observation only, routine, semi-urgent, and urgent.

[0133] In some implementations, each class of the plurality of classes may represent a respective condition score for each of a plurality of medical conditions. Each condition score may represent a predicted likelihood that a patient has a medical condition conditioned on a segmentation map of the patient's medical image. For example, if the medical image is an OCT image of the patient's eye, the medical conditions may include, among others, normal, macular retinal edema (MRO), choroidal neovascularization (CNV), and geographic atrophy.

[0134] In some implementations, each class of the plurality of classes may represent a respective progression score for each of a plurality of medical condition states. Each progression score may represent a predicted likelihood that the corresponding medical condition state will progress to the medical condition state at a particular future time conditioned on the segmented map of medical images of the patient. For example, the medical condition states may include a stable state, a slowly progressing state, or a rapidly progressing state.

[0135] In some implementations, each class of the plurality of classes may represent a respective treatment score for each of a plurality of medical treatments. Each treatment score may represent a predicted likelihood of being the best medical care for a patient conditioned on the segmented map of the patient's medical images, or more specifically, the relative best treatment among a set of treatments. For example, the medical treatments may include no treatment, surgery, or the administration of a variety of different medications.

[0136] As another example, the task may be a medical text processing task, where the input is a sequence of medical reports or other sequences derived from electronic health record data, and the output is another sequence of text, for example, a completion of the medical report, responses to questions posed in the medical report, or a summary of the medical report that is shorter than the medical report (but represents the most important or relevant information in the medical report).

[0137] As another example, the task can be a genomics task, where the input is a sequence representing a fragment of a DNA sequence or other molecular sequence, and the output is either an embedding of the fragment for use in a downstream task, e.g., by using unsupervised learning techniques on a dataset of DNA sequence fragments, or the output of a downstream task. Examples of downstream tasks include promoter site prediction, methylation analysis, predicting the functional impact of non-coding variants, etc.

[0138] In some cases, tasks in the medical domain include multimodal processing tasks that require processing of multimodal data, which is generally a combination of two or more different types of data, for example, both image data and text data.

[0139] For example, the task may be a medical visual question answering task that requires generating answers to questions posed about a medical image. As another example, the task may be a medical report generation task that requires generating a text report characterizing a medical image. The medical image may be any one of the exemplary medical images described above.

[0140] The output for any of these tasks in the medical domain can be generated by the language model neural network 530, for example, as a sequence of text tokens or as a probability distribution generated from probabilities assigned to a given set of tokens.

[0141] 5D is an exemplary diagram of the performance of a medical visual question answering task by encoder subsystem 510 and language model neural network 530. As shown, encoder subsystem 510 receives natural language text sequence 552 and observation 554, which includes a chest x-ray image. Natural language text sequence 552 includes a question posed regarding the chest x-ray image included in observation 554: "Can you write a report analyzing this chest x-ray?"

[0142] In response, the encoder subsystem 510 generates a sequence of input tokens including an encoded representation of the text sequence and an encoded representation of the chest x-ray image, and the language model neural network 530 processes the sequence of input tokens to generate an output text sequence 556 that is a response to the question: findings -Device: None. -Lungs: No pneumothorax. No substantial pleural effusion. Lungs appear clean. -Septum: normal cardiac size. Mediastinal contour within normal limits. -Other: No acute skeletal abnormalities. impression: There is no active chest disease.” Generate.

[0143] This specification uses the term "configured" in the context of systems and computer program components. When one or more computer systems are configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when running, causes the system to perform the operation or action. When one or more computer programs are configured to perform a particular operation or action, it means that the one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0144] Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more modules of computer program instructions, i.e., as one or more computer programs encoded on a tangible, non-transitory storage medium for execution by or controlling the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random-access or serial-access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be encoded in an artificially generated propagated signal, such as a mechanically generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a receiving device suitable for execution by the data processing apparatus.

[0145] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or include special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus may optionally include code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0146] A computer program, which may be called or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted, declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, for example, one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple cooperating files, for example, files that store one or more modules, subprograms, or code portions. A computer program can be deployed to be executed on one computer or to be executed on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communications network.

[0147] As used herein, the term "database" is used broadly to refer to any collection of data. The data need not be structured in any particular way, or even at all, and may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed in a different way.

[0148] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines may be installed and run on the same computer or computers.

[0149] The processes and logic flows described herein may be implemented by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by special purpose logic circuitry, such as an FPGA or an ASIC, or a combination of special purpose logic circuitry and one or more programmed computers.

[0150] A computer suitable for running a computer program may be based on a general-purpose or special-purpose microprocessor, or both, or on other types of central processing units. Generally, the central processing unit receives instructions and data from a read-only memory, a random-access memory, or both. The basic components of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Generally, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operatively connected to receive data from or transmit data to them, or both. However, a computer need not have such devices. Furthermore, a computer may be incorporated in other devices, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0151] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0152] To provide for user interaction, embodiments of the subject matter described herein may be implemented in a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices may also be used to interact with the user. For example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. Furthermore, a computer may interact with a user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. A computer may also interact with a user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and then receiving a reply message from the user.

[0153] A data processing device for implementing machine learning models may also include, for example, dedicated hardware accelerator units for handling the general and numerically intensive portions (i.e., inference, workloads) of machine learning training or production.

[0154] The machine learning model can be implemented and deployed using a machine learning framework, for example, the TensorFlow framework or the JAX framework.

[0155] Embodiments of the subject matter described herein may be implemented in a computing system that includes a back-end component, e.g., a data server, or a middleware component, e.g., an application server, or a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an application that enables a user to interact with an embodiment of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), e.g., the Internet.

[0156] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data (e.g., HTML pages) to a user device, for example, to display the data to and receive user input from a user interacting with the device functioning as a client. Data generated at the user device, e.g., results of user interaction, can be received from the device by the server.

[0157] While the specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or the scope of patentable subject matter, but rather as descriptions of features that may be unique to particular embodiments of a particular invention. Certain features described herein as separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented in multiple embodiments, either separately or in any suitable subcombination. Furthermore, even if features may be described above as functioning in a particular combination and originally claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to subcombinations or variations of the subcombination.

[0158] Similarly, while operations are illustrated in the figures and described in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order or sequential order shown, or that all of the operations shown be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0159] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still produce desirable results. As one example, the methods depicted in the accompanying figures do not necessarily require the particular order or sequential order shown to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.< / textn> < / obsn> < / textn> < / obsn> < / obs1> < / text1>

Claims

1. 1. A computer-implemented method comprising: receiving one or more observations of an environment including one or more objects; receiving an input text sequence describing a task to be performed by a robot in the environment and referencing the one or more objects in the environment; generating an encoded representation of the input text sequence in an embedding space; generating a corresponding encoded representation for each of the one or more observations in the embedding space; generating a sequence of input tokens comprising the encoded representation of the input text sequence and the corresponding encoded representation of each observation; processing the sequence of input tokens using a language model neural network to autoregressively generate an output text sequence comprising high-level natural language instructions for the robot; determining from the high-level natural language instructions one or more actions to be performed by the robot; A method comprising:

2. receiving one or more observations of the environment; receiving a first state vector representing one or more of an pose, a size, or a color of an object; receiving a second state vector representing a pose of the robot; Including, The method of claim 1.

3. receiving the one or more observations of the environment; receiving an image characterizing the environment; The method according to any one of claims 1 to 2.

4. generating the corresponding encoded representation of each of the one or more observations; processing each of the one or more observations using an encoder neural network, a projector neural network, or both; The method according to any one of claims 1 to 3.

5. the encoder neural network is configured as an image encoder neural network and the projector neural network is configured as a multi-layer perceptron (MLP) to process each of the one or more observations; processing an input comprising the image using the image encoder neural network to generate an image encoder embedding of the image; processing the image encoder embedding of the image using the MLP to generate an encoded representation of the image; The method of claim 4, comprising:

6. The method of claim 5 , wherein the image encoder neural network is configured as a convolutional neural network, a transformer neural network, or a vision transformer (ViT) neural network.

7. A method according to any one of claims 5 to 6, wherein the input comprises the image and an object instance mask that identifies different parts of the image as each containing a corresponding object.

8. 7. The method of claim 6, wherein the image encoder neural network includes a slot-mixer decoder that computes outputs by applying an attention mechanism across multiple slots, each representing a respective portion of the image, using one or more query rays having predetermined directions.

9. 9. The method of claim 1, wherein the corresponding encoded representation of each observation is interleaved between the encoded representations of the input text sequence within the sequence of input tokens.

10. generating the sequence of input tokens, assigning an index to each of the one or more objects contained within the environment; generating an encoded representation of a prefix that defines a mapping between the one or more objects and the assigned index; generating a sequence of said input tokens including said prefix; The method according to any one of claims 1 to 9, comprising:

11. generating the encoded representation of the input text sequence, determining a respective product of each one-hot encoded vector in the one-hot encoded representation of the input text sequence and an embedding matrix having values ​​trained in collaboration with the language model neural network; The method according to any one of claims 1 to 10.

12. 12. The method of claim 1, wherein the high-level natural language instructions for the robot specify (i) two or more actions to be performed by the robot, and (ii) a sequential order in which the two or more actions should be performed.

13. determining the one or more actions to be performed by the robot; processing the high-level natural language instructions using a policy neural network to generate one or more policy network outputs; selecting the one or more actions to be performed by the robot from a space of possible actions according to the one or more policy network outputs; The method according to any one of claims 1 to 12, comprising:

14. 14. A system comprising one or more computers and one or more storage devices storing instructions, the instructions being operable, when executed by the one or more computers, to cause the one or more computers to perform the operations of each of the methods of any one of claims 1 to 13.

15. 14. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of each of the methods recited in any one of claims 1 to 13.

Images