Training and / or utilizing machine learning models for natural language-based robot control

Abstract

Techniques are disclosed that enable training a goal-conditional policy based on multiple datasets, each of which describes a robotic task in a different manner. For example, the multiple datasets can include: a target image dataset, in which the task is captured in a target image; a natural language instruction dataset, in which the task is described in natural language instructions; a task ID dataset, in which the task is described by a task ID, etc. In various embodiments, each of the multiple datasets has a corresponding encoder, wherein the encoder is trained to generate a shared latent space representation of the corresponding task description. Additional or alternative techniques are disclosed that enable the use of a goal-conditional policy network to control a robot. For example, the goal-conditional policy network can be used to control a robot based on free-form natural language input describing (multiple) robotic tasks.

Description

Background Art

[0001] Many robots are programmed to perform specific tasks. For example, a robot on an assembly line can be programmed to recognize specific objects and perform specific manipulations on those objects.

[0002] Additionally, some robots can perform specific tasks in response to explicit user interface input corresponding to the specific task. For example, a vacuum cleaning robot can perform a general vacuum cleaning task in response to the spoken utterance "Robot, clean." However, typically, user interface input that causes a robot to perform a specific task must be explicitly mapped to the task. As a result, the robot may not be able to perform a specific task in response to various free-form natural language inputs from a user attempting to control the robot. For example, the robot may not be able to navigate to a target location based on free-form natural language input provided by the user. For example, the robot may not be able to navigate to a specific location in response to a user request to "go out, turn left, and go through the door at the end of the hallway." Summary of the Invention

[0003] The technology disclosed herein relates to training a goal-conditional policy network based on multiple datasets, where the training tasks are described differently in each dataset. For example, a robot task can be described using a goal image, using natural language text, using a task ID, using natural language speech, and / or using additional or alternative task descriptions. For example, a robot can be trained to perform the task of putting a ball into a cup. The goal image description of the example task can be a picture of a ball inside a cup, the natural language text description of the task can be the natural language instruction "put the ball into the mug," and the task ID description of the task can be "task id=4," where 4 is the ID associated with the task of putting the ball into the cup. In some embodiments, multiple encoders can be trained (i.e., one encoder per dataset) so that each encoder can generate a shared latent goal space representation of the task by processing the task description. In other words, different ways of describing the same task (e.g., an image of a ball in a cup, a natural language instruction "put the ball into the mug," and / or "task id=4") can be mapped to the same latent goal representation based on processing the task description with the corresponding encoder. While the technology described herein relates to training a robot to perform a task, this is not intended to be limiting. According to the techniques described herein, additional and / or alternative networks can be trained based on multiple datasets, each of which has a different context.

[0004] Additional or alternative embodiments relate to controlling a robot based on outputs generated using a goal-conditional policy network. In some embodiments, the robot can be trained using multiple datasets (e.g., using a target image dataset and a natural language instruction dataset), and only one type of task description can be used to describe the robot's task(s) at inference time (e.g., providing only natural language instructions, only target images, only task IDs, etc. to the system at inference time). For example, the system can be trained based on a target image dataset and a natural language instruction dataset, where natural language instructions are provided to the system to describe the robot's tasks at runtime. Additionally or alternatively, in some embodiments, multiple instruction description types can be provided to the system at runtime (e.g., providing natural language instructions, target images, and task IDs to the system at runtime, providing natural language instructions and task IDs to the system at runtime, etc.). For example, the system can be trained based on a natural language instruction dataset and a target image dataset, where natural language instructions and / or target image instructions can be provided to the system at runtime.

[0005] In some embodiments, a robotic agent can use a goal-conditional policy network to implement task-agnostic control, where a single robot can reach any achievable goal state in its environment. In conventional remote control multi-task demonstrations, the diversity of the data collected can be limited by pre-defined tasks (e.g., providing a list of tasks to be demonstrated to the human operator). In contrast, when generating game data, the human operator in a remote control "game" is not limited to predefined tasks. In some embodiments, a target image dataset can be generated based on the remote control "game" data. The game data can include a continuous log (e.g., a data stream) of low-level observations and actions collected while the human remotely controls the robot and engages in behaviors that satisfy their own curiosity. Unlike collecting expert demonstrations, collecting game data may not require task segmentation, labeling, or resetting to an initial state, thereby enabling rapid and large-scale collection of game data. Additionally or alternatively, game data can be constructed based on human knowledge of object visibility (e.g., if people see a button in a scene, they tend to press it). Human operators can try multiple ways to achieve the same result and / or explore new behaviors. In some embodiments, it can be expected that the game data naturally covers the interaction space of the environment in a way that expert demonstrations may not.

[0006] In some embodiments, a target image dataset can be generated based on remote control game data. A segment of the game data stream (e.g., a sequence of image frames) can be selected as an imitation trajectory, where the last image in the selected segment of the data stream is the target image. In other words, the target image in the target image dataset describing the imitation trajectory can be generated ex post facto, where the target image is determined based on the action sequence, as opposed to generating the action sequence based on the target image. In some embodiments, short-term target image training instances can be quickly and / or inexpensively generated based on the data stream of remote control game data.

[0007] In some embodiments, additionally or alternatively, the natural language instruction dataset can be based on remote control game data. Segments of the game data stream (e.g., sequences of image frames) can be selected as imitation trajectories. One or more humans can then describe the imitation trajectories, thereby generating natural language instructions ex post facto (in contrast to generating the imitation trajectories based on natural language instructions). In some embodiments, the collected natural language instructions can include functional behaviors (e.g., "open the drawer," "press the green button," etc.), general non-task-specific behaviors (e.g., "move your hand slightly to the left," "do nothing," etc.), and / or other behaviors. In some embodiments, the natural language instructions can be free-form natural language, without any constraints on the content that can be provided. In some embodiments, multiple humans can use free-form natural language to describe the imitation trajectories, which can result in different descriptions of the same object(s), behavior(s), etc. For example, a simulation trajectory can capture a robot picking up a wrench. Multiple human describers can provide different free-form natural language instructions for the simulation trajectory, such as "grab the tool," "pick up the wrench," "grab the object," and / or additional free-form natural language instructions. In some embodiments, this diversity in free-form natural language instructions can lead to more robust goal-conditional policy networks, where a wide range of free-form natural language instructions can be implemented by the agent.

[0008] The target-conditional policy network and corresponding encoder can be trained in various ways based on the image-target dataset and the free-form natural language instruction dataset. For example, the system can use the target image encoder to process the target image portion of a target image training instance to generate a latent target space representation of the target image. The latent target space representation of the target image and the initial frame of the imitation trajectory portion of the target image training instance can be used to generate a target image candidate output. A target image loss can be generated based on the target image candidate output and the target image imitation trajectory. Similarly, the system can process the natural language instruction portion of a natural language instruction training instance to generate a latent space representation of the natural language instruction. The target-conditional policy network can be used to process the natural language instruction and the initial frame of the imitation trajectory portion of the natural language instruction training instance to generate a natural language instruction candidate output. A natural language instruction loss can be generated based on the natural language instruction candidate output and the imitation trajectory portion of the natural language instruction training instance. In some embodiments, the system can generate a target-conditional loss based on the target image loss and the natural language instruction loss. One or more components of the target-conditional policy network, the target image encoder, and / or the natural language instruction encoder can be updated based on the target-conditional loss. However, this is merely an example of training a goal-conditional policy network, a goal image encoder, and / or a natural language instruction encoder. Additional and / or alternative training methods can be used.

[0009] In some embodiments, the target-conditional policy network can be trained using target image datasets and natural language instruction datasets of different sizes. For example, the target-conditional policy network can be trained based on a first number of target image training examples and a second number of natural language instruction training examples, where the second number is 50 percent of the first number, less than 50 percent of the first number, less than 10 percent of the first number, less than 5 percent of the first number, less than 1 percent of the first number, and / or additional or alternative percentages greater than or less than the first number.

[0010] Therefore, various embodiments describe techniques for learning a shared potential target space for many task descriptions for training a single target-conditional policy network. In contrast, conventional techniques train multiple policy networks, one for each task description type. Training a single policy network can allow for the utilization of more data when training the network. Additionally or alternatively, a large number of training instances of a data type can be used to train the policy network. For example, a target-conditional policy network can be trained using post-hoc target image training instances, which can be automatically generated from an imitation learning data stream (e.g., when compared to natural language instruction training instances that may require natural language instructions provided by humans, the automatic generation of post-hoc target image training instances is cheap). By using both a target image dataset and a natural language instruction dataset to train a target-conditional policy network, wherein the majority of the training instances are automatically generated target image training instances, the resulting target-conditional policy network can be robust when generating the robot's actions based on natural language instructions, without requiring computational resources (e.g., processor cycles, memory, power, etc.) and / or human resources (e.g., the time required for a group of people to provide natural language instructions, etc.) to generate a large natural language instruction dataset.

[0011] The above description is provided only as an overview of some of the embodiments disclosed herein. These and other embodiments of the present technology are disclosed below with additional details.

[0012] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are contemplated as part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are contemplated as part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 An example environment is illustrated in which implementations described herein may be implemented.

[0014] Figure 2 Illustrated is an example of using a goal-conditional policy network to generate action outputs according to various embodiments described herein.

[0015] Figure 3 is a flow chart illustrating an example process for controlling a robot based on natural language instructions according to various embodiments disclosed herein.

[0016] Figure 4 is a flow chart illustrating an example process for generating target image training instance(s) according to various embodiments disclosed herein.

[0017] Figure 5is a flow chart illustrating an example process for generating natural language instruction training instance(s) according to various implementations disclosed herein.

[0018] Figure 6 is a flow diagram illustrating an example process for training a target-conditional policy network, a natural language instruction encoder, and / or a target image encoder according to various embodiments disclosed herein.

[0019] Figure 7 An example architecture of a robot is schematically depicted.

[0020] Figure 8 An example architecture of a computer system is schematically depicted. DETAILED DESCRIPTION

[0021] Natural language is a universal and intuitive way for humans to communicate tasks to robots. Existing approaches enable learning a wide range of robotic behaviors from general-purpose sensors. However, each task must be specified with a target image—something impractical in open environments. Embodiments disclosed herein provide a simple and / or scalable approach to conditioning policies in place of human language. Brief robotic experiences in games can be paired with relevant human language after the fact. To achieve this efficiency, some embodiments utilize multi-context imitation, which allows a single agent to be trained to follow either image or language goals, using only language conditioning at test time. This can reduce the cost of language pairing to less than a fraction of the collected robotic experience (e.g., less than 10%, 5%, or 1%), with most control still learned via self-supervised imitation. At test time, a single agent trained in this manner can continuously perform many different robotic manipulation skills in a 3D environment, directly from images and specified only with natural language (e.g., "Open the drawer... now pick up the block... now press the green button..."). Additionally, some embodiments utilize techniques for transferring knowledge from large unlabeled text corpora to robotic learning. This transfer can significantly improve downstream robotic operations. For example, this could also allow the agent to quickly follow thousands of novel instructions in multiple different languages during testing.

[0022] A long-standing motivation for robotic learning has been the concept of a generalist robot—a single agent capable of solving many tasks in everyday settings using only general-purpose onboard sensors. Along with task and observation space universality, a fundamental but less considered aspect is universal task specification: the ability for an untrained user to guide the agent's behavior using the most intuitive and flexible mechanisms. Along these lines, it's difficult to imagine a truly generalist robot without also imagining one capable of following instructions expressed in natural language.

[0023] More broadly, children learn language in the context of rich, relevant sensorimotor experiences. This motivates the long-standing question of embodied language acquisition in artificial intelligence: How can intelligent agents ground language understanding in their own embodied perception? The ability to relate language to the physical world potentially allows robots and humans to communicate on a common basis of shared sensory experience—something that could lead to more meaningful forms of human-robot interaction.

[0024] Furthermore, language acquisition (at least in humans) can be a highly social process. During their earliest interactions, infants contribute actions and caregivers contribute related words. While the actual learning mechanisms at work in humans are not fully understood, the embodiments disclosed herein explore what robots can learn from similar paired data.

[0025] However, even simple instruction following can pose notoriously difficult learning challenges, encompassing many long-standing problems in AI. For example, a robot presented with the command “sweep the blocks into the drawer” must be able to relate language to low-level perception (what do the blocks look like? What is a drawer?). It must perform visual reasoning (what does it mean for the blocks to be in the drawer?). Additionally, it must solve complex sequential decision-making problems (what command do I send to my arm to “sweep”?). It’s worth noting that these problems only cover a single task, whereas generalist robotics require a single agent that can perform many tasks.

[0026] In some embodiments, the setting of open robotic manipulation can be combined with open human language regulation. Prior art typically includes constrained observation spaces (e.g., games, 2D grid worlds, simplified actuators (e.g., binary pick and place primitives, and synthetic language data. Embodiments herein involve a combination of 1) human language instructions, 2) high-dimensional continuous sensory input and actuators, and / or 3) complex tasks (e.g., long-term robotic object manipulation). At test time, in some embodiments a single agent can be considered that can perform many tasks in succession, where each task can be specified by a person in natural language. For example, "always open the door to the right... now pick up the block... now press the red button... now close the door." Furthermore, the agent should be able to perform any combination of subtasks in any order. This can be called an "ask me anything" scenario, which can test aspects of generality, such as general control, learning from onboard sensors, and / or general task specifications.

[0027] Existing techniques can provide a starting point for learning general skills on-machine. However, like other methods that combine relabeling with image observation, existing techniques require specifying the task to be achieved using a target image. While trivial in simulators, this form of task specification can be impractical in open environments.

[0028] In some implementations, the system is able to extend existing techniques to natural language settings by:

[0029] (1) Covering the space with remote control games. In some embodiments, the system can collect remote control "game" data sets. These long state action logs can be (automatically) re-labeled into many short-term presentations to solve the image target.

[0030] (2) Pairing games with human language. Existing techniques typically pair instructions with optimal behaviors. In contrast, in some embodiments described herein, behaviors from games can be paired with optimal instructions post-facto (i.e., post-instruction pairing). This can generate demonstration datasets that address human language objectives.

[0031] (3) Multi-context imitation learning. In some embodiments, a single policy can be trained to solve image and / or language goals. Additionally or alternatively, in some embodiments, only language conditioning is used at test time. To enable this, the system can utilize multi-context imitation learning. Multi-context imitation learning can be highly data efficient. For example, this reduces the cost of language pairing to less than a small fraction of the collected robot experience (e.g., less than 10%, less than 5%, less than 1%) to achieve language conditioning, with most control still learned via self-supervised imitation.

[0032] (4) Conditioning on Human Language at Test Time In some embodiments, at test time, a single policy trained in this manner is able to perform many complex robotic manipulation skills continuously, directly from images and specified entirely in natural language.

[0033] Additionally or alternatively, some embodiments include transferring learning from an unlabeled text corpus to robotic manipulation. This can be enhanced using transfer learning, which can be applied to any language-conditional policy. In some embodiments, this can improve downstream robotic manipulation. Importantly, this technique can allow an agent to quickly follow novel instructions (e.g., follow thousands of novel instructions and / or follow instructions across multiple languages). Goal-conditional learning can be used to train a single agent to achieve any goal. This can be formalized as a goal-conditional policy π θ (a|s, g), which outputs the next action a∈A and the task descriptor g∈G conditioned on the current state s∈S. The imitation approach can use a dataset of expert state-action trajectories τ={(s0, a0), ...} This mapping can be learned by supervised learning to solve paired task descriptors (such as one-hot task encodings). A convenient choice of task descriptor can be some target state g = s g∈S. This allows any state visited during acquisition to be relabeled as a “reached goal state”, where the previous state and action are considered the best behavior for reaching that goal. Applied to some original dataset D, this can produce a much larger dataset of relabeled examples This provides input to a simple maximum likelihood purpose for goal-directed control: relabeled goal-conditioned behavior cloning (GCBC):

[0034]

[0035] While relabeling can automatically generate a large number of goal-oriented demonstrations during training, it may fail to account for the diversity of these demonstrations, which can be derived entirely from the underlying data. To be able to achieve any user-provided goal, we incentivize data collection methods upstream of relabeling that fully cover the state space.

[0036] Human teleoperated "game" acquisition directly addresses the state space coverage problem. In this setting, the operator is no longer restricted to a set of predefined tasks, but can participate in the manipulation of every available object in the scene. The motivation is to use the prior human knowledge of object visibility to fully cover the state space. During acquisition, the onboard robot observation and action streams are recorded. This results in an unsegmented dataset of unstructured but semantically meaningful behaviors, which can be useful in relabeling imitation learning settings.

[0037] Learning from games can combine relabeled imitation learning with remote control games. First, use Algorithm 2 to relabel the unsegmented game logs. This can generate a training set It has many different short-term examples. In some embodiments, these can be fed into a standard maximum likelihood objective conditional modeling object:

[0038]

[0039] A limitation of learning from games—and other approaches that combine relabeling with image state spaces—is that at test time the behavior must be in the form of a target image s. gSome embodiments described herein can focus on a more flexible mode of regulation: humans describing tasks in natural language. Doing this can require solving complex foundational problems. To address this, post hoc instruction pairing can be used, which is a method for pairing large amounts of diverse robot sensor data with relevant human language. In some embodiments, to leverage both image target and language target datasets, multi-context imitation learning can be used. Additionally or alternatively, language learned from games (LangLfP) can be used, which ties these components together to learn a single policy that follows many human instructions over a long period of time.

[0040] From a statistical machine learning perspective, a candidate for embedding human language into robot sensor data is a large corpus of robot sensor data paired with relevant language. One way to collect this data is to select instructions and then collect the best behavior. Additionally or alternatively, some embodiments can sample any robot behavior from the game and then collect the best instructions, which can be called post-hoc instruction pairing (Algorithm 3). Just as post-goal images are a post-facto answer to the question "Which goal state makes this trajectory best?", post-facto instructions are a post-facto answer to the question "Which language instructions make this trajectory best?" In some embodiments, these pairs can be obtained by showing humans onboard robot sensor video and then asking them "What instructions would you give the agent to get from the first frame to the last frame?"

[0041] The post-command pairing process can assume access to D which can be obtained using Algorithm 2 play and a group of non-professional human supervisors. play In the , you can create a new dataset It consists of a short-term game sequence τ paired with l∈L (which are human-provided post hoc instructions with no constraints on vocabulary and / or grammar).

[0042] In some embodiments, this process can be scalable because pairing occurs after the fact, thereby making parallelization become straightforward (e.g., via crowdsourcing). The language collected can also be naturally rich, because it is located above the game and is similarly not subject to pre-task definition constraints. This can produce instructions (e.g., "open the drawer," "press the green button") and general non-task specific behaviors (e.g., "move your hand slightly to the left" or "do nothing") of functional behavior. In some embodiments, it may not be necessary to pair the language of each experience from the game with learning to follow instructions. It can be made possible by multi-situational imitation learning described herein.

[0043] So far, we have described a method to create two context imitation datasets: D with examples of post hoc target images and play and D with examples of post-instructions (play，lang) In some embodiments, a single policy can be trained that is agnostic to any one task description. This can allow for sharing statistical power across multiple datasets during training and / or can allow for using only the language specification at test time.

[0044] Based on this motivation, some embodiments use Multi-Context Imitation Learning (MCIL), which is a simple and / or widely applicable generalization of context imitation for multiple heterogeneous contexts. The main idea is to represent a large set of policies by a single unified function approximator that can generalize over states, tasks, and / or task descriptions. MCIL can assume access to multiple imitation learning datasets D = {D 0 ,...,D K}, the imitation learning datasets have different ways of describing tasks. In some embodiments, each There is a state-action trajectory pair τ paired with a context c∈C. For example, D 0 May contain demonstrations paired with one-hot task ids (regular multi-task imitation learning datasets), D 1 May contain image target presentation, and D 2 May include language target demonstrations.

[0045] Instead of training a policy per dataset, MCIL trains a single latent goal-conditional policy π on all datasets simultaneously. θ (a t |s t , z), thereby learning to map each task description type to the same latent target space This latent space can be viewed as a common abstract target representation shared across many imitation learning problems. To make this possible, MCIL can assume a parameterized encoder set Each dataset has a parameterized encoder, and the set of parameterized encoders is responsible for mapping a specific type of task description to a common latent target space, i.e., For example, these may be a task id embedding lookup, an image encoder, a language encoder, one or more additional or alternative values, and / or combinations thereof, respectively.

[0046] In some embodiments, MCIL has a simple training process: at each training step, for each dataset D in D k , for a small batch of trajectory context pairs (τ k , c k )~D kSampling is performed in the potential target space The situation is encoded in , and then a simple maximum likelihood situation imitation objective is calculated:

[0047]

[0048] The full MCIL objective is to average the per-dataset objective over all datasets at each training step.

[0049]

[0050] The policy and all target encoders are trained end-to-end to Maximize. See Algorithm 1 for the complete mini-batch training pseudocode.

[0051] In some embodiments, multi-context learning has properties that make it more useful beyond learning from games. Although the dataset D can be set as D = {D play , D (play，lang)}, but this approach can be more generally used to train on any collection of imitation datasets with different descriptions (e.g., task ids, languages, human video demonstrations, speech, etc.). Being context-agnostic enables an efficient training scheme: most of the control is learned from the cheapest data source, while the most general forms of task regulation are learned from a small number of labeled examples. In this way, multi-context learning can be understood as transfer learning via a shared target space. This can reduce the cost of human supervision to a level that is practical for practical applications. Multi-context learning can allow an agent to be trained to follow human instructions, where a small fraction (e.g., less than 10%, less than 5%, less than 1%, etc.) of the collected robot experience requires paired language, while most of the control is learned from re-labeled target image data.

[0052] In some embodiments, Language Conditioned Learning from Games (LangLfP) is a special case of multi-context imitation learning. At a high level, LangLfP is trained on a dataset D consisting of a post-hoc target image task and a post-hoc instruction task. play , D (play，lang)} to train a single multi-context policy π θ (a t |s t , z). In some embodiments, It can be a neural network encoder that maps from image targets and commands to the same latent visual-language target space respectively. LangLfP can learn perception, natural language understanding, and end-to-end control without auxiliary losses.

[0053] Perception module. In some embodiments, τ in each example is represented by the onboard observation O t and sequence of actions Each observation can contain a high-dimensional image and / or internal proprioceptive sensor readings. The learned perception module P θ Map each observation tuple to a low-dimensional embedding, e.g., s t =P θ (O t ), which is fed to the rest of the network. This perception module can be used with g enc Share, g enc An additional network is defined on top to transform the encoded target observation s g Mapped to a point in z space.

[0054] Language module. In some embodiments, the language target encoder S enc Tokenize the original text l into sub-words, retrieve the sub-word embeddings from a lookup table, and / or summarize the embeddings as points in z-space. The sub-word embeddings can be randomly initialized at the beginning of training and learned end-to-end through the final imitation loss.

[0055] Control module. Many architectures can be used to implement multi-context strategies θ (a t |s t , z). For example, latent motor planning (LMP) can be used. LMP is a goal-oriented imitation architecture that uses latent variables to model the large amount of multimodality inherent in free-form imitation datasets. Specifically, it can be a sequence-to-sequence conditional variation autoencoder (seq2seq CVAE), which automatically encodes contextual demonstrations through a latent "plan" space. The decoder is a goal-conditional policy. As a CVAE, LMP defines a lower bound for maximum likelihood contextual imitation and can be easily adapted to multi-context settings.

[0056] LangLfP training. LangLfP training can be compared with the existing LfP training. At each training step, it is possible to play A batch of image target tasks are sampled from D (play，lang) A batch of language target tasks are sampled in the perception module P θ Encode the observation into the state space. The encoder g can be used enc and s enc Encode the image and language targets into the latent target space z. Policy π θ (a t |s t, z) can be used for multi-context imitation purposes to compute the average over the two task descriptions. In some embodiments, a combined gradient step can be taken with respect to all modules (perception, language, and control) that optimize the entire architecture end-to-end as a single neural network.

[0057] Follow human instructions during testing. At the beginning of the test scenario, the agent receives its onboard observation O t and human-specified natural language target l as input. The agent uses the trained sentence encoder s enc Encodes l in the latent goal space z. The agent then solves the goal in a closed loop, repeatedly feeding the current observation and goal back to the learned policy π θ (a t |s t ,z), samples actions and executes them in the environment. A human operator can type in a new language target l at any time.

[0058] Large "wild" natural language corpora reflect a wealth of human knowledge about the world. Many recent works have successfully transferred this knowledge to downstream tasks in NLP via pre-trained embeddings. In some embodiments described herein, can similar knowledge transfer be achieved for robotic manipulation?

[0059] There are many benefits to this type of transfer. First, if there is a semantic match between the source corpus and the target environment, the more structured input can act as a strong prior to shape the basis or control. Additionally or alternatively, language embeddings have been shown to encode the similarities between a large number of words and sentences. This can allow the agent to quickly follow many novel instructions, as long as the instructions are sufficiently "close" to the instructions the agent was trained to follow. It should be noted that given the complexity of natural language, robots in open environments will likely need to be able to follow synonymous commands outside of the specific training set.

[0060] Algorithm 1 Multi-scenario imitation learning

[0061]

[0062]

[0063] Ends when:

[0064] Algorithm 2 creates millions of target image-conditional imitation examples from the remote control game.

[0065]

[0066] Ends when:

[0067] Algorithm 3 pairs robot sensor data with natural language instructions.

[0068]

[0069]

[0070] Now turning to the diagram, Figure 1 An example robot 100 is illustrated in FIG. The robot 100 is a "robot arm" having multiple degrees of freedom to enable traversal of a gripping end effector 102 along any one of multiple potential paths to position the gripping end effector 102 at a desired location. The robot 100 further controls two opposing "jaws" of its gripping end effector 102 to actuate the jaws between at least one open position and a closed position (and / or optionally, multiple "partially closed" positions).

[0071] Figure 1 An example visual component 106 is also illustrated. Figure 1 In the embodiment of the present invention, the vision assembly 106 is mounted in a fixed position relative to the base or other stable reference point of the robot 100. The vision assembly 106 includes one or more sensors that can generate images and / or other visual data related to the shape, color, depth and / or other characteristics of (multiple) objects within the sensor's line of sight. The vision assembly 106 can be, for example, a thematic camera, a stereo camera and / or a 3D laser scanner. The 3D laser scanner can be, for example, a time-of-flight 3D laser scanner or a triangulation-based 3D laser scanner and can include a position sensitive detector (PDS) or other optical position sensor.

[0072] The vision component 106 has a field of view of at least a portion of the workspace of the robot 100, such as a portion of the workspace that includes the example object 104. Figure 1 The resting surface(s) of the objects 104 are not shown, but the objects may rest on a table, a tray, and / or other surface(s). The objects 104 may include a spatula, a stapler, and a pencil. In other embodiments, more objects, fewer objects, additional objects, and / or alternative objects may be provided during all or part of a grasping attempt by the robot 100, as described herein.

[0073] Despite Figure 1 A particular robot 100 is illustrated in FIG, but additional and / or alternative robots may be utilized, including additional robotic arms similar to robot 100, robots having other robotic arm forms, robots having humanoid forms, robots having animal forms, robots that move via one or more wheels (e.g., self-balancing robots), submersible vehicle robots, unmanned aerial vehicles (“UAVs”), etc. Furthermore, although in Figure 1 A specific gripping end effector is illustrated in FIG, but additional and / or alternative end effects may be utilized, such as alternative impact gripping end effectors (e.g., impact gripping end effectors with gripping "plates," impact gripping end effectors with more or less "fingers" / "claws"), entry gripping end effectors, retracting gripping end effectors, proximity gripping end effectors, or non-grasping end effectors. Additionally, although in Figure 1 Specific accessories for the vision assembly 106 are illustrated in FIG, but additional and / or alternative accessories may be utilized. For example, in some embodiments, the vision assembly may be mounted directly to the robot, such as on a non-actuable component of the robot or on an actuable component of the robot (e.g., on an end effector or on a component proximate to an end effector). Furthermore, for example, in some embodiments, the vision assembly may be mounted on a non-stationary structure separate from the robot with which it is associated and / or may be mounted in a non-stationary manner on a structure separate from the robot with which it is associated.

[0074] Data from the robot 100 (e.g., visual data captured using the vision component 106) along with natural language instruction(s) 130 captured using the user interface input device(s) 128 can be used by the action output engine 108 to generate action outputs. In some embodiments, the robot 100 can be controlled (e.g., one or more actuators of the robot 100 can be controlled) to perform one or more actions based on the action outputs. In some embodiments, the user interface input device(s) 128 can include, for example, a physical keyboard, a touch screen (e.g., implementing a virtual keyboard or other text input mechanism), a microphone, and / or a camera. In some embodiments, the natural language instruction(s) 130 can be free-form natural language instruction(s).

[0075] In some implementations, the potential goal engine 110 can process the natural language instruction 130 using the natural language instruction encoder 114 to generate a latent state representation of the natural language instruction. For example, the keyboard user interface input device 128 can capture the natural language instruction "press the green button." The potential goal engine 110 can process the natural language instruction 130 "press the green button" using the natural language instruction encoder 114 to generate a latent goal representation of "press the green button."

[0076] In some embodiments, the target image training instance engine 126 can be used to generate (multiple) target image training instance 124 based on the remote control "game" data 122. The remote control "game" data 122 can be generated by a human controlling a robot in an environment in which the human controller has not defined a task to be performed. In some embodiments, each target image training instance 124 can include an imitation trajectory portion and a target image portion, wherein the target image portion describes the task, the robot task. For example, the target image can be an image of a closed drawer, which can describe the (multiple) robot actions of closing the drawer. As another example, the target image can be an image of an open drawer, which can describe the (multiple) robot actions of opening a door. In some embodiments, the target image training instance engine 126 can select an image frame sequence from the remote control game data stream. The target image training instance engine 126 can generate one or more target instance training instances by storing the selected image frame sequence as the imitation trajectory portion of the training instance and storing the last image frame in the image frame sequence as the target image portion of the training instance. In some embodiments, the target image training instance (s) 124 can be generated according to the methods described herein. Figure 4 The process 400 is generated.

[0077] In some embodiments, the natural language instruction training instance engine 120 can be used to generate (multiple) natural language training instances 118 using the remote control game data 122. The natural language instruction training instance engine 120 can select an image frame sequence from the data stream of the remote control game data 122. In some embodiments, a human describer can provide natural language instructions that describe a task performed by the robot in the selected image frame sequence. In some embodiments, multiple human describers can provide natural language instructions that describe a task performed by the robot in the same selected image frame sequence. Additionally or alternatively, multiple human describers can provide natural language instructions that describe tasks performed in different image frame sequences. In some embodiments, multiple human describers can provide natural language instructions in parallel. The natural language instruction training instance engine 120 can generate one or more natural language instruction training instances by storing the selected image frame sequence as the imitation trajectory portion of the training instance and storing the natural language instructions provided by the human as the natural language instruction portion of the training instance. In some embodiments, the (multiple) natural language training instance 124 can be generated according to the method described herein. Figure 5 The process 500 is generated.

[0078] In some embodiments, the training engine 116 can be used to train the target-conditional policy network 112, the natural language instruction encoder 114, and / or the target image encoder 132. In some embodiments, the target-conditional policy network 112, the natural language instruction encoder 114, and / or the target image encoder 132 can be trained according to the Figure 6 The training is performed in process 600.

[0079] Figure 2 An example of generating an action output 208 according to various embodiments is illustrated. Example 200 includes receiving a natural language instruction input 202 (e.g., via Figure 1 The natural language instruction input 202 can be received by one or more user interface input devices 128. In some embodiments, the natural language instruction input 202 can be a free-form natural language input. In some embodiments, the natural language instruction input 202 can be a textual natural language input. The natural language instruction encoder 114 can process the natural language instruction input 202 to generate a potential target space representation of the natural language instruction 204. The target condition policy network 112 can be used to process the potential target 204 together with the current instance of the visual data 206 (e.g., via Figure 1 ) to generate action output 208. In some embodiments, the action output 208 can describe one or more actions for the robot to perform to perform the task indicated by the natural language instruction input 202. In some embodiments, the robot can be controlled based on the action output 208 (e.g., Figure 1 One or more actuators of the robot 100) are used for the robot to perform a task indicated by the natural language instruction input 202.

[0080] Figure 3 is a flow chart illustrating a process 300 for using a goal-conditional policy network to generate outputs when controlling a robot based on natural language instructions, according to embodiments described herein. For convenience, the operations of the flow chart are described with reference to a system in which the operations are performed. The system may include various components of various computer systems, such as robot 100, robot 725, and / or one or more components of computing system 810. Furthermore, while the operations of process 300 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0081] At block 302, the system receives a natural language instruction describing a task for the robot. For example, the system can receive a natural language instruction of "press the red button," "close the door," "pick up the screwdriver," and / or additional or alternative natural language instructions describing a task to be performed by the robot.

[0082] At block 304 , the system processes the natural language instructions using a natural language encoder to generate a latent space representation of the natural language instructions.

[0083] At block 306 , the system receives an instance of visual data capturing at least a portion of the robot's environment.

[0084] At block 308 , the system generates an output based on processing the potential goal representation of at least (a) the visual data instance and (b) the natural language instruction using the goal-conditional policy network.

[0085] At block 310 , the system controls one or more actuators of the robot based on the generated output.

[0086] Controlling a robot based on natural language instructions describes Figure 3 In an additional or alternative embodiment, the system can control the robot based on a target image, a task ID, speech, etc., instead of or in addition to natural language instructions. For example, the system can control the robot based on natural language instructions and target image instructions, where the natural language instructions are processed using a corresponding natural language instruction encoder and the target image is processed using a corresponding target image encoder.

[0087] Figure 4 4 is a flow chart illustrating a process 400 for generating (a plurality of) target image training examples according to embodiments disclosed herein. For convenience, the operations of the flow chart are described with reference to a system in which the operations are performed. The system may include various components of various computer systems, such as the robot 100, the robot 725, and / or one or more components of the computing system 810. Furthermore, while the operations of the process 400 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0088] At block 402, the system receives a data stream that captures remote gaming data.

[0089] The system selects a sequence of image frames from the data stream at block 404. For example, the system can select a one-second sequence of image frames in the data stream, a two-second sequence of image frames in the data stream, a ten-second sequence of image frames in the data stream, and / or select image frame segments of additional or alternative lengths in the data stream.

[0090] At block 406 , the system determines a final image frame in the selected sequence of image frames.

[0091] At block 408, the system stores the training instance, which includes (1) the sequence of image frames that are the simulated trajectory portion of the training instance and (2) the final image frame that is the target image portion of the training instance. That is, the system stores the final image as the target image that describes the task captured in the sequence of image frames.

[0092] At block 410, the system determines whether to generate additional training examples. In some embodiments, the system can determine to generate additional training examples until one or more conditions are met. For example, the system can continue to generate training examples until a threshold number of training examples are generated, until the entire data stream has been processed, and / or until additional or alternative conditions are met. If the system determines to generate additional training examples, the system returns to block 404, selects an additional image frame sequence from the data stream, and performs additional iterations of blocks 406 and 408 based on the additional image frame sequence. If not, the process ends.

[0093] Figure 5 is a flow chart illustrating a process 500 for generating (multiple) natural language instruction training examples according to embodiments disclosed herein. For convenience, the operations of the flow chart are described with reference to a system in which the operations are performed. The system may include various components of various computer systems, such as robot 100, robot 725, and / or one or more components of computing system 810. Furthermore, while the operations of process 500 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0094] At block 502, the system receives a data stream capturing remote gaming data.

[0095] At block 504, the system selects a sequence of image frames from the data stream. For example, the system can select a one-second sequence of image frames in the data stream, a two-second sequence of image frames in the data stream, a ten-second sequence of image frames in the data stream, and / or select image frame segments of additional or alternative lengths in the data stream.

[0096] At block 506 , the system receives natural language instructions describing a task in the selected sequence of image frames.

[0097] At block 508 , the system stores a training instance that includes (1) a sequence of image frames that is the imitation trajectory portion of the training instance and (2) a received natural language instruction that describes the task as the natural language instruction portion of the training instance.

[0098] At block 510, the system determines whether to generate additional training examples. In some embodiments, the system can determine to generate additional training examples until one or more conditions are met. For example, the system can continue generating training examples until a threshold number of training examples are generated, until the entire data stream has been processed, and / or until additional or alternative conditions are met. If the system determines to generate additional training examples, the system returns to block 504, selects an additional image frame sequence from the data stream, and performs additional iterations of blocks 506 and 508 based on the additional image frame sequence. If not, the process ends.

[0099] Figure 6 6 is a flow chart illustrating a process 600 for training a target-conditional policy network, a natural language instruction encoder, and / or a target image encoder according to embodiments disclosed herein. For convenience, the operations of the flow chart are described with reference to a system in which the operations are performed. The system may include various components of various computer systems, such as robot 100, robot 725, and / or one or more components of computing system 810. Furthermore, while the operations of process 600 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0100] At block 602 , the system selects a target image training instance that includes (1) an imitation trajectory and (2) a target image.

[0101] At block 604 , the system processes the target image using a target image encoder to generate a latent target space representation of the target image.

[0102] At block 606 , the system uses a goal-conditional policy network to process at least (1) an initial image frame of the imitated trajectory and (2) a latent space representation of the target image to generate a candidate output.

[0103] At block 608 , the system determines a target image loss based on (1) the candidate output and (2) at least a portion of the simulated trajectory.

[0104] At block 610 , the system selects a natural language instruction training instance that includes (1) an additional imitation trajectory and (2) a natural language instruction.

[0105] At block 612 , the system processes the natural language instruction portion of the natural language instruction training instance using a natural language encoder to generate a latent space representation of the natural language instruction.

[0106] At block 614 , the system uses the goal-conditional policy network to process (1) the initial image frame of the additional imitation trajectory and (2) the latent space representation of the natural language instruction to generate additional candidate outputs.

[0107] At block 616 , the system determines a natural language loss based on (1) the additional candidate output and (2) at least a portion of the additional imitation trajectory.

[0108] At block 618 , the system generates a target-conditional loss based on (1) the image target loss and (2) the natural language instruction loss.

[0109] At block 620 , the system updates one or more portions of the goal-conditional policy network, the goal image encoder, and / or the natural language command encoder based on the goal-conditional loss.

[0110] At block 622, the system determines whether to perform additional training on the target-conditional policy network, the target image encoder, and / or the natural language instruction encoder. In some embodiments, the system can determine that more training is necessary if one or more additional unprocessed training examples exist and / or if one or more other criteria have not been met. The one or more other criteria can include, for example, whether a threshold number of epochs has occurred and / or whether a threshold training duration has occurred. Process 600 can utilize non-batch learning techniques, batch learning techniques, and / or additional or alternative techniques for training. If the system determines that additional training is necessary, the system returns to block 602, selects an additional target image training example, performs additional iterations of blocks 604, 606, and 608 based on the additional target image training example, selects an additional natural language instruction training example at block 610, performs additional iterations of blocks 612, 614, and 616 based on the additional natural language instruction training example, and performs additional iterations of blocks 618 and 610 based on the additional target image training example and the additional natural language instruction training example. If not, the process ends.

[0111] Figure 7 An example architecture of a robot 725 is schematically depicted. The robot 725 includes a robot control system 760, one or more operating components 740a to 740n, and one or more sensors 742a to 742m. The sensors 742a to 742m may include, for example, a vision component, a light sensor, a pressure sensor, a pressure wave sensor (e.g., a microphone), a proximity sensor, an accelerometer, a gyroscope, a thermometer, a barometer, and the like. While the sensors 742a-m are depicted as being integrally formed with the robot 725, this is not intended to be limiting. In some embodiments, the sensors 742a-m may be located external to the robot 725, such as as a stand-alone unit.

[0112] The operating components 740a to 740n may include, for example, one or more end effectors and / or one or more servo motors or other actuators to achieve movement of one or more components of the robot. For example, the robot 725 may have multiple degrees of freedom, and each of the actuators may control actuation of the robot 725 within one or more of the degrees of freedom in response to control commands. As used herein, the term actuator also encompasses a mechanical or electrical device (e.g., a motor) that produces motion, in addition to any driver that may be associated with an actuator and converts a received control command into one or more signals for driving the actuator. Thus, providing a control command to an actuator may include providing the control command to a driver that converts the control command into an appropriate signal for driving the electrical or mechanical device to produce the desired motion.

[0113] The robotic control system 760 can be implemented in one or more processors, such as a CPU, GPU, and / or other controller of the robot 725. In some embodiments, the robot 725 can include a "brain box" that can include all or multiple aspects of the control system 760. For example, the brain box can provide real-time bursts of data to the operating components 740a-n, where each of the real-time bursts includes a set of one or more control commands that, among other things, indicate motion parameters (if any) for each of one or more of the operating components 740a-n. In some embodiments, the robotic control system 760 can perform one or more aspects of processes 300, 400, 500, 600, and / or other methods described herein.

[0114] As described herein, in some embodiments, all or multiple aspects of the control commands generated by the control system 760 when positioning an end effector to grasp an object can be based on end effector commands generated using a goal-condition policy network. For example, the vision components of the sensors 742a-m can capture environmental state data. The policy network of the meta-learning model can be used to process this environmental state data along with the robot state data to generate one or more end effector control commands for controlling movement and / or grasping of the end effector of the robot. Although in Figure 7 760 is illustrated as an integral part of the robot 725, but in some embodiments, all or multiple aspects of the control system 760 may be implemented in a component that is separate from, but in communication with, the robot 725. For example, all or multiple aspects of the control system 760 may be implemented on one or more computing devices (such as the computing device 810) that are in wired and / or wireless communication with the robot 725.

[0115] Figure 88 is a block diagram of an example computing device 810 that can optionally be used to perform one or more aspects of the techniques described herein. The computing device 810 generally includes at least one processor 814 that communicates with a number of peripheral devices via a bus subsystem 812. These peripheral devices may include a storage subsystem 824, which includes, for example, a memory subsystem 825 and a file storage subsystem 826, a user interface output device 820, a user interface input device 822, and a network interface subsystem 816. The input and output devices allow for user interaction with the computing device 810. The network interface subsystem 816 provides an interface to an external network and couples to corresponding interface devices in other computing devices.

[0116] The user interface input device 822 may include a keyboard, a pointing device (such as a mouse, trackball, touchpad, or graphic tablet), a scanner, a touch screen incorporated into a display, an audio input device (such as a voice recognition system, a microphone), and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computing device 810 or onto a communication network.

[0117] The user interface output device 820 may include a display subsystem, a printer, a fax machine, or a non-visual display, such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device (such as a liquid crystal display (LCD)), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and ways of outputting information from the computing device 810 to a user or another machine or computing device.

[0118] The storage subsystem 824 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 824 may include Figure 3 、 Figure 4 、 Figure 5 、 Figure 6 The processes and / or logic of selected aspects of other methods described herein.

[0119] These software modules are typically executed by the processor 814 alone or in combination with other processors. The memory 825 used in the storage subsystem 824 can include several memories, including a primary random access memory (RAM) 830 for storing instructions and data during program execution and a read-only memory (ROM) 832 for storing fixed instructions. The file storage subsystem 826 can provide permanent storage for program and data files and can include a hard drive, a floppy disk drive together with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of certain embodiments can be stored in the storage subsystem 824 by the file storage subsystem 826, or in other machines accessible by the processor(s) 814.

[0120] The bus subsystem 812 provides a mechanism for the various components and subsystems of the computing device 810 to communicate with each other as intended. Although the bus subsystem 812 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0121] The computing device 810 can be of different types including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, there are Figure 8 The description of computing device 810 depicted in FIG is intended only as a specific example for purposes of illustrating some embodiments. Figure 8 Many other configurations of computing device 810 are possible with more or fewer components than the computing device depicted.

[0122] Where the systems described herein collect or may utilize personal information about users (or generally referred to herein as "participants"), users may be provided with the opportunity to control whether programs or features collect information about the user (e.g., information about the user's social network, social actions or activities, occupation, the user's preferences, or the user's current geographic location), or to control whether and / or how content that may be more relevant to the user is received from content servers. In addition, prior to storage or use of particular data, the particular data may be processed in one or more ways such that personally identifiable information is removed. For example, the user's identity may be processed such that the user's personally identifiable information cannot be determined, or the user's geographic location, from which geographic location information (such as a city, zip code, or state / county level) may be generalized such that the user's specific geographic location cannot be determined. Thus, users may have control over how information about the user is collected and / or used.

[0123] In some embodiments, a method implemented by one or more processors is provided, the method comprising receiving free-form natural language instructions describing a task for a robot, the free-form natural language instructions generated based on user interface input provided by a user via one or more user interface input devices. In some embodiments, the method comprises processing the free-form natural language instructions using a natural language instruction encoder to generate a potential goal representation of the free-form natural language instructions. In some embodiments, the method comprises receiving an instance of visual data, the visual data instance generated by at least one visual component of the robot, and the visual data instance capturing at least a portion of the robot's environment. In some embodiments, the method comprises generating an output based on processing at least (a) the visual data instance and (b) the potential goal representation of the free-form natural language instructions using a goal-conditional policy network, wherein the goal-conditional policy network is trained based on at least (i) a set of target images of training instances, wherein the target images are used to describe the training task, and (ii) a set of natural language instructions of training instances, wherein the free-form natural language instructions are used to describe the training task. In some embodiments, the method comprises controlling one or more actuators of the robot based on the generated output, wherein controlling the one or more actuators of the robot causes the robot to perform at least one action indicated by the generated output.

[0124] These and other implementations of the technology disclosed herein can include one or more of the following features.

[0125] In some embodiments, the method includes receiving additional free-form natural language instructions describing additional tasks for the robot, the additional free-form natural language instructions generated based on additional user interface input provided by a user via one or more user interface input devices. In some embodiments, the method includes processing the additional free-form natural language instructions using a natural language instruction encoder to generate additional potential goal representations for the additional free-form natural language instructions. In some embodiments, the method includes receiving additional visual data instances generated by at least one visual component of the robot. In some embodiments, the method includes using a goal-conditional policy network to generate additional outputs based on processing at least (a) the additional visual data instances and (b) the additional potential goal representations for the additional free-form natural language instructions. In some embodiments, the method includes controlling one or more actuators of the robot based on the generated additional outputs, wherein controlling the one or more actuators of the robot causes the robot to perform at least one additional action indicated by the generated additional outputs.

[0126] In some embodiments, the additional task of the robot is different from the task of the robot.

[0127] In some embodiments, each training instance in a target image set of training instances that uses a target image to describe a training task includes an imitation trajectory provided by a human and a target image that describes the training task performed by a robot in the imitation trajectory. In some embodiments, generating each training instance in the target image set of training instances includes receiving a data stream, capturing the state of the robot and the corresponding action of the robot when a human is controlling the robot to interact with an environment. In some embodiments, the method includes, for each training instance in the target image set of training instances, selecting an image frame sequence from the data stream; selecting the last image frame in the image frame sequence as a training target image that describes the training task performed in the image frame sequence; and generating a training instance by storing the selected image frame sequence as part of the imitation trajectory of the training instance and the training target image as part of the target image of the training instance as the training instance.

[0128] In some embodiments, each training instance in a natural language instruction set that uses free-form natural language instructions to describe a training instance includes an imitation trajectory provided by a human and a free-form natural language instruction that describes a training task performed by a robot in the imitation trajectory. In some embodiments, generating each training instance in the natural language instruction set for the training instance includes receiving a data stream, capturing a state of the robot and a corresponding action of the robot when a human is controlling the robot to interact with an environment. In some embodiments, the method includes, for each training instance in the natural language instruction set for the training instance, selecting an image frame sequence from the data stream; providing the image frame sequence to a human reviewer; receiving training free-form natural language instructions that describe a training task performed by the robot in the image frame sequence; and generating the training instance by storing the selected image frame sequence as part of the imitation trajectory of the training instance and the training free-form natural language instructions as part of the free-form natural language instruction of the training instance as the training instance.

[0129] In some embodiments, training a target-conditional policy network based on at least (i) a set of target images of training instances, wherein the target images are used to describe a training task, and (ii) a set of natural language instructions for the training instances, wherein the training task is described using free-form natural language instructions, includes: selecting a first training instance from the set of target images of training instances, wherein the first training instance includes a first imitation trajectory and a first target image describing the first imitation trajectory. In some embodiments, the method includes generating a latent space representation of the first target image by processing a first target image portion of the first training instance using a target image encoder. In some embodiments, the method includes processing at least (1) an initial image frame in the first imitation trajectory and (2) a latent space representation of the first target image portion of the first training instance using the target-conditional policy network to generate a first candidate output. In some embodiments, the method includes determining a target image loss based on the first candidate output and one or more portions of the first imitation trajectory. In some embodiments, the method includes selecting a second training instance from the set of natural language instructions for training instances, wherein the second training instance includes a second imitation trajectory and a second free-form natural language instruction describing the second imitation trajectory. In some embodiments, the method includes generating a latent space representation of a second free-form natural language instruction by processing a second free-form natural language instruction portion of a second training instance using a natural language encoder, wherein the latent space representation of the first target image and the latent space representation of the second free-form natural language instruction are represented in a shared latent space. In some embodiments, the method includes processing at least (1) an initial image frame in a second imitation trajectory and (2) the latent space representation of the second free-form natural language instruction portion of the second training instance using a target-conditional policy network to generate a second candidate output. In some embodiments, the method includes determining a natural language instruction based on the second candidate output and one or more portions of the second imitation trajectory. In some embodiments, the method includes determining a target-conditional loss based on a target image loss and a natural language instruction loss. In some embodiments, the method includes updating one or more portions of a target image encoder, a natural language instruction encoder, and / or a target-conditional policy network based on the determined target-conditional loss.

[0130] In some embodiments, a target-conditional policy network is trained based on a first number of training instances of a set of target images of training instances and a second number of training instances of a set of natural language instructions of the training instances, wherein the second number is less than fifty percent of the first number. In some embodiments, the second number is less than ten percent of the first number, less than five percent of the first number, or less than one percent of the first number.

[0131] In some implementations, the generated output comprises a probability distribution over an action space of the robot, and wherein controlling the one or more actuators based on the generated output comprises selecting at least one action based on at least one action having a highest probability in the probability distribution.

[0132] In some embodiments, generating an output based on processing at least (a) a visual data instance and (b) a potential target representation of a free-form natural language instruction using a goal-conditional policy network further includes: generating an output based on processing (c) at least one action using the goal-conditional policy network, and wherein controlling one or more actuators based on the generated output includes: selecting at least one action based on at least one action satisfying a threshold probability.

[0133] In some embodiments, a method implemented by one or more processors is provided, the method comprising receiving free-form natural language instructions describing a task for a robot, the free-form natural language instructions generated based on user interface input provided by a user via one or more user interface input devices. In some embodiments, the method comprises processing the free-form natural language instructions using a natural language instruction encoder to generate a potential goal representation of the free-form natural language instructions. In some embodiments, the method comprises receiving a visual data instance generated by at least one visual component of the robot, the visual data instance capturing at least a portion of the robot's environment. In some embodiments, the method comprises generating an output based on processing at least (a) the visual data instance and (b) the potential goal representation of the free-form natural language instructions using a goal-conditional policy network. In some embodiments, the method comprises controlling one or more actuators of the robot based on the generated output, wherein controlling the one or more actuators of the robot causes the robot to perform at least one action indicated by the generated output. In some embodiments, the method comprises receiving a target image instruction describing an additional task for the robot, the target image instruction provided by the user via one or more user interface input devices. In some embodiments, the method comprises processing the target image instruction using a target image encoder to generate a potential goal representation of the target image instruction. In some embodiments, the method includes receiving an additional visual data instance, the additional visual data instance generated by at least one visual component of the robot, and the additional visual data instance capturing at least a portion of the robot's environment. In some embodiments, the method includes generating an additional output based on processing at least (a) the additional visual data instance and (b) a potential goal representation of a goal image instruction using a goal-conditional policy network. In some embodiments, the method includes controlling one or more actuators of the robot based on the generated additional output, wherein controlling the one or more actuators of the robot causes the robot to perform at least one additional action indicated by the generated additional output.

[0134] In some embodiments, a method implemented by one or more processors is provided, the method comprising selecting a first training instance from a set of target images of training instances, wherein the first training instance comprises a first imitation trajectory and a first target image describing the first imitation trajectory. In some embodiments, the method comprises generating a latent space representation of the first target image by processing a first target image portion of the first training instance using a target image encoder. In some embodiments, the method comprises processing at least (1) an initial image frame in the first imitation trajectory and (2) a latent space representation of the first target image portion of the first training instance using a target-conditional policy network to generate a first candidate output. In some embodiments, the method comprises determining a target image loss based on the first candidate output and one or more portions of the first imitation trajectory. In some embodiments, the method comprises selecting a second training instance from a set of natural language instructions of training instances, wherein the second training instance comprises a second imitation trajectory and a second free-form natural language instruction describing the second imitation trajectory. In some embodiments, the method comprises generating a latent space representation of a second free-form natural language instruction by processing the second free-form natural language instruction portion of the second training instance using a natural language encoder, wherein the latent space representation of the first target image and the latent space representation of the free-form natural language instruction are represented in a shared latent space. In some embodiments, the method includes processing, using a goal-conditional policy network, a latent space representation of at least (1) an initial image frame in a second imitation trajectory and (2) a second free-form natural language instruction portion of a second training instance to generate a second candidate output. In some embodiments, the method includes determining a natural language instruction based on the second candidate output and one or more portions of the second imitation trajectory. In some embodiments, the method includes determining a goal-conditional loss based on a goal image loss and a natural language instruction loss. In some embodiments, the method includes updating one or more portions of a goal-conditional policy network based on the determined goal-conditional loss.

[0135] In addition, some embodiments include one or more processors (e.g., central processing unit(s) (CPUs), graphics processing unit(s) (GPUs), and / or tensor processing unit(s) (TPUs)) of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause any of the methods described herein to be performed. Some embodiments also include one or more transitory or non-transitory computer-readable storage media storing computer instructions, which instructions can be executed by the one or more processors to perform any of the methods described herein.

Claims

1. A method implemented by one or more processors, the method comprising: receiving free-form natural language instructions describing a task for the robot, the free-form natural language instructions generated based on user interface input provided by a user via one or more user interface input devices; processing the free-form natural language instruction using a natural language instruction encoder to generate a latent target representation of the free-form natural language instruction, wherein the latent target representation of the free-form natural language instruction is represented in a shared latent space with a latent target representation of a target image; receiving a visual data instance generated by at least one vision component of the robot and capturing at least a portion of an environment of the robot; generating an output based on processing at least (a) the visual data instance and (b) the latent goal representation of the free-form natural language instruction using a goal-conditional policy network, wherein the goal-conditional policy network is trained based on at least (i) a set of target images of training instances, wherein the target images are used to describe the training tasks, and (ii) a set of natural language instructions of training instances, wherein the training tasks are described using free-form natural language instructions; and One or more actuators of the robot are controlled based on the generated output, wherein controlling the one or more actuators of the robot causes the robot to perform at least one action indicated by the generated output.

2. The method according to claim 1, further comprising: receiving additional free-form natural language instructions describing additional tasks for the robot, the additional free-form natural language instructions generated based on additional user interface input provided by the user via the one or more user interface input devices; processing the additional free-form natural language instructions using the natural language instruction encoder to generate additional potential target representations for the additional free-form natural language instructions; receiving additional visual data instances generated by the at least one visual component of the robot; generating, using the goal-conditional policy network, additional outputs based on processing the additional potential goal representations of at least (a) the additional visual data instances and (b) the additional free-form natural language instructions; as well as The one or more actuators of the robot are controlled based on the generated additional output, wherein controlling the one or more actuators of the robot causes the robot to perform at least one additional action indicated by the generated additional output.

3. The method according to claim 2, wherein: The additional task of the robot is different from the task of the robot.

4. The method according to claim 3, wherein: Each training instance in the target image set of the training instances that uses target images to describe a training task includes an imitation trajectory provided by a human and a target image that describes the training task performed by the robot in the imitation trajectory.

5. The method according to claim 4, wherein Generating each training instance in the target image set of the training instance includes: When the human is controlling the robot to interact with the environment, receiving a data stream, capturing the state of the robot and the corresponding action of the robot; For each training instance in the target image set of the training instance: selecting a sequence of image frames from the data stream; selecting the last image frame in the image frame sequence as a training target image for describing the training task in the image frame sequence; and The training instance is generated by storing the selected image frame sequence as the imitation trajectory portion of the training instance and the training target image as the target image portion of the training instance as the training instance.

6. The method according to claim 5, wherein: Each training instance in the natural language instruction set of the training instances that uses free-form natural language instructions to describe training includes an imitation trajectory provided by a human and a free-form natural language instruction describing the training task performed by the robot in the imitation trajectory.

7. The method according to claim 6, wherein: Each training instance in the natural language instruction set for generating the training instance includes: receiving a data stream, capturing the state of the robot and the corresponding action of the robot while the human is controlling the robot to interact with the environment; For each training instance in the set of natural language instructions for the training instance: selecting a sequence of image frames from the data stream; providing the sequence of image frames to a human reviewer; receiving training free-form natural language instructions describing a training task to be performed by the robot in the sequence of image frames; The training instance is generated by storing the selected image frame sequence as the imitation trajectory portion of the training instance and the training free-form natural language instructions as the free-form natural language instruction portion of the training instance as the training instance.

8. The method according to claim 7, wherein: Training the goal-conditional policy network based on at least (i) a set of target images of the training instances, wherein the target images are used to describe a training task, and (ii) a set of natural language instructions of the training instances, wherein the training task is described using free-form natural language instructions, comprises: Selecting a first training instance from the target image set of training instances, wherein the first training instance includes a first imitation trajectory and a first target image describing the first imitation trajectory; generating a latent space representation of the first target image by processing a first target image portion of the first training instance using a target image encoder; processing, using the target-conditional policy network, the latent space representation of at least (1) an initial image frame in the first imitation trajectory and (2) the first target image portion of the first training instance to generate a first candidate output; determining a target image loss based on the first candidate output and one or more portions of the first imitation trajectory; selecting a second training instance from the set of natural language instructions for the training instance, wherein the second training instance includes a second imitation trajectory and a second free-form natural language instruction describing the second imitation trajectory; generating a latent space representation of the second free-form natural language instruction by processing a second free-form natural language instruction portion of the second training instance using the natural language encoder, wherein the latent space representation of the first target image and the latent space representation of the second free-form natural language instruction are represented in a shared latent space; processing the latent space representation of at least (1) an initial image frame in the second imitation trajectory and (2) the second free-form natural language instruction portion of the second training instance using the goal-conditional policy network to generate a second candidate output; determining a natural language instruction loss based on the second candidate output and one or more portions of the second imitation trajectory; determining a target conditional loss based on the target image loss and the natural language instruction loss; and One or more portions of the target image encoder, the natural language instruction encoder, and / or the target-conditional policy network are updated based on the determined target-conditional loss.

9. The method according to claim 8, wherein The goal-conditional policy network is trained based on a first number of training instances of the set of target images of the training instances and a second number of training instances of the set of natural language instructions of the training instances, wherein the second number is less than fifty percent of the first number.

10. The method according to claim 9, wherein: The second amount is less than ten percent of the first amount, less than five percent of the first amount, or less than one percent of the first amount.

11. The method according to any one of claims 1 to 10, wherein: The generated output comprises a probability distribution over an action space of the robot, and wherein controlling the one or more actuators based on the generated output comprises selecting the at least one action based on the at least one action having a highest probability in the probability distribution.

12. The method according to any one of claims 1 to 10, wherein: Generating an output based on using the goal-condition policy network to process at least (a) the visual data instance and (b) the potential goal representation of the free-form natural language instruction further includes: generating an output based on using the goal-condition policy network to process (c) at least one action, and wherein, controlling the one or more actuators based on the generated output includes: selecting the at least one action based on a probability that the at least one action satisfies a threshold.

13. A method implemented by one or more processors, the method comprising: receiving free-form natural language instructions describing a task for the robot, the free-form natural language instructions generated based on user interface input provided by a user via one or more user interface input devices; processing the free-form natural language instruction using a natural language instruction encoder to generate a latent target representation of the free-form natural language instruction, wherein the latent target representation of the free-form natural language instruction is represented in a shared latent space with a latent target representation of a target image; receiving a visual data instance generated by at least one vision component of the robot and capturing at least a portion of an environment of the robot; generating an output based on processing at least (a) the visual data instance and (b) the latent goal representation of the free-form natural language instruction using a goal-conditional policy network; controlling one or more actuators of the robot based on the generated output, wherein controlling the one or more actuators of the robot causes the robot to perform at least one action indicated by the generated output; receiving a target image instruction describing an additional task for the robot, the target image instruction being provided by the user via the one or more user interface input devices; processing the target image command using a target image encoder to generate a latent target representation of the target image command; receiving an additional visual data instance generated by the at least one visual component of the robot and capturing at least a portion of the environment of the robot; generating additional outputs based on processing at least (a) the additional visual data instances and (b) the latent goal representation of the goal image instruction using the goal-conditional policy network; and The one or more actuators of the robot are controlled based on the generated additional output, wherein controlling the one or more actuators of the robot causes the robot to perform at least one additional action indicated by the generated additional output.

14. A method implemented by one or more processors, the method comprising: Selecting a first training instance from the target image set of training instances, wherein the first training instance includes a first imitation trajectory and a first target image describing the first imitation trajectory; generating a latent space representation of the first target image by processing a first target image portion of the first training instance using a target image encoder; processing, using a target-conditional policy network, the latent space representation of at least (1) an initial image frame in the first imitation trajectory and (2) the first target image portion of the first training instance to generate a first candidate output; determining a target image loss based on the first candidate output and one or more portions of the first imitation trajectory; selecting a second training instance from the set of natural language instructions for training instances, wherein the second training instance includes a second imitation trajectory and a second free-form natural language instruction describing the second imitation trajectory; generating a latent space representation of the second free-form natural language instruction by processing a second free-form natural language instruction portion of the second training instance using a natural language encoder, wherein the latent space representation of the first target image and the latent space representation of the second free-form natural language instruction are represented in a shared latent space; processing the latent space representation of at least (1) an initial image frame in the second imitation trajectory and (2) the second free-form natural language instruction portion of the second training instance using the goal-conditional policy network to generate a second candidate output; determining a natural language instruction loss based on the second candidate output and one or more portions of the second imitation trajectory; determining a target conditional loss based on the target image loss and the natural language instruction loss; and One or more portions of the target image encoder, the natural language instruction encoder, and / or the target-conditional policy network are updated based on the determined target-conditional loss.

15. A computer program product comprising instructions which, when executed by one or more processors of a computing system, cause the computing system to perform the method according to any one of claims 1 to 14.

16. A computing system configured to perform the method according to any one of claims 1 to 14.

17. A computer-readable storage medium storing instructions executable by one or more processors of a computing system to perform the method according to any one of claims 1 to 14.

Images