End-to-end video-conditioned robotic policy learning

Abstract

Implementations relate to training and / or utilizing an end-to-end model that can, at each of multiple iterations, process, as input, (a) prompt frames of a prompt video that captures performance of a task and (b) current robot observations (e.g., current image(s) from a camera of a robot) to generate, as output, a robotic action for the current iteration. A sequence of robotic actions can be generated through multiple iterations and, when implemented by the robot over the iterations, can lead to successful performance of the task that is captured by the prompt video.

Description

END-TO-END VIDEO-CONDITIONED ROBOTIC POLICY LEARNINGATTORNEY REFERENCE: DEEP-0008-WO-01Background

[0001] Robotic transformers (RTs), such as RT-1, RT-2, and RT-X, have been proposed that can use language conditioning in generating a robot trajectory. For example, RTs can be used to process natural language, along with images from a robot camera, to generate a sequence of robotic actions that, when implemented by a robot, result in performance of a robotic task that conforms to the processed natural language.

[0002] However, relying solely on natural language to specify a task can have various drawbacks. For example, RTs conditioned only on natural language can fail to generate an appropriate robot trajectory when term(s) of the natural language are polysemic and / or for robotic tasks whose realizations vary dramatically based on context. For example, “open drawer”, “open cabinet”, “open container with lid” and “open jar with screw cap” might share the same verb, but very different robotic motor control is required for each interaction.

[0003] A separate approach that has been proposed is the utilization of a final goal image in goal -conditioned behavior cloning tasks. While several task specifications can be defined in terms of the resulting state of the environment that is reflected in the final goal image, there are many tasks for which the manner in which the action is performed is part of the specification. For example, “hold the flag” and “wave the flag” can have the same final goal image, but very different robotic motor control is required for each interactionSummary

[0004] Implementations disclosed herein generate a sequence of robotic actions for performing a robotic task utilizing conditioning that is based on a prompt video that captures a prior occurrence of the task. For example, the prompt video can capture a prior occurrence of human performance of the task. Such conditioning on a prompt video can be utilized instead of or in addition to any conditioning that is based on natural language description of a task. In these and other manners, reliance on detailed, and potentially ambiguous, natural language prompts that describe the task can be reduced or eliminated. Moreover, performance of various tasks, that are difficult to or incapable of being fully described in natural language, can be enabled.

[0005] Various implementations disclosed herein are directed to training and / or utilizing an end- to-end model that can, at each of multiple iterations, process, as input, (a) prompt frames of a prompt video that captures performance of a task and (b) current robot observations (e.g., current image(s) from a camera of a robot) to generate, as output, a robotic action for the current iteration. A sequence of robotic actions can be generated through multiple iterations and, when implemented by the robot over the iterations, can lead to successful performance of the task that is captured by the prompt video.

[0006] As one example, the end-to-end model can include a prompt encoder, a state encoder, and a policy model. The prompt encoder can be used, at each iteration, to process prompt image frames of a prompt video to generate a corresponding prompt video encoding. The state encoder can be used, at each iteration, to process current robot observations to generate a corresponding state encoding. The policy model can be used, at each iteration, to process a corresponding prompt video encoding of the iteration and a corresponding state encoding of the iteration to generate a corresponding action of the sequence of actions. The policy model can optionally also be used to process an action position embedding at each iteration and along with the corresponding prompt video encoding and the corresponding state encoding. Also, optionally, a language encoder can be utilized to process natural language, that is descriptive of the text, to generate a natural language encoding and the policy model can be used to process the natural language encoding along with the corresponding prompt video encoding of the iteration and the corresponding state encoding of the iteration.

[0007] In some implementations, the policy model includes a transformer encoder and a transformer decoder. In some versions of those implementations, processing both the corresponding prompt video encoding and the corresponding state encoding using the policy model includes cross-attending, using the transformer encoder, between the corresponding state encoding and the corresponding prompt video encoding to generate corresponding crossattended state-prompt output. For example, the corresponding state encoding can be a query and the corresponding prompt video encoding can be a key and a value. In some of those versions, generating the corresponding action can include processing the corresponding cross-attended state-prompt output, using the transformer decoder, to generate the corresponding action. Additionally or alternatively, in some of those versions the transformer encoder includes crossattention transformer layers.

[0008] Implementations that are directed to training the policy model can utilize training data that includes multiple prompt video, robot trajectory pairs. The prompt video of a pair can capture a prior occurrence of performance of a task. The robot trajectory of the pair can include a robot video and a sequence of robotic actions. The robot video can capture an additional prior occurrence, by a robot, of performance of the task. The sequence of robotic actions can be those performed by the robot in the additional prior occurrence of performance of the task.

[0009] Some prompt video, robot trajectory pairs can include a prompt video that is of a first robot in a first setting performing a task and a robot video that is of the first robot, or a second robot, in a distinct second setting performing the task. The performances of the task by the robots can be based on a separate policy or can be human-guided (e.g., teleoperation and / or kinesthetic guidance).

[0010] Additionally or alternatively, some prompt video, robot trajectory pairs can include a robot video that is of a robot in a first setting performing a task and a robot video that is of a human, in the first setting or a distinct second setting, performing the task by attempting to recreate or mimic the previous performance of the task by the robot. The performance of the task by the robot can be based on a separate policy or can be human-guided (e.g., teleoperation and / or kinesthetic guidance).

[0011] Additionally or alternatively, some prompt video, robot trajectory pairs can include a robot video that is of a robot in a first setting performing a task and a robot video that is of a human, in the first setting or a distinct second setting, independently performing the task. The performance of the task by the robot can be based on a separate policy or can be human-guided (e.g., teleoperation and / or kinesthetic guidance).

[0012] In training based on a prompt video, robot trajectory pair the prompt encoder can be used to process prompt image frames of the prompt video to generate a prompt video encoding. For example, the prompt image frames can include multiple frames sampled from the prompt video and, optionally, an initial frame (e.g., at or near the beginning of the task performance) and a terminal frame (e.g., at or near the end of the task performance). The state encoder can be used to process robot image frames from the robot video of the robot trajectory to generate a state encoding. For example, the robot state data can include multiple robot image frames that precede a sampled timestamp of the robot video and robot trajectory. The policy model can be used to process the prompt video encoding and the state encoding to generate a predicted action.

[0013] The predicted action can be compared to a ground truth action to generate a loss, and the loss utilized in updating the policy model (e.g., using backpropagation). The ground truth action can be from the sequence of robotic actions of the robot trajectory, and can be used based on it occurring at or near the sampled timestamp. For example, the ground truth action can be used based on it occurring at the sampled timestamp or being the first action, of the sequence of robotic actions, that occurs following the timestamp. Optionally, a Temporal Cycle Consistency (TCC) loss for per-frame alignment can additionally be generated and utilized in the updating to encourage similar representation between the prompt video and the robot video.

[0014] Once trained, the end-to-end model can be utilized to enable robotic performance of a task, conditioned on a prompt video that captures performance of a task. The task captured by the prompt video can be distinct from the tasks on which the policy model of the end-to-end model has been trained. For example, when a robot is deployed in an environment a human can demonstrate a task within the view of a camera of the robot and the camera of the robot can be used to record that performance as a prompt video. As another example, when a robot is deployed in an environment a human can teleoperate and / or kinesthetically control the robot to demonstrate a task and the camera of the robot can be used to record that performance as a prompt video. As yet another example, a prompt video can capture a prior robot or human performance of a task in an environment that is unique from the environment in which the robot is deployed. In such examples, after the prompt video is recorded the robot can process the prompt video, and current image(s) from the camera of the robot, using the end-to-end model to generate a sequence of robotic actions for implementation by the robot in performing the task.

[0015] In various implementations, a corpus of prompt videos can be accessible for utilization with the end-to-end model in controlling a robot. Which end-to-end model is selected for utilization in controlling the robot at a given time can be based on various considerations. For example, each prompt video can be stored with metadata such as word(s) and / or word embedding(s) that are descriptive of the task captured by the prompt video. In such an example, one of the prompt videos can be selected for utilization based on spoken or typed natural language user interface input that directly or indirectly describes the task. For example, word(s) from the natural language user interface input and / or an embedding thereof can be compared to the metadata to select a prompt video. For instance, the closest matching video, optionally that satisfies a matching threshold, can be selected. In such an example, one of the prompt videoscan additionally or alternatively be selected for utilization based on a robotic task being indicated as needing to be performed (e.g., reflected in output from a robotic planner or other system). For example, the indicated robotic task can be compared to the metadata of the corpus of prompt videos to select a prompt video to utilize.

[0016] Some implementations include a transitory or non-transitory computer readable storage medium storing instructions executable by a processor to perform a method such as one or more of the methods described herein. Some implementations include a robot and / or a control system including memory and one or more processors operable to execute instructions, stored in the memory, to implement one or more modules or engines that, alone or collectively, perform a method such as one or more of the methods described herein.

[0017] It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.Brief Description of the Drawings

[0018] FIG. 1 depicts an example method of training a policy model, based on prompt video, robot trajectory pairs according to implementations disclosed herein.

[0019] FIG. 2 depicts an example method of controlling a robot to perform a task based on a prompt video that captures the task, according to implementations disclosed herein.

[0020] FIG. 3 depicts an example robot, in accordance with various implementations.

[0021] FIG. 4 schematically depicts an example architecture of a computer system.

[0022] FIG. 5 schematically depicts an example of utilizing a prompt encoder, state encoder, and a policy model in generating robotic actions based on a prompt video and robot visual observations, according to implementations disclosed herein.

[0023] FIGS. 6A, 6B, 6C, and 6D schematically depict various losses, one or more of which can be utilized in training at least a policy model, according to implementations disclosed herein.Detailed Description

[0024] FIG. 1 is a flowchart illustrating an example method 100 of training a policy model, based on prompt video, robot trajectory pairs according to implementations disclosed herein.For convenience, the operations of the method 100 are described with reference to a system that performs the operations. This system can include one or more components of one or morecomputer systems, such as one or more servers. Moreover, while operations of method 100 are shown in a particular order, this is not meant to be limiting. One or more operations can be reordered, omitted or added.

[0025] At block 152, the system identifies a prompt video, robot trajectory pair. The prompt video captures a prior occurrence of performance of a task. The robot trajectory includes a robot video and a sequence of robotic actions. The robot video captures an additional prior occurrence, by a robot, of performance of the task. The sequence of robotic actions is a recorded sequence performed by the robot in the additional prior occurrence, by the robot, of performance of the task. The sequence of robotic actions can be generated responsive to control of the robot according to a separate policy and / or scripted movements and / or can be generated responsive to human-based control of the robot, such as teleoperation and / or kinesthetic manipulation of the robot.

[0026] In some implementations, a given iteration of block 152 includes one of sub-blocks 152A, 152B, or 152C.

[0027] At sub-block 152A the prompt video, robot trajectory pair that is identified is a robotrobot pair. That is, the prior occurrence of performance of the task that is captured in the prompt video is a prior occurrence by a robot, which can optionally be a different robot than the one of the robot video. The prior occurrence of performance of the task that is captured in the prompt video can be in a different environment than the environment of the robot video.

[0028] At sub-block 152B the prompt video, robot trajectory pair that is identified is a hindsight human-robot pair. That is, the prior occurrence of performance of the task that is captured in the prompt video is a prior occurrence by a human, where the human performs the task of the robot video, optionally in the same environment as that of the robot video. For example, the human can be provided with hindsight guidance, such as the robot video and / or with natural language instructions that were utilized by the robot in performing the task of the robot video, and perform the task of the prompt video with the guidance.

[0029] At sub-block 152B the prompt video, robot trajectory pair that is identified is a humanrobot pair. That is, the prior occurrence of performance of the task that is captured in the prompt video is a prior occurrence by a human, where the human performs the task of the robot video without hindsight guidance, optionally in the same environment as that of the robot video.

[0030] At block 154, the system processes, using a prompt encoder of an end-to-end model, prompt image frames of the identified prompt video, to generate a prompt video encoding. Block 154 optionally includes sub-block 154A, in which the system selects the prompt image frames, processed in block 154, based on sampling the prompt video, such as truly or pseudo random sampling of the prompt video. For example, the system can select an initial frame and a terminal frame of the prompt video, and randomly sample N frames from between the initial and the terminal frame.

[0031] At block 156, the system processes, using a state encoder, robot image frames of the robot video of the identified robot video of the robot trajectory, to generate a state encoding. Block 156 optionally includes sub-block 156A, in which the system selects the robot image frames based on the robot image frames being the N frames that precede a sampled timestamp of the robot video. For example, the system can randomly sample a timestamp from the robot video, and select N frames that immediately precede the sampled timestamp, optionally padding if less than N frames precede the sampled timestamp.

[0032] At block 158, the system processes, using a policy model, both the prompt video encoding of a most recent iteration of block 154 and the state encoding of a most recent iteration of block 156, to generate a predicted robotic action. Optionally, at an iteration of block 158 the predicted robotic action is a next robotic action and additional following predicted robotic actions are also generated at the iteration of block 158. For example, the next three or four robotic actions can be predicted at each iteration of block 158.

[0033] In some implementations, block 158 includes sub-blocks 158A and 158B. At sub-block 158A, the system cross-attends, using a transformer encoder of the policy model, between the state encoding and the prompt video encoding to generate cross-attended state-prompt output. At sub-block 158B, the system processes, using a transformer decoder of the policy model, the cross-attended state-prompt output to generate the robotic action.

[0034] At block 160, the system uses the predicted robotic action and a ground truth robotic action, from the robotic trajectory, in updating at least the policy model. In some implementations, the system selects the ground truth robotic action based on it occurring at or near the sampled timestamp of optional sub-block 156A. In some implementations, the system generates a loss, such as a Cross Entropy Loss, based on comparing the predicted robotic actionand the ground truth robotic action, and updates the model based on the loss (e.g., using backpropagation or other updating technique(s)).

[0035] In implementations where multiple robotic actions are predicted at block 158, the system can, at block 160, use the multiple predicted robotic actions and multiple ground truth robotic actions, from the robotic trajectory, in updating at least the policy model. For example, if a sequence of four predicted robotic actions is predicted at block 158, the system can select the sequence of four ground truth robotic actions that occur following the sampled timestamp of optional sub-block 156A. In some implementations, the system can additionally utilize one or more auxiliary losses in updating the policy model and / or encoder(s). Examples of such auxiliary losses include, but are not limited to, one or more of the auxiliary losses described in FIGS. 6B, 6C, and 6D.

[0036] Auxiliary loss(es) can include one or more that encourage learning features that are helpful in understanding semantics in prompt videos. For example, auxiliary loss(es) can include an alignment loss between prompt videos and robot videos performing the same task, such as a temporal-cycle consistency loss. A temporal-cycle consistency loss has been shown to encode task progress when trained on videos of different agents performing the same task. By aligning prompt videos and robot videos, the image encoder is encouraged to be invariant to different embodiments, lighting, backgrounds, view-angles and distractor objects while still encoding features relevant to predicting task progress. To apply the loss, an alignment pooling layer on the per-frame image embedding can be used, such as an average pool in spatial dimensions on the per-frame embeddings output from ViT and followed by a 2-layer MLP. As another example, auxiliary loss(es) can include a contrastive loss between prompt tokens produced by robot or prompt video performing the same task. This loss encourages the prompt token to encode task semantics from video tokens only in a self-supervised manner. An Attention Pooling layer can be used to merge features from the N prompt tokens to produce a single embedding for each video. For N robot videos and N prompt videos in the batch, N robot embeddings and N prompt embeddings can be retrieved. They can be matmuled and a SigLIP loss applied to encourage videos of the same task (also motion and interacting objects) to be close to each other while being away from other videos in the batch. In other words, the videos of robot and prompt performing the same task can be labeled as positives and assign them a label of 1 along the diagonal and 0 for off-diagonal pairs. As yet another example, auxiliary loss(es) can include acontrastive loss between prompt tokens produced by robot video and the text instructions of the task. This encourages a part of the embedding space to be aware of object names and verbs present in the prompt and robot videos. An Attention Pooling layer with one latent query can be used to merge features from the N prompt tokens to produce a single embedding for each video. For B videos in a batch, B pairs of video and text embeddings can be retrieved. They can be matmuled and a SigLIP loss applied to encourage videos of the same task (also motion and interacting objects) to be close to each other while being away from other videos in the batch.

[0037] At block 162, the system determines whether to perform additional sampling from the currently identified prompt video, robot trajectory pair. If so, the system proceeds back to block 154. If not, the system proceeds to block 164 and determines whether to process an additional prompt video, robot trajectory pair. If so, the system proceeds back to block 152 and identifies an additional prompt video, robot trajectory pair. If not, the system proceeds to block 166 and deploys the end-to-end model, including the prompt encoder, state encoder, and policy model, for utilization on one or more real robots.

[0038] Multiple (e.g., hundreds, thousands) iterations of method 100 can be performed in training at least a policy model of an end-to-end model. How many iterations are performed, via decisions at block 162 and / or 164, can depend on various factors such as an available amount of prompt video, robot trajectory pairs, a desired quantity and / or duration of training, and / or whether one or more performance criteria for the end-to-end model have been achieved.

[0039] FIG. 2 is a flowchart illustrating an example method 200 of controlling a robot to perform a task based on a prompt video that captures the task, according to implementations disclosed herein. For convenience, the operations of the method 200 are described with reference to a system that performs the operations. This system can include one or more components of a robot, such as a robot processor and / or robot control system of robot 320, and / or other robot and / or can include one or more components of a computer system, such as computer system 410. Moreover, while operations of method 200 are shown in a particular order, this is not meant to be limiting. One or more operations can be reordered, omitted or added.

[0040] At block 252, the system retrieves a prompt video capturing prior performance of a task. The prior performance, captured by the prompt video, can be a prior performance by a human or a prior performance by robot(s).

[0041] In some implementations, block 252 includes sub-block 252A in which the system selects the prompt video from a corpus of prompt videos of different tasks, such as a locally and / or a remotely stored corpus. For example, the system can receive natural language input that is based on a spoken or typed utterance from a user and that is descriptive of the task and select the prompt video based on determining that metadata, for the prompt video, matches the natural language input. For instance, a user can speak “robot, clean the countertops” and a prompt video, that captures cleaning of countertops, can be selected based on it having metadata (e.g., word(s), word embedding(s)) that matches the natural language input (e.g., match word(s) and / or word embedding(s) thereof). As another example, the system can determine, based on output from a robotic planner, a need for the robot to perform the robotic task and can select the prompt video responsive to determining the need for the robot to perform the robotic task and based on determining that metadata, for the prompt video, matches the robotic task. For instance, a higher-level robotic planner system can direct the robot to “clean the countertops” and a prompt video, that captures cleaning of countertops, can be selected based on it having metadata that matches the output from the robotic planner.

[0042] At block 254, the system processes, using a prompt encoder of an end-to-end model, prompt image frames of the retrieved prompt video, to generate a prompt video encoding.

[0043] At block 256, the system processes, using a state encoder, current robot image frames from a vision component of a robot (e.g., RGB images from an RGB camera, 2.5D images from an RGBD camera, or point cloud images from a Lidar component), to generate a state encoding.

[0044] At block 258, the system processes, using a policy model, both the prompt video encoding of a most recent iteration of block 254 and the state encoding of a most recent iteration of block 256, to generate a predicted robotic action. Optionally, at an iteration of block 258 the predicted robotic action is a next robotic action and additional following predicted robotic actions are also generated at the iteration of block 258. For example, the next three or four robotic actions can be predicted at each iteration of block 258.

[0045] In some implementations, block 258 includes sub-blocks 258 A and 258B. At sub-block 258 A, the system cross-attends, using a transformer encoder of the policy model, between the state encoding and the prompt video encoding to generate cross-attended state-prompt output. At sub-block 258B, the system processes, using a transformer decoder of the policy model, the cross-attended state-prompt output to generate the robotic action.

[0046] At block 262, the system causes the robotic action to be implemented via the robot (ie., the robot associated with e robot image frames of block 256). For example, when the robotic action is not a terminal / terminate / end action, the system can cause the robotic action to be implemented by causing control of actuator(s) of the robot in accordance with the action. For instance, when the action indicates robot base movement, robot arm movement(s), and / or robot end effector movement(s), the system can provide control commands that cause such movement(s) to be effectuated via control of actuators of the robot.

[0047] At block 264, the system determines whether the robotic action, of a most recent iteration of block 258, is a terminal / terminate / end action. If so, the system proceeds to block 266, awaits an additional task and, when an additional task is received, proceeds back to block 252 based on the additional task. If not, the system proceeds back to block 254 and performs another iteration of blocks 254, 256, and 258 in generating a next action to cause to be implemented at block 262.

[0048] Multiple (e.g., tens, hundreds, thousands) iterations of blocks 254, 256, 258, 262, and 264 of method 200 can be performed in controlling a robot to perform a task that is reflected in a currently retrieved prompt video.

[0049] FIG. 3 schematically depicts an example architecture of a robot 320. The robot 320 includes a robot control system 360, one or more operational components 340a-340n, and one or more sensors 342a-342m. The sensors 342a-342m may include, for example, vision sensors, light sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors, accelerometers, gyroscopes, thermometers, barometers, and so forth. While sensors 342a-m are depicted as being integral with robot 320, this is not meant to be limiting. In some implementations, sensors 342a-m may be located external to robot 320, e.g., as standalone units.

[0050] Operational components 340a-340n may include, for example, one or more end effectors and / or one or more servo motors or other actuators to effectuate movement of one or more components of the robot. For example, the robot 320 may have multiple degrees of freedom and each of the actuators may control the actuation of the robot 320 within one or more of the degrees of freedom responsive to the control commands. As used herein, the term actuator encompasses a mechanical or electrical device that creates motion (e.g., a motor), in addition to any driver(s) that may be associated with the actuator and that translate received control commands into one or more signals for driving the actuator. Accordingly, providing a controlcommand to an actuator may comprise providing the control command to a driver that translates the control command into appropriate signals for driving an electrical or mechanical device to create desired motion.

[0051] The robot control system 360 may be implemented in one or more processors, such as a CPU, GPU, and / or other controller(s) of the robot 320. In some implementations, the robot 320 may comprise a “brain box” that may include all or aspects of the control system 360. For example, the brain box may provide real time bursts of data to the operational components 340a- n, with each of the real time bursts comprising a set of one or more control commands that dictate, inter alia, the parameters of motion (if any) for each of one or more of the operational components 340a-n. In some implementations, the robot control system 360 may perform one or more aspects of method(s) described herein, such as method 200 of FIG. 2.

[0052] As described herein, in some implementations all or aspects of the control commands generated by control system 360, in controlling a robot during performance of a robotic task, can be generated based on robotic skill(s) determined to be relevant for the robotic task based on world-grounding and task-grounding measures as described herein. Although control system 360 is illustrated in FIG. 3 as an integral part of the robot 320, in some implementations, all or aspects of the control system 360 may be implemented in a component that is separate from, but in communication with, robot 320. For example, all or aspects of control system 360 may be implemented on one or more computing devices that are in wired and / or wireless communication with the robot 320, such as computing device 410.

[0053] FIG. 4 is a block diagram of an example computer system 410. Computer system 410 typically includes at least one processor 414 which communicates with a number of peripheral devices via bus subsystem 412. These peripheral devices may include a storage subsystem 424, including, for example, a memory subsystem 425 and a file storage subsystem 426, user interface output devices 420, user interface input devices 422, and a network interface subsystem 416. The input and output devices allow user interaction with computer system 410. Network interface subsystem 416 provides an interface to outside networks and is coupled to corresponding interface devices in other computer systems.

[0054] User interface input devices 422 may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touch screen incorporated into the display, audio input devices such as voice recognition systems, microphones, and / or other typesof input devices. In general, use of the term "input device" is intended to include all possible types of devices and ways to input information into computer system 410 or onto a communication network.

[0055] User interface output devices 420 may include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices. The display subsystem may include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual display such as via audio output devices. In general, use of the term "output device" is intended to include all possible types of devices and ways to output information from computer system 410 to the user or to another machine or computer system.

[0056] Storage subsystem 424 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 424 may include the logic to perform selected aspects of method 100, method 200, and / or to implement one or more aspects of robot 300. Memory 425 used in the storage subsystem 424 can include a number of memories including a main random-access memory (RAM) 430 for storage of instructions and data during program execution and a read only memory (ROM) 432 in which fixed instructions are stored. A file storage subsystem 426 can provide persistent storage for program and data files, and may include a hard disk drive, a CD- ROM drive, an optical drive, or removable media cartridges. Modules implementing the functionality of certain implementations may be stored by file storage subsystem 426 in the storage subsystem 424, or in other machines accessible by the processor(s) 414.

[0057] Bus subsystem 412 provides a mechanism for letting the various components and subsystems of computer system 410 communicate with each other as intended. Although bus subsystem 412 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0058] Computer system 410 can be of varying types including a workstation, server, computing cluster, blade server, server farm, smart phone, smart watch, smart glasses, set top box, tablet computer, laptop, or any other data processing system or computing device. Due to the everchanging nature of computers and networks, the description of computer system 410 depicted in FIG. 4 is intended only as a specific example for purposes of illustrating some implementations.Many other configurations of computer system 410 are possible having more or fewer components than the computer system depicted in FIG. 4.

[0059] FIG. 5 schematically depicts an example of utilizing a prompt encoder 520, state encoder 525, and a policy model 530 in generating robotic actions 505 based on a prompt video 501 and robot visual observations 502, according to implementations disclosed herein.

[0060] The prompt encoder 520 is used to process and encode the prompt video 501 video demonstration provided as a reference to convey the desired task semantics. The prompt encoder 520 implicitly learns to infer what task to perform and how to do it. In some implementations, the prompt encoder 520 can include a per-frame image encoder (pp(e.g. , a ViT as described in Transformers for Image Recognition at Scale by Dosovitskiy et al.) followed by a perceiver resampler ip. The output of the prompt encoder ip< >p(V) = Zprompt) can be a set of N prompt tokens 503 of d-dimension to condition the policy model 530 with the task-relevant attributes from the video.

[0061] The state encoder 525 is used to process and encode the current state of the robot and information about the objects and environment visible to the robot. For example, the state encoder 525 can process and encoder the current frame and last k frames, of robot visual observations 502, as input. In some implementations, the state encoder 525 can include a per- frame Image encoder (psfollowed by a perceiver resampler is. The output of the state encoder can be state tokens 504, such as those represented by i s f Ps St))=Zstatethat encodes the latent environment and robot state information from the history of recent observations. In some implementations, the same image encoder weights can be used for both the prompt encoder 520 and the state encoder 525 (e.g., (pp= (ps= ). The role of the image encoder (p is to capture spatial visual information in each frame. The perceiver resampler enables temporal learning across frames and reduces the number of video tokens passed into the action decoder.

[0062] In some implementations, the policy model 530 includes a state-prompt encoder 532 and an action decoder 534. The state-prompt encoder 532 takes the prompt video encoding Zpromp t(prompt tokens 503) and robot state encoding Zstate(state tokens 504) and outputs a task encoding relevant for action prediction, which are also called prompt-aware state tokens Zstate \prompt- The state tokens 504 can act as queries, and the prompt tokens 503 can act as keys and values. The state-prompt encoder 532 can enable the fusion of the state and promptinformation. For example, suppose prompt video 501 shows picking up an apple in a basket, and the current state, reflected by robot visual observations 502, contains an apple, a banana, and an orange. The state-prompt encoder 532 cross-attends and learns which object to attend to in the current state based on the prompt video 501. Capturing interdependencies between prompt and state can be beneficial for action decoding.

[0063] The action decoder 534 predicts an action vector at, represented by robotic action 505, for the current state Stsuch that it completes the task shown in the prompt video 501. The action decoder 534 can be a transformer decoder architecture that uses fixed action position tokens as input queries and the prompt-aware state tokens Zstate\prompt, generated using state-prompt encoder 532, for keys and values. The size of the action position embedding is N x d where N is the number of action dimensions, and d is the transformer embedding dimension.

[0064] The action position embeddings cross-attend to the prompt-aware state tokens to predict the target binned action values as output. Each output token of the action decoder corresponds to an action dimension for the mode, arm, and base. Each token embedding can be projected to 256 dimensions, and a softmax layer applied on the top to obtain the bin corresponding to the target action vector. Unlike techniques that use autoregressive action decoding that requires multiple forward passes during inference, various implementations use action position embeddings for one forward pass prediction. Instead of predicting one action for the next timestep, various implementations train the policy with a prediction horizon of multiple steps, such as four steps. In various implementations, the action bin with the highest probability is always used, that is, argmax over predicted probabilities, to choose the action value for execution.

[0065] In various implementations, cross-attention transformer layers are utilized in the prompt resampler that can be included in the prompt encoder 520, the state resampler that can be included in the state encoder 525, the state-prompt encoder 532, and / or the action decoder 534. Compared to standard self-attention layers, which require more memory to process the same video, cross-attention layers help manage the high number of tokens and the resulting large attention matrices when processing prompt and robot state videos. For example, when using ViT-B / 16, the total number of video tokens for a 16 frame reference video and a 8 frame robot state video at 224 x 224 resolution would be 8 x 196+16 x 196 = 4704. An entire self-attention operation would lead to an attention matrix with 47042 ~ 22M entries. However, using two perceiver resamplers with 64 latent tokens, training can be performed with attention matrices ofthe size 8 * 196 * 64 + 16 * 196 * 64 ~ ,3M. Thus, cross-attention layers reduce attention computation and enable training with paired videos.

[0066] FIGS. 6A, 6B, 6C, and 6D schematically depict various losses, one or more of which can be utilized in training at least a policy model, according to implementations disclosed herein.

[0067] FIG. 6A illustrates generating an action prediction loss 607A based on comparison between a predicted robotic action 605 A and a ground truth (GT) robotic action 606A. In FIG. 6A, the predicted robotic action 605A is generated based on processing a prompt video 601 A using the prompt encoder 520 and processing a current robot observation 602A (e.g., the most recent A observations) using the state encoder 525, and processing the outputs from the prompt encoder 520 and the state encoder 525 using the policy model 530. The GT robotic action 606A can be a corresponding expert action (e.g., a human-provided action) from the current robot observation 602A. In some implementations, the action prediction loss 607A is a classification loss (e.g., cross entropy) on actions discretized into multiple (e.g., 256) bins. Put another way, given a demonstration robot trajectory for performing a task with current observation xtI 602A, the corresponding expert action atI GT action 606A is provided from the demonstration. The action prediction loss can be Cross Entropy between the predicted action and the expert action as: LCE(at, at) =atlog log at. This action prediction loss can be used to train the policy model 530 and, optionally, the prompt encoder 520 and the state encoder 525.

[0068] In some implementations, to mitigate e.g., prevent) any over-fitting on the training set utilized in generating the action prediction losses, one or more auxiliary losses can be used to e.g., encourage learning features in prompt videos.

[0069] For example, FIG. 6B illustrates a video alignment loss in the form of a temporal cycle consistency loss 607B. Such a loss can encourage temporal alignment between prompt videos and robot videos that show the same task. By aligning prompt videos with the robot videos, the per-frame image encoder 527, optionally included in the prompt encoder 520 and / or the state encoder 525, can learn to be invariant to different embodiments, lighting, backgrounds, view angles, and distractor objects while still encoding features relevant to predicting task progress.

[0070] The illustrated temporal-cycle consistency loss 607B can encode the task progress when trained on videos of different agents performing the same task. This loss is applied on per-frame image embeddings, of a prompt video Vp160 IB and a robot video Vr1602B, that are generated during training based on processing using the per-frame image encoder 527. To apply the loss,an alignment pool 640 is used to average pool the per-frame embeddings output in spatial dimensions from per-frame image encoder (p I 52, and apply a projector head of multi-layer perceptron (MLP). This is also referred to herein as applying an alignment pooling layer <t> on the per-frame image embeddings. For each video Vt(e.g., prompt video 60 IB and robot video 602B), this results in a sequence of embeddings Et(v?), ... ^(t^1)}’ where Ltis the number of frames in the ithvideo. Temporal-cycle consistency loss can be applied on encoding Epand Erfor prompt video 60 IB and robot video 602B, respectively. The temporalcycle consistency loss ensures that the representation of every frame of Epshould correspond to Erand vice versa. Applying temporal-cycle consistency can involve two steps. First, the soft neighbor of tthframe of Ep(Epin short) in Ercan be computed and referred to as Evr

[0072] Second, the corresponding frame can be found for this newly computed soft-neighbor in Ep. This can be called cycle-back and it involves similar soft-neighbor computation as in Equation 1 to obtain EPrt-, which ideally should be same as t, that is, (£,Prt-t)2should be minimized. Temporal-cycle consistency loss minimizes such mean squared error between all frames for prompt and robot video encodings, and vice-versa, that is:

[0074] FIG. 6C illustrates a prompt-robot video contrastive loss (VVCL), in the form of a SigLIP loss 607C, that can encourage the prompt encoder 520 to learn task semantics from video tokens in a self-supervised manner. While prompt video and robot video can be paired using natural language, this does not effectively capture the visual similarity of low-level motions like reaching for objects and rotating the robot arm. For this, contrastive loss between the latent features of the robot videos of a robot video batch 602C and the prompt videos of a prompt video batch 601C can be generated and applied. An attention pooling layer, V-V pool 642, can be used to merge features from the A prompt tokens to produce a single embedding for each video. A SigLIP loss 607C, or other contrastive loss, can be generated between video-video pairs and applied to encourage videos showing the same task, involving similar motions and interacting objects, to be close to each other while being away from other videos in the batch(601C and 602C). A batch contains the same number of robot and prompt videos, say B. The prompt encoder ^C^G)) / 520 is used to process the batch of robot and prompt videos to obtain a batch of full robot video embeddings Zrobotand prompt video embeddings Zprompt, each of size Bxd. They are multiplied, Zrobot.Zpromptto obtain B B matrix. Adding a learnable temperature T and bias Z>, provides a logit matrix as Y = Zrobot.Zprompt) * r + b. The videos of a robot and prompt performing the same task are considered as positives and assigned a positive label of 1 along the diagonal and -1 for off-diagonal pairs, that is, the label matrix Y = 2 / B— 1. SigLIP loss is the negative loglikelihood a' (Z1, Z2) =—S' (Zx' Zj) * t + b). The video-video contrastive loss can be as follows:

[0076] FIG. 6D illustrates a video-text contrastive loss (VTCL), in the form of a SigLIP loss 607D, that can encourage a part of the embedding space to be aware of object names and verbs, as shown in the prompt and the robot videos. The VTCL is generated based on a loss between prompt tokens generated based on processing the robot video using the prompt encoder and text tokens generated based on processing the text instructions of the task using a text encoder 644 (e.g., a frozen text encoder). An attention pooling layer, V-T pool 646, can be utilized with one latent query to merge features from the N prompt tokens, generated using the prompt encoder 520, to produce a single embedding for each video. B pairs of robot videos and text embeddings can be retrieved and processed as a batch, as represented by text instruction batch 60 ID and robot video batch 602D. A SigLIP loss, or other VTCL loss, can be generated and applied to encourage every video to have similar embeddings to their textual description embeddings, and be different from other text embeddings in the batch. This can be represented as:

[0078] In various implementations, all four losses of FIGS. 6A-D can be utilized in training the policy model 530, the prompt encoder 520 and / or the state encoder 525. For example, the mean of all four losses can be applied for training, as represented by L = - LCE+LTCC+ LVVCL+LVTCL)-

[0079] Implementations disclosed herein seek to learn a policy that can infer the underlying task in a prompt video and perform relevant actions to accomplish it given current observations. For example, given a prompt video V and the robot state S_t={x_i }_(i=t-k- 1 )At where x_i is theframe from the robot’s camera stream at time i, k is the maximum number of historical frames, and t is the current time-step. A policy 7t(a_t |S_t,V) can be trained that infers the underlying task from V and predicts task relevant action a t. To train the model representing the policy, a dataset of paired prompt videos and robot trajectories can be used, which can be referred to as D_paired={(VJ,({S_i,a_i } (_i=0AN)) J }, where S_i={ S_iA0. . . S_iAT} is the robot trajectory of a successful demonstration and V i is a prompt video that should result in the trajectory S i, and N is the number of samples in the dataset.

[0080] As referenced above, to train a video-conditioned robot policy a dataset of pairs can be utilized: prompt videos and robot trajectories performing the same task. For example, reference videos can be utilized where the task is performed by both humans and robots. Generating this dataset can rely on Robot-Robot data, Hindsight Human-Robot data, and / or Human-Robot data.

[0081] For Robot-Robot data, existing robot-robot videos of the same task can be paired. For this pairing, two videos can be considered to match if they were performing the same task in different settings. The ‘task‘ can be defined based on natural language instructions used when recording robot trajectories. These instructions can include of one or two verbs sur-rounded by nouns, such as ‘place water bottle upright’, ‘move the soda can to the green chip bag’ or ‘open top drawer’. The objective of this pairing is two-fold: first, to be able to take advantage of an already labeled and collect dataset of robot trajectories and second to ensure robots are able to imitate when the same task is demonstrated in a different environment.

[0082] For Hindsight Human-Robot data, the task instruc_,tions from the robot trajectories dataset are used and human participant(s) are asked to perform the task and record a demonstration video from the robot’s perspective / view. The set of instructions are the same as with Robot-Robot data, but there is significant embodiment and speed variability due to different humans performing the task with left or right hands and at a randomized robot camera angle. This requires some manual effort but provides a lot of paired data for training the policy for the available set of instructions in robot dataset without having to collect new robot trajectories.

[0083] For Human-Robot data, a human and robot perform the same task. This data can be collected in different spaces such as office, living area, hardware workstations, play area, office supplies section and more.

[0084] The multiple different data sources and conditions can represent varying levels of difficulty and expense to collect. Robot-Robot data requires less additional effort but lacks diversity in how the task is done. Hindsight Human-Robot data is created by asking humans to copy existing robot trajectories. While this adds some diversity in prompt videos, it does not cover any new tasks on the robot side. Finally, the presumed gold-standard is to collect the Human-Robot data where both humans and robots perform diverse tasks. This takes the most amount of time as labor is required both of the humans and robot operators to collect new robot trajectories.

[0085] Implementations of the policy disclosed herein can take in the prompt video and current robot state and output robot actions. It can include three modules: 1) a prompt video encoder, 2) a robot state encoder, and 3) robot action decoder.

[0086] The prompt video encoder can encode the video demonstration provided as a reference to convey the desired task semantics. The prompt video encoder implicitly learns to infer what task should be performed and how it needs to be done. The architecture can be a per-frame Image encoder (e.g., ViT) 4>_p followed by perceiver resampler \| / _p. The output of the prompt video encoder \| / _p ((|>_p (V))=z_prompt can be a set of N tokens of d-dimension to condition the policy with the task relevant attributes from the video.

[0087] The robot state encoder can encode the current state of the robot given the current frame and last k frames as input. The architecture can be a per-frame Image encoder c])_s followed by perceiver resampler \| / _s. Similar to prompt encoder outputs, the output of the state encoder can be \p_s ((|)_s (S t ))=z_state that encodes the latent environment and robot state information from the history of recent observations.

[0088] Both the prompt video encoder and the robot state encoder can capture spatial visual informa_,tion in each frame. The perceiver resampler can be used to enable temporal learning across frames as well as reduce the number of video tokens that must be passed into the action decoder.

[0089] The policy can include state-prompt encoder 7t_sp and action decoder 7t_a, that is 7t=[7t_sp,7t_a]. The state-prompt encoder can take the prompt video encoding and robot state encoding and output task encoding relevant for action prediction. The module can be trained to output robot actions by cross-attending between the state encoding as queries and prompt video encoding as keys and values. The state-prompt encoder enables fusion of the state and promptinformation. For example, if the prompt video demonstrates picking up of an apple in the basket and the current state contains apple, banana and orange, then the cross attention between the state and prompt encoding enables learning for which object to attend to in the state, which can be beneficial for the next step of action decoding. The output of the state-prompt encoder can be referred to as prompt-aware state tokens.

[0090]

[0091] The action decoder can take action position embeddings (each corresponding to a single action dimension) and the prompt-aware state tokens as inputs. The action position embeddings can cross-attend to a prompt-aware state to predict the target binned action values as output. Each output token of the action decoder can correspond to an action category for the mode, arm and base. For example, each token embedding can be projected to 256 dimensions and a softmax layer can be applied on the top to obtain the bin corresponding to the target action vector. Unlike using autoregressive action decoding that requires multiple forward pass or caching during inference, action category embeddings can be used for one forward pass prediction like in ACT. The argmax of the action bin probabilities can be used to choose the correct action. Instead of predicting one action for the next timestep, the policy can be trained with a prediction horizon of multiple (e.g., four) steps.

[0092] To handle the varying lengths of videos for efficient training, multiple frames (e.g., N = 16) can be randomly sampled, optionally always including first and last frames and sorting them in increasing order of time. During training, a robot state St can be sampled by sampling a random timestep first. The preceding k - 1 frames can then be sampled to create a robot state video comprising of a total of k frames before (e.g., k=8). In case there are less than k - 1 frames before the current time-step the first frame can be repeated to create a fixed size robot state video. The pixel values in each frame can be normalized between 0 to 1. Each frame can be resized (e.g., to (224, 224)). Photometric distortions like cropping, brightness, contrast, hue, and / or saturation can be applied during training.

[0093] The action at that timestep can include multiple components such as: Mode: whether to terminate episode, move only arm, move only base or both. Arm: gripper position (x, y, z), orientation (rotation along xy, yz, zx), and degree of closedness. Base: displacement (x, y) and rotation (9). Each of the values have different ranges, which can first be used to scale the valuesin between 0 and 1. The values can then be discretized into 256 bins each. For example, an 11- dim action vector can be constructed as a target, each of which has value within 0 to 256.

[0094] Provided in the following paragraphs is a non-limiting example of some particular implementations disclosed herein, including success metrics therefor.

[0095] The model, implemented in Jax, can be trained for 200K iterations. AdamW optimizer can be used with an initial learning rate of 8e-5 using a cosine learning rate schedule with warmup steps 2,000 and final learning rate of le-6.

[0096] Metrics A rollout can refer to a sequence of actions inferred from the policy and executed on the robot from an initial state observation and prompt video, until the policy terminates or max number of steps are taken, whichever is lower. Success for a rollout can refer to when the policy executes the task instruction shown in the prompt video. A successful rollout involves successively correct decisions in the environment, without any resets or recovery assistance. A mistake made early on in a rollout can result in poor performance, even if the model’s offline overall prediction accuracy is high. For example, if a policy mistakenly fails to grasp a water bottle and it slips to an unreachable location, the rollout will be marked as a failure, even if the policy had good prediction accuracy for the later steps.

[0097] For each task instruction, many rollouts can be recorded per pol-icy. Average of success recorded across all the rollouts can be determined and called Success Rate for that task.Aggregated success rate across tasks is referred as Overall Success Rate. To record partial progress, it can be recorded whether the robot reached at the correct location, grasped the correct object, released the object at the correct location, and terminated the task correctly.

[0098] The model is compared with BC-Z, a video conditioned policy using ResNet-18 encoder. While earlier reported performance in BC-Z was by training on a limited dataset, the baseline is trained and evaluated as the same split of train, validation and held-out tasks used in the model for a fair comparison. The same training mixture as the model is used, that is, 90% of robotrobot paired data, 5% hindsight human-robot paired data, and 5% human-robot paired data. BC- Z doesn’t have a terminate action so the rollouts run for a fixed number of steps.

[0099] Human operators are asked to evaluate success for a policy’s rollout on a robot. All the policies are evaluated for a task successively rather than randomly sampling a task and policy for a rollout, so that each policy receives similar initial robot observation as input. Additionally,keeping the same scene layout between successive rollouts also saves time required to arrange task-specific initial scene.

[0100] Each operator has a different robot hardware, lighting, chest of drawers and interactions objects. This allows for some redundancy to account for the variability and unforeseen factors in an individual’s subjective evaluation. Each operator is assigned to use either 0, 1, 2 or 3 distractors for every rollout, for post-hoc analysis on the impact of distractor objects presence on success rate. Up to 3 videos are held out for evaluation for each task. For a rollout, a prompt video is sampled from this held-out set of videos, thereby, ensuring that the policies are evaluated on a few visually different prompt videos that are unseen during training. Overall, the same configuration rollouts for all policies are ensured, redundancy and realistic variability in terms of distractors and unseen prompt videos in evaluations.

[0101] Questions addressed herein include: 1) Can end-to-end trained video-conditioned policies per-form the task as shown in an unseen video during training? 2) What is the gap in success rate due to embodiment difference in prompt videos? 3) Can the learned motion representations for out-of-distribution object interactions be leveraged?

[0102] To answer these questions, two main results are presented. First, BC-Z for task-based success with robot and human prompt videos is compared. There, each model’s ability to infer the task specification from prompt video and successfully complete the task in the unseen visual observations is assessed. To analyze the cause of failures, the partial success metrics are visualized as well. Second, the ability to transfer motion from prompt videos to another object is evaluated. This test is useful to disambiguate whether the policy can focus on the prompt video, or has it merely learned object-action affordances. Plus, some interesting emergent behaviors with new objects are noted.

[0103] We compare the model and baselines on robot and human prompt videos in Table I. Both prompt videos and robot state are similar but not the same as in training, so each model’s ability to infer the task specification from prompt video as well as the current observed state of the robot is evaluated.

[0104] Tasks: In order to test the capabilities of the model in dif-ferent settings on real robot, it is evaluated across 8 categories of manipulation tasks as shown in Table I. Specifically, it is evaluated for 9 tasks: ‘knock water bottle over’, ‘move rxbar chocolate near soda can’, ‘move green jalapeno chip bag near soda can’, ‘pick green rice chip bag’, ‘place soda can upright’,‘pick soda can from bottom drawer and place on counter’, ‘open middle drawer’, ‘close middle drawer’, and ‘place apple into top drawer’.

[0105] Setup: We have 4 operators carrying out 2 trials per task for a prompt video dataset and policy setting (a row in Table I), that is, 8 trials per task to evaluate a policy’s task success rate. We report overall success rate per row over 9 tasks with 8 trials per task, that is, 9 * 8 = 72 trials. In total, evaluations presented in Table I required 72 * 4 = 288 real robot rollouts.

[0106] Observations: The model is compared with BC-Z when prompted with robot and human videos. BC-Z serves as a strong baseline for comparisons. The overall performance of the model outperforms BC-Z for Human prompt videos by 20%, and is comparable for Robot prompt videos. Note that there is an order of magnitude more training samples for robot trajectories than human videos. Hence, there isn’t a significant gap in performance for robot prompt videos. For human prompt videos, the model outperforms BC-Z in most tasks, except placing upright. The model outperforms in tasks like picking from drawer and placing on the counter, and opening / closing drawer tasks by a large margin. The most challenging task is placing upright and knocking over.

[0107] Analysis: In addition to marking a rollout as a success, partial success annotations per rollout are recorded. It is observed that the model reaches to the correct object more consistently than baseline. The policies often fail to reach the correct object due to reaching a distractor. Next, grasping errors happen, parti cu-larly with small and deformable objects and in collision prone areas like drawer handle or counter’s edge. After grasping, most tasks require releasing at a correct location. While BC-Z runs for a fixed number of steps, the policy predicts when to terminate. It is observed that the rate of release and terminate is almost identical for the model, that means, that after releasing at correct location, the policy mostly successfully terminates.

[0108] To understand the impact of distractors, the success rate grouped by number of distractors used in the rollouts is analyzed. The success rate of all 4 models when either no distractor or 1, 2, 3 distractor objects were used is compared. It is observed that while the policies are overall more successful when present in fewer distractors state, the model outperforms BC-Z in presence of 3 distractors, for both robot and human prompt videos.

[0109] The policy and baseline were trained with paired videos. This implies that the training data in-cluded only those scenarios where the interaction object shown in prompt is present in the current robot observations. But what if we provided a prompt video of one object and testedon other objects. Does it do the same action? Interestingly, it is found the model to perform learned manipulation actions on objects that it has not seen in train set. This can be referenced as emergent behavior as cross object motion transfer. For example, for a prompt video to ‘place soda can upright’, it is observed that the policy is able transfer the action of ‘placing upright’ to several objects, like a green can, a chips bag, a stapler, and a soft toy. Note that the policy adheres to the prompt video and chooses green can over chips bag or banana for placing upright.

[0110] Cross object motion transfer ability with BC-Z on 5 prompt videos is compared. Each prompt video is evaluated with unrelated objects in the robot's initial observation. The objects used for evaluation are "orange". "green can', "chips bag" , "banana" , "pink piggy soft toy" , "wrist watch" . The objects are selected to have diverse shape, size and deformability to evaluate situations requiring different grasps. BC-Z is often unable to successfully complete the tasks in out of distribution settings. In contrast, the model generalizes better to unseen objects and performs the motion indicated in the prompt video.TABLE I: Task Success Rate for Robot and Human prompts.TABLE II: Cross-object motion transfer success.

[0111] While several implementations have been described and illustrated herein, a variety of other means and / or structures for performing the function and / or obtaining the results and / or one or more of the advantages described herein may be utilized, and each of such variations and / or modifications is deemed to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and / or configurations will depend upon the specific application or applications for which the teachings is / are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific implementations described herein. It is, therefore, to be understood that the foregoing implementations are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, implementations may be practiced otherwise than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent, is included within the scope of the present disclosure.

[0112] In some implementations, a method implemented by processor(s) is provided and includes retrieving a prompt video that captures a prior occurrence of performing a task. The method further includes generating a sequence of robotic actions for performing, by a robot in a current environment of the robot, a robotic task that corresponds to the task of the prompt video. Generating the sequence of robotic actions can include, at each of a plurality of iterations: processing, using a prompt encoder, corresponding prompt image frames of the prompt video, to generate a corresponding prompt video encoding; processing, using a state encoder, corresponding robot image frames from a vision component of the robot, to generate a corresponding state encoding; and processing, using a policy model, both the corresponding prompt video encoding and the corresponding state encoding to generate at least onecorresponding action of the sequence of robotic actions. The method can further include causing the sequence of robotic actions to be implemented by the robot in performing the robotic task.

[0113] These and other implementations of the technology disclosed herein can optionally include one or more of the following features.

[0114] In some implementations, the policy model includes a transformer encoder and a transformer decoder. In some versions of those implementations, processing, using the policy model, both the corresponding prompt video encoding and the corresponding state encoding to generate the at least one corresponding action of the sequence of robotic actions includes crossattending, using the transformer encoder, between the corresponding state encoding as a query and the corresponding prompt video encoding as a key and a value to generate a corresponding cross-attended state-prompt output. In some of those versions, processing, using the policy model, both the corresponding prompt video encoding and the corresponding state encoding to generate the at least one corresponding action of the sequence of robotic actions further includes processing the corresponding cross-attended state-prompt output, using the transformer decoder, to generate the corresponding action. In some of those versions, processing, using the policy model, both the corresponding prompt video encoding and the corresponding state encoding to generate the at least one corresponding action of the sequence of robotic actions further includes processing an action position embedding, along with the corresponding cross-attended stateprompt output and using the transformer decoder, to generate the at least one corresponding action.

[0115] In some implementations, the prompt encoder includes a per-frame image encoder topologically followed by a perceiver resampler.

[0116] In some implementations, the at least one corresponding action includes a given action that includes one or more arm parameters for a robot arm of the robot and / or one or more base parameters for a robot base of the robot. In some of those implementations, the given action further includes a mode parameter. For example, the mode parameter can indicate whether the task performance should terminate or end and / or can indicate which of multiple component(s) of the robot should be controlled by the action (e.g., base only, base and arm, arm only).

[0117] In some implementations, the at least one corresponding action includes two or more actions in sequence.

[0118] In some implementations, retrieving the prompt video includes selecting the prompt video from a corpus of prompt videos of performing different tasks. In some versions of those implementations, the method further includes receiving natural language input that is based on a spoken or typed utterance from a user and that is descriptive of the task, and selecting the prompt video includes selecting the prompt video based on determining that metadata, for the prompt video, matches the natural language input. In some other versions of those implementations, the method further includes determining, based on output from a robotic planner, a need for the robot to perform the robotic task, and selecting the prompt video responsive to determining the need for the robot to perform the robotic task and based on determining that metadata, for the prompt video, matches the robotic task.

[0119] In some implementations, the method further includes; receiving natural language input that is based on a spoken or typed utterance from a user and that is descriptive of the task and generating, using a language encoder, a language encoding of the natural language input. In those implementations, processing, using the policy model, both the corresponding prompt video encoding and the corresponding state encoding to generate the at least one corresponding action further includes processing the language encoding, using the policy model and along with the corresponding video encoding and the corresponding state encoding, to generate the at least one corresponding action.

[0120] In some implementations, a method implemented by processor(s) is provided and includes identifying training data that includes a prompt video that captures a prior occurrence of performance of a task, a robot video that captures an additional prior occurrence, by a robot, of performance of the task, and a sequence of robotic actions performed by the robot in the additional prior occurrence of performance of the task. The method further includes generating at least one predicted robotic action. Generating the at least one predicted robotic action includes: processing, using a prompt encoder, prompt image frames of the prompt video, to generate a prompt video encoding; processing, using a state encoder, robot image frames of the robot video, to generate a state encoding; and processing, using a policy model, both the prompt video encoding and the state encoding to generate the at least one predicted robotic action. The method further includes updating at least the policy model based on the at least one predicted robotic action and a ground truth robotic action from the sequence of robotic actions.

[0121] These and other implementations of the technology disclosed herein can optionally include one or more of the following features.

[0122] In some implementations, the prior occurrence of performance of the task, captured by the prompt video, is a prior human performance of the task.

[0123] In some implementations, the prior occurrence of performance of the task, captured by the prompt video, is a prior robotic performance of the task that is by the robot or an additional robot and that is in an alternate environment than an environment of the additional prior occurrence of performance of the task.

[0124] In some implementations, the additional prior occurrence, by the robot, of performance of the task is a human-guided performance, such as a human-guided performance that is via teleoperation and / or that is via kinesthetic guiding.

[0125] In some implementations, the method further includes selecting the prompt image frames based on sampling of the prompt video, selecting the robot image frames based on the robot image frames preceding a sampled timestamp, and / or selecting the ground truth robotic action based on it corresponding to the sampled timestamp. In some of those implementations, selecting the prompt image frames includes selecting an initial frame and a terminal frame of the prompt video for inclusion in the prompt image frames and selecting additional prompt image frames, based on random sampling of the robot video, for inclusion in the prompt image frames.

[0126] In some implementations, the loss is Cross Entropy Loss.

[0127] In some implementations, the policy model includes a transformer encoder and a transformer decoder. In some versions of those implementations, processing, using the policy model, both the prompt video encoding and the state encoding to generate the at least one predicted robotic action includes cross-attending, using the transformer encoder, between the state encoding as a query and the prompt video encoding as a key and a value to generate a cross-attended state-prompt output. In some of those versions, processing, using the policy model, both the prompt video encoding and the state encoding to generate the at least one predicted robotic action further includes processing the cross-attended state-prompt output, using the transformer decoder, to generate the corresponding action. In some of those versions, processing, using the policy model, both the prompt video encoding and the state encoding to generate the at least one predicted robotic action further includes processing an action positionembedding, along with the cross-attended state-prompt output and using the transformer decoder, to generate the at least one predicted robotic action.

[0128] In some implementations, the prompt encoder includes a per-frame image encoder topologically followed by a perceiver resampler.

[0129] In some implementations, the at least one corresponding action includes a given action that includes one or more arm parameters for a robot arm of the robot, one or more base parameters for a robot base of the robot, and / or a mode parameter.

[0130] In some implementations, the training data further includes natural language input that is descriptive. In some versions of those implementations, the method further includes generating, using a language encoder, a language encoding of the natural language input. In those versions, processing, using the policy model, both the prompt video encoding and the state encoding to generate the at least one predicted robotic action further includes processing the language encoding, using the policy model and along with the video encoding and the state encoding, to generate the at least one predicted robotic action.

Claims

CLAIMSWhat is claimed is:

1. A method implemented using one or more processors, the method comprising: retrieving a prompt video that captures a prior occurrence of performing a task; generating a sequence of robotic actions for performing, by a robot in a current environment of the robot, a robotic task that corresponds to the task of the prompt video, wherein generating the sequence of robotic actions comprises: at each of a plurality of iterations: processing, using a prompt encoder, corresponding prompt image frames of the prompt video, to generate a corresponding prompt video encoding; processing, using a state encoder, corresponding robot image frames from a vision component of the robot, to generate a corresponding state encoding; and processing, using a policy model, both the corresponding prompt video encoding and the corresponding state encoding to generate at least one corresponding action of the sequence of robotic actions; and causing the sequence of robotic actions to be implemented by the robot in performing the robotic task.

2. The method of claim 1, wherein the policy model comprises a transformer encoder and a transformer decoder.

3. The method of claim 2, wherein processing, using the policy model, both the corresponding prompt video encoding and the corresponding state encoding to generate the at least one corresponding action of the sequence of robotic actions comprises: cross-attending, using the transformer encoder, between the corresponding state encoding as a query and the corresponding prompt video encoding as a key and a value to generate a corresponding cross-attended state-prompt output.

4. The method of claim 3, wherein processing, using the policy model, both the corresponding prompt video encoding and the corresponding state encoding to generate the at least one corresponding action of the sequence of robotic actions further comprises: processing the corresponding cross-attended state-prompt output, using the transformer decoder, to generate the corresponding action.

5. The method of claim 4, wherein processing, using the policy model, both the corresponding prompt video encoding and the corresponding state encoding to generate the at least one corresponding action of the sequence of robotic actions further comprises: processing an action position embedding, along with the corresponding cross-attended state-prompt output and using the transformer decoder, to generate the at least one corresponding action.

6. The method of any preceding claim, wherein the prompt encoder comprises a per-frame image encoder topologically followed by a perceiver resampler.

7. The method of any preceding claim, wherein the at least one corresponding action comprises a given action that includes: one or more arm parameters for a robot arm of the robot, and / or one or more base parameters for a robot base of the robot.

8. The method of claim 7, wherein the given action further includes a mode parameter.

9. The method of any preceding claim wherein the at least one corresponding action comprises two or more actions in sequence.

10. The method of any preceding claim wherein retrieving the prompt video comprises selecting the prompt video from a corpus of prompt videos of performing different tasks.

11. The method of claim 10, further comprising: receiving natural language input that is based on a spoken or typed utterance from a user and that is descriptive of the task; and wherein selecting the prompt video comprises selecting the prompt video based on determining that metadata, for the prompt video, matches the natural language input.

12. The method of claim 10, further comprising: determining, based on output from a robotic planner, a need for the robot to perform the robotic task; wherein selecting the prompt video is responsive to determining the need for the robot to perform the robotic task and comprises selecting the prompt video based on determining that metadata, for the prompt video, matches the robotic task.

13. The method of any preceding claim, further comprising:receiving natural language input that is based on a spoken or typed utterance from a user and that is descriptive of the task; and generating, using a language encoder, a language encoding of the natural language input; wherein processing, using the policy model, both the corresponding prompt video encoding and the corresponding state encoding to generate the at least one corresponding action further comprises processing the language encoding, using the policy model and along with the corresponding video encoding and the corresponding state encoding, to generate the at least one corresponding action.

14. A method implemented using one or more processors, the method comprising: identifying training data that includes: a prompt video that captures a prior occurrence of performance of a task, a robot video that captures an additional prior occurrence, by a robot, of performance of the task, and a sequence of robotic actions performed by the robot in the additional prior occurrence of performance of the task; generating at least one predicted robotic action, wherein generating the at least one predicted robotic action comprises: processing, using a prompt encoder, prompt image frames of the prompt video, to generate a prompt video encoding; processing, using a state encoder, robot image frames of the robot video, to generate a state encoding; and processing, using a policy model, both the prompt video encoding and the state encoding to generate the at least one predicted robotic action; and updating at least the policy model based on the at least one predicted robotic action and a ground truth robotic action from the sequence of robotic actions.

15. The method of claim 14, wherein the prior occurrence of performance of the task, captured by the prompt video, is a prior human performance of the task.

16. The method of claim 14, wherein the prior occurrence of performance of the task, captured by the prompt video, is a prior robotic performance of the task that is by the robot or an additional robot and that is in an alternate environment than an environment of the additional prior occurrence of performance of the task.

17. The method of any one of claims 14 to 16, wherein the additional prior occurrence, by the robot, of performance of the task is a human-guided performance.

18. The method of claim 17, wherein the human-guided performance is via teleoperation and / or is via kinesthetic guiding.

19. The method of any one of claims 14 to 18, further comprising: selecting the prompt image frames based on sampling of the prompt video; selecting the robot image frames based on the robot image frames preceding a sampled timestamp; and selecting the ground truth robotic action based on it corresponding to the sampled timestamp.

20. The method of claim 19, wherein selecting the prompt image frames comprises: selecting an initial frame and a terminal frame of the prompt video for inclusion in the prompt image frames; and selecting additional prompt image frames, based on random sampling of the robot video, for inclusion in the prompt image frames.

21. The method of any one of claim 14 to 20, wherein the loss is Cross Entropy Loss.

22. The method of any one of claim 14 to 21, wherein the policy model comprises a transformer encoder and a transformer decoder.

23. The method of claim 22, wherein processing, using the policy model, both the prompt video encoding and the state encoding to generate the at least one predicted robotic action comprises: cross-attending, using the transformer encoder, between the state encoding as a query and the prompt video encoding as a key and a value to generate a cross-attended state-prompt output.

24. The method of claim 23, wherein processing, using the policy model, both the prompt video encoding and the state encoding to generate the at least one predicted robotic action further comprises: processing the cross-attended state-prompt output, using the transformer decoder, to generate the corresponding action.

25. The method of claim 24, wherein processing, using the policy model, both the prompt video encoding and the state encoding to generate the at least one predicted robotic action further comprises: processing an action position embedding, along with the cross-attended state-prompt output and using the transformer decoder, to generate the at least one predicted robotic action.

26. The method of any one of claims 14 to 25, wherein the prompt encoder comprises a per-frame image encoder topologically followed by a perceiver resampler.

27. The method of any one of claims 14 to 26, wherein the at least one corresponding action comprises a given action that includes: one or more arm parameters for a robot arm of the robot, and / or one or more base parameters for a robot base of the robot.

28. The method of claim 27, wherein the given action further includes a mode parameter.

29. The method of any one of claims 14 to 26, wherein the training data further includes natural language input that is descriptive of the task and further comprising: generating, using a language encoder, a language encoding of the natural language input; wherein processing, using the policy model, both the prompt video encoding and the state encoding to generate the at least one predicted robotic action further comprises processing the language encoding, using the policy model and along with the video encoding and the state encoding, to generate the at least one predicted robotic action.

30. A system comprising memory storing instructions and one or more processors operable to execute the instructions to perform the method of any preceding claim.

31. A robot comprising a vision component, actuators, memory storing instructions, and one or more processors operable to execute the instructions to perform the method of any one of claims 1 to 13.

Images