Diffusion model predictive control

Abstract

Implementations are provided for improving the performance of model predictive control (MPC) by learning a joint trajectory-level dynamics models over sequences of states rather than using one-step models. In various implementations, an input prompt may be assembled with representations of a proposed sequence of actions to be performed by a robot to carry out a task and the current state of the robot. The input prompt may be processed using a trajectory dynamics diffusion model to generate output from which a proposed sequence of predicted states to be reached by the robot upon performance of the sequence of actions by the robot can be derived. The proposed sequences of actions and predicted states may form a proposed state-action trajectory. One or more alternative state-action trajectories may be sampled and evaluated to determine one or more actions to be performed by the robot.

Description

Attorney Docket No: DEEP-0018-WO-01 DIFFUSION MODEL PREDICTIVE CONTROL ATTORNEY REFERENCE: DEEP-0018-WO-01Background

[0001] Model predictive control (MPC), also called receding horizon control, uses a dynamicsmodel and an action selection mechanism (planner) to construct “agents” that can solve and / or optimize a wide variety of reward functions, and hence solve a wide variety of tasks, including novel tasks. MPC is more flexible than other methods, such as model-based reinforcement learning (RL), for learning a model-free policy that is designed to optimize a fixed reward function. In addition, learning a dynamics models usually requires less data than learning a policy, since the supervisory signal (next state prediction) is high dimensional, and dynamics models can often be adapted more easily than policies to novel environments. Summary

[0002] Various implementations described herein are directed to improving the performance ofmodel predictive control (MPC) by learning joint trajectory-level dynamics models over sequences of states rather than using one-step models. More particularly, but not exclusively, various implementations described herein relate to learning such trajectory dynamics models using a diffusion model, and then using the diffusion model with online MPC to optimize novel reward function(s) at run time. Techniques described herein may provide improved performance compared to other MPC techniques, such as MPC techniques based on single-step dynamics models. Techniques described herein may also exhibit improved sampling efficiency compared to model-free offline RL methods.

[0003] Several implementations described herein relate to methods for performing selectedaspects of the present disclosure. Other implementations may include a non-transitory computer readable storage medium storing instructions executable by a processor to perform a method such as one or more of the methods described herein. Yet another implementation may include a control system including memory and one or more processors operable to execute instructions, stored in the memory, to implement one or more modules or engines that, alone or collectively, perform a method such as one or more of the methods described herein.Attorney Docket No: DEEP-0018-WO-01

[0004] It should be appreciated that all combinations of the foregoing concepts and additionalconcepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. Brief Description of the Drawings

[0005] Fig. 1 schematically depicts an example environment in which selected aspects of thepresent disclosure may be implemented.

[0006] Fig. 2 schematically depicts an example robot.

[0007] Fig. 3 schematically depicts an example method for carrying out selected aspects of thepresent disclosure.

[0008] Fig. 4 schematically depicts another example method for carrying out selected aspects ofthe present disclosure.

[0009] Fig. 5 schematically depicts another example method for carrying out selected aspects ofthe present disclosure.

[0010] Fig. 6 schematically depicts an example computer architecture.Detailed Description

[0011] Various implementations described herein are directed to improving the performance ofmodel predictive control (MPC) by learning joint trajectory-level dynamics models over sequences of states rather than using one-step models. More particularly, but not exclusively, various implementations described herein relate to learning such trajectory dynamics models using a diffusion model, and then using the diffusion model with online MPC to optimize novel reward function(s) at run time. Techniques described herein may provide improved performance compared to other MPC techniques, such as MPC techniques based on single-step dynamics models. Techniques described herein may also exhibit improved sampling efficiency compared to model-free offline RL methods.

[0012] An aspect of some forms of MPC is to design an agent that maximizes an objectivefunction ^^(^^^^:^^+^^−1, ^^^^+1:^^+^^) over a planning horizon F from the current time step t:Attorney Docket No: DEEP-0018-WO-01where ℎ^^ =is the history. Put another way, MPC may factorize the problem thatthe agent needs to solve into two parts: a modeling problem (representing the dynamics model^^^^(^^^^+1:^^+^^|^^1:^^,^^1:^^+^^−1, ℎ^^) which can be learned from an offline trajectory) and a planningproblem (e.g., using argmax to find the best sequence of actions for a given reward function). In some implementations, once the best sequence of actions is determined, the first action ^^^^(or a chunk of multiple actions) may be selected / executed, and then replanning may occur after observing the next result state, thus creating a closed loop policy.

[0013] Various techniques described herein may be used to solve such a planning problem. Insome implementations, diffusion models may be used to learn joint trajectory-levelrepresentations of (1) the world dynamics, ^^^^(^^^^+1:^^+^^|^^^^,^^^^:^^+^^−1, ℎ^^), which may be learned insome cases using offline “play” data. Additionally, in some implementations, an actionsequence proposal distribution ^^(^^^^:^^+^^−1, |^^^^ℎ^^) may be learned offline using behavior cloningon some demonstration data. Although such a proposal distribution might suggest actions that are not optimal for solving new rewards that were not seen during training, this can be compensated for by using what will be referred to herein as a “sample, score and rank” (SSR) method that outperforms more complex methods, such as trajectory optimization or the cross- entropy method. The approach of various implementations described herein will be referred to as “Diffusion Model Predictive Control” (D-MPC).

[0014] With “one-step world models” ^^(^^^^+1 |^^^^, ^^^^, ℎ^^), the next state ^^^^+1 may be predictedgiven the current state ^^^^, the next action ^^^^, and the history ℎ^^ = (^^1:^^−1, ^^1:^^−1). Thedistribution over future states ^^(^^^^+1:^^+^^ |^^1:^^, ^^1:^^+^^−1) is defined autoregressively to givesome implementations in which the transition model is assumed to be first-order Markov, the history ℎ^^can be dropped,yielding ^^(^^^^+1 |^^^^, ^^^^). Additionally or alternatively, the transition model can be conditionedon a finite history of length H (analogous to frame stacking), so that ℎ^^ =.The joint trajectory-level dynamics models may then be expressed as follows:However, one challenge with one-step world models is that they may compound errors over time.

[0015] Accordingly, in various implementations, the trajectory-level joint distribution (or“world dynamics”) may be modeled over future sequences of multiple states, given actions, asAttorney Docket No: DEEP-0018-WO-01 expressed below in what will be referred to herein as a “diffusion-based world dynamics model”:This approach may be distinct from other diffusion-based strategies because it is used for MPC, whereas other diffusion-based strategies create imaginary data to help train a model-free policy. to optimize for novel rewards at run time. With various techniques described herein, the conditional distribution ^^(^^1:^^|^^1:^^) is learned, so that a planner can be used to select the actions. Rather than requiring a continuous action space, techniques described herein can additionally or alternatively be used for discrete actions / action spaces.

[0016] In various implementations, the diffusion world model may be trained using an offlinedataset of multiple state-action trajectories, ^^ = {(^^1^^,^^^^ ^^ ^^ ^^1 , ^^2 , ^^2 , ^^3 … ):^^ = 1 ∶ ^^}.Rewards may not be required in some implementations. In other implementations, the offline dataset ^^ may include rewards, such thatIn some implementations, the trajectories of ^^ need not be derived from expert demonstration of tasks—“play-based” data collection may suffice in many cases—but it is nonetheless beneficial for ^^ to provide a reasonably dense coverage of the state-action space. In some implementations, ^^ may be used to fit the diffusion-based world dynamics model^^(^^^^+1:^^+^^ |^^1:^^, ^^^^:^^+^^−1, ℎ^^) and another diffusion-based policy proposal (e.g., actionprediction) diffusion model ^^(^^^^:^^+^^−1|^^^^, ℎ^^), e.g., by minimizing the denoising score matchingloss.

[0017] Once trained, the diffusion-based dynamics model may be used within a larger MPCmethod. In some implementations, the MPC action selection (planning) process may be accelerated by learning a policy network which can be used as a proposal distribution. Forexample, the policy network may be a distribution over action sequences ^^(^^^^:^^+^^ |^^^^, ℎ^^) forwhich another diffusion model (e.g., the aforementioned “policy proposal” or “action prediction” diffusion model) may be trained, e.g., using behavior cloning. This also increases the likelihood that the proposed actions that are used to query the trajectory dynamics diffusion model are close to the training set, avoiding out-of-distribution issues that arise with other planning techniques.Attorney Docket No: DEEP-0018-WO-01

[0018] The main MPC loop can be expressed as follows, with ^^^^ representing the worlddynamics (or “planner”) model and ^^ representing the diffusion-based policy proposal model: (^^)∞ ^^^^planner.plan(^^^^, ℎ^^ ,^^^^, ^^, ^^,^^, ^^, ^^)= env.step(^^^^,^^^^)ℎ^^ = append(^^^^, ^^^^+1, ^^^^+1)ℎ^^ = suffix(ℎ^^,^^)^^^^^^N represents the total number of samples to be taken, ^^ represents a reward, F represents theforecast horizon, and H represents the history length. J represents a heuristic value function that approximates the reward-to-go given any proposed sequence of states and actions:with ^^ representing a discount factor and V(s) representing the value function from state s, e.g., as an estimate of future reward at the leaves of this search process. The value function J may be learned in some implementations by using a transformer to regress from (^^^^:^^+^^,^^^^:^^+^^−1) to the discounted future reward in the above equation. A transformer may also be used to learn J, although J could also be computed directly if the reward function R is known. In some cases, J may be used as the objective function for optimization in MPC as a way to specify novel tasks.

[0019] Within the main MPC loop set forth above, various planning techniques may beimplemented in the planner.plan(…) operation(s). In a discrete action space, techniques such as the Monte Carlo Tree Search, e.g., as used in the MuZero algorithm, may be used. When the action space is continuous, techniques such as the random shooting method, the cross entropy method (CEM), the CEM used in conjunction with the Planet algorithm, the model-predictive path integral (MPPI) approach (which in some implementations can be modified to include a learned action proposal and / or a learning value function), model-based offline planning trajectory optimizer (MBOP-TrajOpt), and so forth, can be used.

[0020] In some implementations, the planning operations performed within the main MPC loopset forth above can be expressed as follows: Def(^^) = Planner(^^0, ℎ0, ^^^^,^^, ^^, ^^,^^, ^^):Attorney Docket No: DEEP-0018-WO-01The planning technique set forth in the above expression may differ from other techniques because, for instance, it samples a whole state sequence ^^0:^^from the trajectory dynamics diffusion model ^^^^, rather than a single state. Similarly, the planning technique set forth above may sample the whole action sequence from the policy proposal diffusion model ^^, rather than a single action.

[0026] Fig. 1 is a schematic diagram of components that can cooperate to carry out selectedaspects of the present disclosure, in accordance with various implementations. The various components depicted in Fig.1, particularly those components forming a robotic planner system 130 and a proprioception system 140, may be implemented using any combination of hardware and software. The components of Fig.1 are depicted as being communicatively coupled with each other via one or more networks 199, which may include one or more personal area networks, local area networks, and / or wide area networks (e.g., the Internet). However, this is not meant to be limiting. Various aspects of the present disclosure that are described as being performed by and / or stored on systems 130 and / or 140 can alternatively be performed by and / or stored on a single system, such as robot planner system 130, or on any combinations of systems 130 and 140.

[0027] In some implementations, techniques described herein may be used to control varioustypes of machines or apparatus. For example, in some implementations, a robot 100 may be in communication with systems 130 and / or 140. In various implementations, and / or all or parts of systems 130 and / or 140 may be implemented onboard robot 100. Other types of machines or apparatus that are not depicted in Fig.1 may also be controlled using selected aspects of the present disclosure, such as autonomous vehicles, industrial equipment, climate control systems, medical systems and / or devices, video games, and so forth.

[0028] Robot 100 may take various forms, including but not limited to a telepresence robot(e.g., which may be as simple as a wheeled vehicle equipped with a display and a camera), a robot arm, a multi-pedal robot such as a “robot dog,” an aquatic robot, a wheeled device, aAttorney Docket No: DEEP-0018-WO-01 submersible vehicle, an unmanned aerial vehicle (“UAV”), and so forth. One non-limiting example of a mobile robot arm is depicted in Fig.2. In various implementations, robot 100 may include logic 102. Logic 102 may take various forms, such as a real time controller, one or more processors, one or more field-programmable gate arrays (“FPGA”), one or more application- specific integrated circuits (“ASIC”), and so forth. In some implementations, logic 102 may be operably coupled with memory 103. Memory 103 may take various forms, such as random- access memory (“RAM”), dynamic RAM (“DRAM”), read-only memory (“ROM”), Magnetoresistive RAM (“MRAM”), resistive RAM (“RRAM”), NAND flash memory, and so forth. In some implementations, a robot controller may include, for instance, logic 102 and memory 103 of robot 100.

[0029] In some implementations, logic 102 may be operably coupled with one or morejoints 104-1 to 104-N, one or more end effectors 106, and / or one or more sensors 108-1 to 108- M, e.g., via one or more buses 109. As used herein, “joint” 104 of a robot may broadly refer to actuators, motors (e.g., servo motors), shafts, gear trains, pumps (e.g., air or liquid), pistons, drives, propellers, flaps, rotors, or other components that may create and / or undergo propulsion, rotation, and / or motion. Some joints 104 may be independently controllable, although this is not required. In some instances, the more joints robot 100 has, the more degrees of freedom of movement it may have.

[0030] As used herein, “end effector” 106 may refer to a variety of tools that may be operatedby robot 100 in order to accomplish various tasks. For example, some robots may be equipped with an end effector 106 that takes the form of a claw with two opposing “fingers” or “digits.” Such a claw is one type of “gripper” known as an “impactive” gripper. Other types of grippers may include but are not limited to “ingressive” (e.g., physically penetrating an object using pins, needles, etc.), “astrictive” (e.g., using suction or vacuum to pick up an object), or “contigutive” (e.g., using surface tension, freezing or adhesive to pick up object). More generally, other types of end effectors may include but are not limited to drills, brushes, force-torque sensors, cutting tools, deburring tools, welding torches, containers, trays, and so forth. In some implementations, end effector 106 may be removable, and various types of modular end effectors may be installed onto robot 100, depending on the circumstances. Some robots, such as some telepresence robots, may not be equipped with end effectors. Instead, some telepresence robots may include displays to render visual representations of the users controllingAttorney Docket No: DEEP-0018-WO-01 the telepresence robots, as well as speakers and / or microphones that facilitate the telepresence robot “acting” like the user.

[0031] Sensors 108-1 to 108-M may take various forms, including but not limited to 3D laserscanners (e.g., light detection and ranging, or “LIDAR”) or other 3D vision sensors (e.g., stereographic cameras used to perform stereo visual odometry) configured to provide depth measurements, two-dimensional cameras (e.g., RGB, infrared), light sensors (e.g., passive infrared), force sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors (also referred to as “distance sensors”), depth sensors, torque sensors, barcode readers, radio frequency identification (“RFID”) readers, radars, range finders, accelerometers, gyroscopes, compasses, position coordinate sensors (e.g., global positioning system, or “GPS”), speedometers, edge detectors, Geiger counters, and so forth. While sensors 108-1 to 108-M are depicted as being integral with robot 100, this is not meant to be limiting.

[0032] In some implementations, robot planner system 130 and / or proprioception system 140may include one or more computing devices cooperating to perform selected aspects of the present disclosure. An example of such a computing device is depicted schematically in Fig.6. In some implementations, one or more of systems 130 and / or 140 may include one or more servers forming part of what is often referred to as a “cloud” infrastructure, or simply “the cloud.” Alternatively, one or more components of systems 130 and / or 140 may be operated by logic 102 of robot 100.

[0033] Machine learning and / or generative model(s) described herein may take various forms,including, but not limited to, generative model(s) such as Pathways Language Model (PaLM), Unified language Model (ULM), PaLM-2-E / ULM-E, BERT, LaMDA, Meena, and / or any other generative model, such as diffusion model(s), flow models, any other generative model that is encoder-only based, decoder-only based, sequence-to-sequence based and that optionally includes an attention mechanism or other memory, etc. Generative models and / or diffusion models may have hundreds of millions, or even hundreds of billions of parameters. In some implementations, generative and / or diffusion models may include multi-modal models such as a VLM and / or a visual question answering (VQA) model, which can have any of the aforementioned architectures, and which can be used to process multiple modalities of data, particularly images and text, and / or images and audio for example, to generate one or more modalities of output. Non-limiting examples of VLMs that may be applied as described hereinAttorney Docket No: DEEP-0018-WO-01 include Gemini and / or Flamingo, to name a few. Another example of a generative model that might be used is described in “RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control” (arXiv:2307.15818), which is incorporated herein for all purposes.

[0034] Robot planner system 130 may include a synthetic image generator 134, an actiongenerator 136, and / or a state generator 138, any of which being operably coupled with one or more generative models 132. Any of generators 134, 136 and / or 136 may be implemented using any combination of hardware and software. Moreover, any of generators 134, 136 and / or 138 may be combined with other(s) of generators 134, 136 and / or 138.

[0035] In various implementations, synthetic image generator 134, action generator 136, and / orstate generator 138 may be configured to process various modalities of inputs, including but not limited to natural language snippets (e.g., requests, queries, commands, etc.), images, videos, set of reference points, etc., using one or more generative models 132 and generate various modalities of output. In many implementations described herein, natural language input that is processed by synthetic image generator 134, action generator 136, and / or state generator 138 may include a natural language request for robot 100 to perform a high-level task (e.g., “put these dishes into the dishwasher”). Synthetic image generator 134, action generator 136, and / or state generator 138 may process such a natural language request using generative model(s) 132 to generate various types of data, such as a plurality of natural language responses each conveying an action, various types of synthetic images, etc. Each natural language response may express a mid-level action to be performed by robot 100 to carry out a respective portion of the high-level task.

[0036] Synthetic image generator 134 may be configured to process various modalities of data,such as natural language, image(s), video, actions, etc., to generate (or predict) one or more synthetic images that depict a robot and / or an environment in which the robot operates in the future. For example, in some implementations, synthetic image generator 134 may assemble, as an input prompt, first data indicative of a natural language command for robot 100 to complete a task, and second data indicative of “real” or “actual” digital image(s) acquired by one or more vision sensors that depict an environment in which the robot operates. Synthetic image generator 134 may then process the input prompt using one or more multimodal generative models 132 (which may take the form of a diffusion model) to generate a synthetic goal imageAttorney Docket No: DEEP-0018-WO-01 that visualizes a predicted state of the robot and / or a predicted state of the environment upon completion of the task by the robot. Such a model may be trained and / or fine-tuned based on, for instance, recorded episodes of robots being operated to perform tasks that include initial and final images.

[0037] Action generator 136 may be configured to process various modalities of data to generateoutput indicative of action(s) to be performed by robot 100 in furtherance of carrying out the task. For example, in some implementations, action generator 136 may be configured to assemble an input prompt that includes the representations of a sequence of states, e.g., generated by state generator 138 as discussed below. In some implementations, other data may be included in this input prompt as well, such as the natural language command, the synthetic goal image, etc. Action generator 136 may then process the input prompt using one or more of the multimodal generative models (e.g., the aforementioned policy proposal diffusion model) to predict one or more actions to be performed by the robot in furtherance of completing the task. Actions may be expressed in various ways, such as robot control data, natural language commands, joint trajectories, etc.

[0038] State generator 138 may be configured to process various modalities of data to generateoutput indicative of states to be reached by robot 100 in furtherance of carrying out the task. These states of robot 100 and / or its environment may be represented and / or expressed in various ways, such as images (real or synthetic, as the case may be), sets of references points, robot poses, etc. In some implementations, state generator 138 may be configured to assemble an input prompt that include representations of an observed initial state of robot 100 (e.g., represented in a digital image, set of reference points, etc.) and / or its environment and a goal state of robot 100 and / or its environment. In some implementations, the goal state may have been generated by synthetic image generator 134. State generator 138 may then process the second input prompt using one or more of the multimodal generative models 132 to generate output from which can be derived, for instance, an interpolated sequence of predicted states to be reached by the robot between the observed initial and goal states.

[0039] Proprioception system 140 may be present in some implementations where robot 100 isbeing controlled using techniques described herein. Proprioception system 140 may be omitted in other circumstances. Proprioception system 140 may include a proprioception prediction process 142 and one or more proprioception machine learning models 144. Examples ofAttorney Docket No: DEEP-0018-WO-01 proprioception machine learning models that may be used are described in “RT-1: Robotics Transformer for Real-World Control at Scale” (arXiv:2212.06817), which is incorporated herein for all purposes, and the aforementioned RT-2 paper.

[0040] In various implementations, proprioception prediction process 142 may process inputtokens indicative of a current (or past) proprioception values of robot 100, e.g., along with other data such as data indicative of a task or action to be performed, state data of the robot and / or its environment, and / or actions predicted by action generator 136, to generate robot control data and / or predict future proprioception values of robot 100. These robot control data and / or future proprioception values may be used to operate robot 100. In instances where action generator 136 generates actions expressed in natural language, proprioception prediction process 142 may use proprioception machine learning model(s) 144 to translate these actions expressed in natural language into robot control data. In other implementations, generative model(s) used by action generator 136 may be trained to directly generate robot control data and / or future proprioception values, in which case proprioception system 140 may be omitted.

[0041] “Robot control data” may include, for instance, low-level actuator commands (alsoreferred to as “joint commands,” and may include torque commands) that directly control the actuators / joints 104-1 to 104-N of the robot, cartesian commands that specify direction(s) for an end effector 106, a target robot pose, code that specifies reward functions that a motion controller can optimize (e.g., using techniques such as receding horizon optimization) to find optimal low-level actuator commands, selected predefined robot primitives, and so forth. In some cases, robot logic 102 may be configured to convert between joint commands and Cartesian commands, e.g., using forward and / or inverse kinematics.

[0042] In various implementations, a user 150 may control robot 100 using a client device 152.While depicted as a tablet computer or smartphone in Fig.1, client device 152 may take other forms, such as a desktop or laptop computer, in-vehicle computing device, augmented reality (AR) and / or virtual reality (VR) headset or glasses, standalone “smart” speakers that host automated assistants that can be interacted with the control robot 100, etc. In various implementations, user 150 may issue one or more natural language commands, e.g., by typing the commands or uttering the commands aloud and having those spoken utterances transcribed using speech-to-text (STT) processing. These natural language commands may specify a task to be completed by robot 100 in an environment in which robot 100 operates. For example, userAttorney Docket No: DEEP-0018-WO-01 150 may ask robot 100 to “pick plate from top drawer and place on counter, and close drawer,” “close the windows,” “take the dishes from the table to the sink,” etc.

[0043] Fig. 2 depicts a non-limiting example of a robot 200 in the form of a robot arm. An endeffector 206 in the form of a gripper claw is removably attached to a sixth joint 204-6 of robot 200. In this example, six joints 204-1 to 204-6 are indicated. However, this is not meant to be limiting, and robots may have any number of joints. In some implementations, robot 200 may be mobile, e.g., by virtue of a wheeled base 255 or other locomotive mechanism. Robot 200 is depicted in Fig.2 in a particular selected configuration or “pose.”

[0044] Referring now to Fig. 3, an example method 300 of practicing selected aspects of thepresent disclosure is described. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, including those depicted in Fig.1. Moreover, while operations of method 300 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted or added.

[0045] At optional block 302, the system, e.g., by way of action generator 136, may assemble,as an initial input prompt, representations of one or more of: a task to be carried out by the robot, a current state of the robot, and an immediate history of the robot leading up to the current state of the robot. At optional block 304, the system, e.g., by way of action generator 136, may process the initial input prompt, e.g., using the aforementioned policy proposal / action prediction diffusion model ^^, to generate initial output that is usable to derive a proposed sequence of actions to be performed by the robot.

[0046] At block 306, the system, e.g., by way of state generator 138, may assemble, as a nextinput prompt, representations of the proposed sequence of actions to be performed (e.g., generated at block 304 or provided separately) by the robot to carry out a task and a current state of the robot. At block 308, the system, e.g., by way of state generator 138, may process the next input prompt, e.g., using a trajectory dynamics diffusion model ^^^^, to generate output that is usable to derive a proposed sequence of predicted states to be reached by the robot upon performance of the sequence of actions by the robot. In various implementations, the proposed sequences of actions and predicted states may together form a proposed state-action trajectory.

[0047] At block 310, the system may sample one or more alternative state-action trajectories.As noted above, this sampling may be performed using various techniques. At block 312, theAttorney Docket No: DEEP-0018-WO-01 system may evaluate the proposed and alternative state-action trajectories, e.g., using reward functions to determine scores. In some implementations, the sampling may be based on the proposed state-action trajectory determined at block 306. In some implementations, the sampling may be random. Additionally or alternatively, the sampling of block 310 may include sampling one or more alternative proposed sequences of actions to be performed by the robot using a policy proposal diffusion model ^^, assembling one or more additional input prompts to include the one or more alternative proposed sequences of actions, and processing the one or more additional input prompts using the trajectory dynamics diffusion model ^^^^to generate one or more additional sequences of states that, with the alternative proposed sequences of actions, comprise the one or more alternative state-action trajectories.

[0048] Based on the evaluating at block 312, at block 314, the system may determine one ormore actions to be performed by the robot. At block 316, the system may generate a control signal for controlling the robot based on one or more of the actions to be performed by the robot. For example, the first action of the highest ranking state-action trajectory may be used to generate the control signal. The control signal may be provided to robot 100, and then the planning process may repeat based on a next observed state of the robot and / or its environment.

[0049] Referring now to Fig. 4, an example method 400 of SSR planning is depicted (anexpression of SSR planning operations was provided previously). For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, including those depicted in Fig.1. Moreover, while operations of method 400 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted or added. As shown in Fig.4, in some cases, the operations of blocks 402-406 may correspond to the sampling performed at block 310 of method 300, while the operations of blocks 408-410 may correspond to the evaluating of block 312 of method 300.

[0050] At block 402, the system, e.g., by way of action generator 138, may sample one or morealternative proposed sequences of actions to be performed by the robot, e.g., using the policyproposal diffusion model (e.g., ^^(^^^^:^^+^^−1, |^^^^, ℎ^^)). At block 404, the system, e.g., by way ofstate generator 138, may assemble one or more additional input prompts to include the one or more alternative proposed sequences of actions. At block 406, the system, e.g., by way of state generator 138, may process the one or more additional input prompts using the trajectoryAttorney Docket No: DEEP-0018-WO-01dynamics diffusion model ^^^^(^^^^+1:^^+^^|^^1:^^, ^^1:^^+^^−1, ℎ^^) to generate one or more additionalsequences of states that, along with the alternative proposed sequences of actions, comprise the one or more alternative state-action trajectories.

[0051] At block 408, the system may score the proposed state-action trajectory (e.g., generatedat block 304 or otherwise provided) and the one or more alternative state-action trajectoriesusing a value function, e.g., ^^^^ = ^^(^^1:^^, ^^^^,1:^^). At block 410, the system may rank theproposed state-action trajectory and the one or more alternative state-action trajectories based onthe scoring. In some implementations, the highest ranking state-action trajectory, e.g., ^̂^ =argmax^^^^^^, may be used to determine the next action ^^^̂^,1to be performed, e.g., by robot 100.

[0052] Referring now to Fig. 5, an example method 500 of training a trajectory dynamicsdiffusion model, e.g., ^^^^ℎ^^), is depicted. For convenience, theoperations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, including those depicted in Fig.1. Moreover, while operations of method 500 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted or added.

[0053] At block 502, the system, e.g., by way of state generator 138, may assemble, as an inputprompt, representations of a proposed sequence of actions to be performed by a robot to carry out a task and a current state of the robot. At block 504, the system, e.g., by way of state generator 138, may process the input prompt using the trajectory dynamics diffusion model^^^^(^^^^+1:^^+^^|^^1:^^,^^1:^^+^^−1, ℎ^^) to generate output that is usable to derive a proposed sequence ofpredicted states to be reached by the robot upon performance of the sequence of actions by the robot.

[0054] At block 506, the system may compare the proposed sequence of predicted states to aground truth sequence of states reached by a robot upon performance of the proposed sequence of actions. At block 508, based on the comparing, the system may train the trajectory dynamics diffusion model. Additionally or alternatively, in some implementations, the trajectory dynamics diffusion model may be trained using an offline dataset ^^ that includes state-action trajectories that may or may not also include rewards, as described previously.

[0055] As with other MPC approaches, diffusion-based MPC approaches described herein maybe more computationally expensive than methods that use a reactive policy without explicitAttorney Docket No: DEEP-0018-WO-01 planning. However, one advantage of using planning-based methods in the offline setting, as described herein, is that they can be adapted to novel reward functions, which can be different from those optimized by the behavior policy that generated the offline data.

[0056] Fig. 6 is a block diagram of an example computer system 610. Computer system 610typically includes at least one processor 614 which communicates with a number of peripheral devices via bus subsystem 612. These peripheral devices may include a storage subsystem 624, including, for example, a memory subsystem 625 and a file storage subsystem 626, user interface output devices 620, user interface input devices 622, and a network interface subsystem 616. The input and output devices allow user interaction with computer system 610. Network interface subsystem 616 provides an interface to outside networks and is coupled to corresponding interface devices in other computer systems.

[0057] User interface input devices 622 may include a keyboard, pointing devices such as amouse, trackball, touchpad, or graphics tablet, a scanner, a touch screen incorporated into the display, audio input devices such as voice recognition systems, microphones, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and ways to input information into computer system 610 or onto a communication network.

[0058] User interface output devices 620 may include a display subsystem, a printer, a faxmachine, or non-visual displays such as audio output devices. The display subsystem may include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystemmay also provide non-visual display such as via audio output devices. In general, use of theterm "output device" is intended to include all possible types of devices and ways to output information from computer system 610 to the user or to another machine or computer system.

[0059] Storage subsystem 624 stores programming and data constructs that provide thefunctionality of some or all of the modules described herein. For example, the storage subsystem 624 may include the logic to perform selected aspects of methods 300-500, and / or to implement one or more aspects of the various components depicted in Fig.1. Memory 625 used in the storage subsystem 624 can include a number of memories including a main random- access memory (RAM) 630 for storage of instructions and data during program execution and a read only memory (ROM) 632 in which fixed instructions are stored. A file storage subsystemAttorney Docket No: DEEP-0018-WO-01 626 can provide persistent storage for program and data files, and may include a hard disk drive, a CD-ROM drive, an optical drive, or removable media cartridges. Modules implementing the functionality of certain implementations may be stored by file storage subsystem 626 in the storage subsystem 624, or in other machines accessible by the processor(s) 614.

[0060] Bus subsystem 612 provides a mechanism for letting the various components andsubsystems of computer system 610 communicate with each other as intended. Although bus subsystem 612 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0061] Computer system 610 can be of varying types including a workstation, server, computingcluster, blade server, server farm, smart phone, smart watch, smart glasses, set top box, tablet computer, laptop, or any other data processing system or computing device. Due to the ever- changing nature of computers and networks, the description of computer system 610 depicted in Fig.6 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computer system 610 are possible having more or fewer components than the computer system depicted in Fig.6.

[0062] In some examples, the machine learning models described herein can be used forcontrolling a robotic device or a simulated robotic device.

[0063] The input to the machine learning model may comprise a natural language description ofa task to be performed by the robotic device. For example the input may comprise speech or text data. Speech data may be captured by a microphone on the robotic device or on a separate device for example. Text data may be entered by a user through a keyboard or touchscreen on the robotic device or on a separate device for example, or may be generated from speech data captured by a microphone on the robotic device or on a separate device for example (for example using automatic speech recognition techniques). Thus the input may include textual or spoken instructions provided to the robotic device by a third-party (e.g., an operator). In particular, a user may control the robotic device using a client device such as a tablet computer or smart phone for example.

[0064] The input may additionally or alternatively comprise sensor data generated by one ormore sensors on the robotic device or in the environment of the robotic device. For example, the input may comprise image data captured by one or more vision sensors such as one or more cameras (e.g., RGB, infrared). The input may comprise a three-dimensional (3D) digitalAttorney Docket No: DEEP-0018-WO-01 representation of the environment captured by one or more sensors such as LIDAR sensors or depth cameras, for example point cloud data generated using a light detection and ranging (LIDAR) sensor. For example, the input may comprise sensor data from a distance or position sensor, or from an actuator. The input may include data from sensors of the agent or data from sensors that are located separately from the agent in the environment.

[0065] The input may include data characterizing the current state of the robot, e.g., one or moreof: joint position, joint velocity, joint force, torque or acceleration, e.g., gravity-compensated torque feedback, and global or relative pose of an item held by the robot. The input data may be defined in 1, 2 or 3 dimensions, and may be absolute and / or relative data. The input may also include, for example, sensed electronic signals such as motor current or a temperature signal. The input may include data captured from e.g. one or more force sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors (also referred to as “distance sensors”), depth sensors, torque sensors, barcode readers, radio frequency identification (“RFID”) readers, radars, range finders, accelerometers, gyroscopes, compasses, position coordinate sensors (e.g., global positioning system, or “GPS”), speedometers, edge detectors, Geiger counters, and so forth.

[0066] The output of the machine learning model may comprise data representing one or moretasks to be performed by the robotic device in order to perform the task.

[0067] For instance, the output may comprise natural language, for example text or speech, thatdescribes steps or sub-tasks for completing a task. The output may define one or more low-level skills, e.g. from a vocabulary of previously learnt skills.

[0068] The output may comprise robot control data that is usable to control a robot to completethe task, for example. The robot control data may include, for instance, low-level actuator commands that directly control actuators of the robotic device, cartesian commands that specify direction(s) for an end effector of the robotic device, a target robot pose, selected predefined robot primitives, and so forth. As an illustration, the output may comprise action tokens, that can be converted into a control signal for the robotic device. For example, the action tokens may represent variables for arm movement (such as one or more of: x, y, z, roll, pitch, yaw, gripper opening), variables for base movement (such as one or more of: x, y, yaw), and variables to switch between modes (such as a variable to switch between controlling arm, controlling base,Attorney Docket No: DEEP-0018-WO-01 or terminating the episode). Each action dimension may be discretized, for example into 256 bins.

[0069] The output may comprise reward parameters that can be optimized by a low-level motioncontroller to determine low-level actuator commands.

[0070] The output may comprise robot policy code expressing functions or feedback loops thatprocess perception outputs and parameterize control primitive APIs. For example, the output may comprise API calls to generate policy code.

[0071] The output may represent candidate robot or end effector trajectories, higher-levelcontrol commands, position, velocity, or force / torque / acceleration data for one or more joints, or electronic control data such as motor control data for example.

[0072] In various implementations, the robot may be simulated in a virtual environment. Theinput may comprise data representing the virtual environment in which the simulated robot operates, for example image data representing the virtual environment.

[0073] The robotic device may take various forms, including but not limited to a telepresencerobot, a robotic arm, a bi-arm robotic device, a humanoid robot or other bipedal robot, a quadraped robot such as a “robot dog”, a wheeled robot, an aquatic robot, and so forth. The robotic device may include control logic. Control logic may take various forms, such as a real time controller, one or more processors, one or more field-programmable gate arrays (“FPGA”), one or more application-specific integrated circuits (“ASIC”), and so forth. In some implementations, the logic may be operably coupled with memory. Memory may take various forms, such as random-access memory (“RAM”), dynamic RAM (“DRAM”), read-only memory (“ROM”), Magnetoresistive RAM (“MRAM”), resistive RAM (“RRAM”), NAND flash memory, and so forth. In some implementations, the control logic may be operably coupled with one or more joints, one or more end effectors, and / or one or more sensors. A joint of a robot may broadly refer to actuators, motors (e.g., servo motors), shafts, gear trains, pumps (e.g., air or liquid), pistons, drives, propellers, flaps, rotors, or other components that may create and / or undergo propulsion, rotation, and / or motion. An end effector may broadly refer to a variety of tools that may be operated by the robotic device in order to accomplish various tasks. For example, an end effector may take the form of a claw with two opposing “fingers” or “digits.” Such a claw is one type of “gripper” known as an “impactive” gripper. The gripper may have more than two digits, for example, three, four or five digits. Other types of grippers may includeAttorney Docket No: DEEP-0018-WO-01 but are not limited to “ingressive” (e.g., physically penetrating an object using pins, needles, etc.), “astrictive” (e.g., using suction or vacuum to pick up an object), or “contigutive” (e.g., using surface tension, freezing or adhesive to pick up object). More generally, other types of end effectors may include but are not limited to drills, brushes, force-torque sensors, cutting tools, deburring tools, welding torches, containers, trays, and so forth. In some implementations, end effector may be removable, and various types of modular end effectors may be installed onto robot. Some robots, such as some telepresence robots, may not be equipped with end effectors.

[0074] In various implementations, a method may be implemented using one or more processorsand may include: assembling, as a first input prompt, representations of a proposed sequence of actions to be performed by a robot to carry out a task and a current state of the robot; processing the first input prompt using a trajectory dynamics diffusion model to generate output that is usable to derive a proposed sequence of predicted states to be reached by the robot upon performance of the sequence of actions by the robot, wherein the proposed sequences of actions and predicted states form a proposed state-action trajectory; sampling one or more alternative state-action trajectories; evaluating the proposed and alternative state-action trajectories; and based on the evaluating, determining one or more actions to be performed by the robot.

[0075] In various implementations, determining the one or more actions may include selectingan optimized sequence of state-action trajectories from the proposed and alternative state-action trajectories based on the evaluating. In various implementations, the method may further include training the trajectory dynamics diffusion model based on the optimized state-action trajectory.

[0076] In various implementations, the method may include generating a control signal forcontrolling the robot based on one or more of the one or more actions to be performed by the robot. In various implementations, the method may further include operating one or more robots based on one or more of the sequence of actions.

[0077] In various implementations, the first input prompt may be further assembled to include ahistorical state-action trajectory of the robot leading up to the current state of the robot. In various implementations, the historical state-action trajectory may include prior sequences of states and actions of the robot that preceded the current state.

[0078] In various implementations, the method may include, prior to assembling the first inputprompt: assembling, as an initial input prompt, representations of: the task to be carried out byAttorney Docket No: DEEP-0018-WO-01 the robot, the current state of the robot, and an immediate history of the robot leading up to the current state of the robot; and processing the initial input prompt using a policy proposal diffusion model to generate initial output that is usable to derive the proposed sequence of actions to be performed by the robot. In various implementations, the immediate history of the robot may include prior sequences of states and actions of the robot that preceded the current state.

[0079] In various implementations, the output may include the proposed sequence of multiplepredicted states to be reached by the robot upon performance of the sequence of actions by the robot. In various implementations, the sampling may be based on the proposed state-action trajectory. In various implementations, the sampling may be random. In various implementations, the sampling may include: sampling one or more alternative proposed sequences of actions to be performed by the robot using a policy proposal diffusion model; assembling one or more additional input prompts to include the one or more alternative proposed sequences of actions; and processing the one or more additional input prompts using the trajectory dynamics diffusion model to generate one or more additional sequences of states that, with the alternative proposed sequences of actions, comprise the one or more alternative state-action trajectories.

[0080] In various implementations, the evaluating may include: scoring the proposed state-action trajectory and the one or more alternative state-action trajectories using a value function; and ranking the proposed state-action trajectory and the one or more alternative state-action trajectories based on the scoring.

[0081] In another aspect, a method may be implemented using one or more processors and mayinclude: assembling, as an input prompt, representations of a proposed sequence of actions to be performed by a robot to carry out a task and a current state of the robot; processing the input prompt using a trajectory dynamics diffusion model to generate output that is usable to derive a proposed sequence of predicted states to be reached by the robot upon performance of the sequence of actions by the robot; comparing the proposed sequence of predicted states to a ground truth sequence of states reached by a robot upon performance of the proposed sequence of actions; and based on the comparing, training the trajectory dynamics diffusion model.

[0082] While several implementations have been described and illustrated herein, a variety ofother means and / or structures for performing the function and / or obtaining the results and / or oneAttorney Docket No: DEEP-0018-WO-01 or more of the advantages described herein may be utilized, and each of such variations and / or modifications is deemed to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and / or configurations will depend upon the specific application or applications for which the teachings is / are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific implementations described herein. It is, therefore, to be understood that the foregoing implementations are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, implementations may be practiced otherwise than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent, is included within the scope of the present disclosure.

Claims

Attorney Docket No: DEEP-0018-WO-01 CLAIMS What is claimed is:

1. A method implemented using one or more processors and comprising: assembling, as a first input prompt, representations of a proposed sequence of actions to be performed by a robot to carry out a task and a current state of the robot; processing the first input prompt using a trajectory dynamics diffusion model to generate output that is usable to derive a proposed sequence of predicted states to be reached by the robot upon performance of the sequence of actions by the robot, wherein the proposed sequences of actions and predicted states form a proposed state-action trajectory; sampling one or more alternative state-action trajectories; evaluating the proposed and alternative state-action trajectories; and based on the evaluating, determining one or more actions to be performed by the robot.

2. The method of claim 1, wherein determining the one or more actions comprises selecting an optimized sequence of state-action trajectories from the proposed and alternative state-action trajectories based on the evaluating.

3. The method of claim 2, further comprising training the trajectory dynamics diffusion model based on the optimized state-action trajectory.

4. The method of any of the preceding claims, further comprising generating a control signal for controlling the robot based on one or more of the one or more actions to be performed by the robot.

5. The method of any of the preceding claims, further comprising operating one or more robots based on one or more of the sequence of actions.

6. The method of any of the preceding claims, wherein the first input prompt is further assembled to include a historical state-action trajectory of the robot leading up to the current state of the robot.

7. The method of claim 6, wherein the historical state-action trajectory comprises prior sequences of states and actions of the robot that preceded the current state.

8. The method of any of the preceding claims, further comprising, prior to assembling the first input prompt: assembling, as an initial input prompt, representations of: the task to be carried out by the robot,Attorney Docket No: DEEP-0018-WO-01 the current state of the robot, and an immediate history of the robot leading up to the current state of the robot; and processing the initial input prompt using a policy proposal diffusion model to generate initial output that is usable to derive the proposed sequence of actions to be performed by the robot.

9. The method of claim 8, wherein the immediate history of the robot comprises prior sequences of states and actions of the robot that preceded the current state.

10. The method of any of the preceding claims, wherein the output includes the proposed sequence of multiple predicted states to be reached by the robot upon performance of the sequence of actions by the robot.

11. The method of any of the preceding claims, wherein the sampling is based on the proposed state-action trajectory.

12. The method of any of the preceding claims, wherein the sampling is random.

13. The method of any of the preceding claims, wherein the sampling comprises: sampling one or more alternative proposed sequences of actions to be performed by the robot using a policy proposal diffusion model; assembling one or more additional input prompts to include the one or more alternative proposed sequences of actions; and processing the one or more additional input prompts using the trajectory dynamics diffusion model to generate one or more additional sequences of states that, with the alternative proposed sequences of actions, comprise the one or more alternative state-action trajectories.

14. The method of any of the preceding claims, wherein the evaluating comprises: scoring the proposed state-action trajectory and the one or more alternative state-action trajectories using a value function; and ranking the proposed state-action trajectory and the one or more alternative state-action trajectories based on the scoring.

15. A method implemented using one or more processors and comprising: assembling, as an input prompt, representations of a proposed sequence of actions to be performed by a robot to carry out a task and a current state of the robot;Attorney Docket No: DEEP-0018-WO-01 processing the input prompt using a trajectory dynamics diffusion model to generate output that is usable to derive a proposed sequence of predicted states to be reached by the robot upon performance of the sequence of actions by the robot; comparing the proposed sequence of predicted states to a ground truth sequence of states reached by a robot upon performance of the proposed sequence of actions; and based on the comparing, training the trajectory dynamics diffusion model.

16. A system comprising one or more processors and memory storing instructions that, in response to execution by the one or more processors, cause the one or more processors to perform the method of any of claims 1-15.

17. At least one non-transitory computer-readable medium comprising instructions that, in response to execution by one or more processors, cause the one or more processors to perform the method of any of claims 1-15.

Images