Fast and slow adaptation for language model predictive control and / or guidance

Abstract

Implementations relate to improving the "teachability" of foundation models to generate (e.g., predict) actions based on iterative human feedback. During in-context learning—e.g., a chat session between a human and a robot—the human's natural language may be processed using a generative model to generate candidate future dialogs. Each dialog turn may include a predicted robot action and a predicted human response. The candidate future dialog satisfying a criterion may be selected, and the predicted robot action from the next turn of the selected candidate future dialog may be used to operate a real / simulated robot to perform the first selected robot action. A human natural language evaluation of the robot's operation may once again be processed using the generative model, and the process may repeat until the human expresses approval. In some implementations, a recording of the chat session may be used as training data to fine-tune the generative model.

Description

FAST AND SLOW ADAPTATION FOR LANGUAGE MODEL PREDICTIVE CONTROL AND / OR GUIDANCEATTORNEY REFERENCE: DEEP-0005-WO-01 Background

[0001] Large generative machine learning models such as large language models (LLMs), vision-language models (VLMs), vision-language-action (VLA) models, etc., may be trained on large amounts of diverse data, e.g., near the scale of the Internet. These models are sometimes referred to as “foundation models” due to their general applicability across a wide variety of tasks and domains. For example, the same foundation model may be prompted to generate a piece of source code, compose a poem, summarize a scientific article, etc.Summary

[0002] Some foundation models may be used to generate output that is usable, directly or indirectly, to control a real or simulated robot. For example, given natural language commands, foundation models may be capable of generating robot control data such as robot control policies (e.g., perception-action loops) to be executed directly on robot hardware, and / or reward / cost functions that can be optimized by robot motion controllers. Thus, it may be possible for experts and non-experts alike to interact with robots in a simple and intuitive way.

[0003] However, leveraging foundation models to adapt robots for performing new tasks remains a challenge. Foundation models tend to be slow to learn due to their size, and the training and fine-tuning can be costly, both computationally and in terms of time. Human-robot interactions (HRIs) are often constrained by factors such as context sizes of the foundation models leading to short adaptations and / or simple behaviors not longer than a few steps. With longer and multi-step HRIs and a receding autoregressive horizon, foundation models may not necessarily retain interactions that fall outside of the context window. Consequently, a human may need to repeat the same HRIs when subsequently requesting the robot to perform the same or similar task.

[0004] Implementations described herein relate to teaching foundation models with limited context lengths using iterative feedback. More particularly, but not exclusively, techniques described herein relate to improving the “teachability” of foundation models to generate (e.g., predict) robot actions based on iterative feedback (e.g., from humans or otherwise). Thisincludes not only enabling foundation models to learn from feedback to improve overall task performance, but also to improve the speed at which foundation models adapt to feedback in future interactions. As used herein, “teachability” refers generally to a foundation model’s ability to adapt to feedback, and a “measure of teachability” may refer to a number (e.g., average, median) of corrections N required before a robot leveraging the foundation model succeeds in performing a task. For example, N=Q corresponds to zero-shot task generalization.Brief Description of the Drawings

[0005] Fig. 1 schematically depicts an example environment in which disclosed techniques may be employed, in accordance with various implementations.

[0006] Fig. 2 depicts an example robot, in accordance with various implementations.

[0007] Fig. 3 schematically depicts an example of how components of Fig. 1 may exchange data during in-context learning, in accordance with various implementations.

[0008] Fig. 4 schematically illustrates an example of a chat dialog interface, in accordance with various implementations.

[0009] Fig. 5A, Fig. 5B, and Fig. 5C schematically illustrate another example of a chat dialog interface, in accordance with various implementations.

[0010] Fig. 6 depicts an example method for practicing selected aspects of the present disclosure.

[0011] Fig. 7 schematically depicts an example architecture of a computer system.Detailed Description

[0012] Implementations described herein relate to teaching foundation models with limited context lengths using iterative feedback, e.g., from humans or otherwise. More particularly, but not exclusively, techniques described herein relate to improving the “teachability” of foundation models to generate (e.g., predict) robot actions based on iterative feedback. This includes not only enabling foundation models to learn from feedback to improve overall task performance, but also to improve the speed at which foundation models adapt to feedback in future interactions. As used herein, “teachability” refers generally to a foundation model’s ability toadapt to feedback, and a “measure of teachability” may refer to a number (e.g., average, median) number of corrections N required before a robot leveraging the foundation model succeeds in performing a task. For example, N=0 corresponds to zero-shot task generalization.

[0013] In various implementations, foundation models leveraged for robot control may be adapted to new tasks by employing two complementary types of leaming / adaptation: in-context learning and fine-tuning. In-context learning involves prompting the foundation model during inference to adapt to new tasks. This process can be iterative; the teaching data (i.e., chat sessions) from fast adaptation is used to fine-tune the model during slow adaptation, then the fine-tuned model is used to collect more teaching data for the next round of fine-tuning, and so on.

[0014] In various implementations, one or more demonstrations of the task may be provided as context in the prompt, and / or one or more previous interactions may be provided as context in the prompt. No parameters are learned by the foundation model during in-context learning. Incontext learning is useful for rapid adaptation to new scenarios and tasks, where robot behaviors can be adjusted in real time, for example through human feedback. Accordingly, in-context learning is alternatively referred to herein as “day learning” or “fast adaptation.”

[0015] In some implementations, fast adaptation may involve users providing multi-turn language inputs to guide generative model (e.g., LLM) outputs towards generating reward code to elicit desired robot behavior(s). This is an interactive process - users provide feedback based on observing robot behaviors online, rather than labeling offline generative model data. As noted above, fast adaptation may be driven by in-context learning, where the generative model is conditioned on a prompt that provides the initial tokens in the sequence x1;fc= (xlz. . xfe) and uses the model to complete xk+1.n. In some implementations, an in-context prompt may use “Promptbook” formatting described in “How to prompt your robot: A promptbook for manipulation skills with code as policies,” in IEEE International Conference on Robotics and Automation (ICRA), 2024. Such a prompt may contain a description of the robot embodiment, the available robot APIs, as well as some number (e.g., 1-2) of example episodes (e.g., chat sessions) between the user and generative model, followed by the current chat session, such as shown in the following:# You are a stationary robot arm with a 3-fingered hand.# . . . class Robot:def set_hand(self, obj): def reach(self, obj): def min_L2_dist(self, obj l, obj 2):# Example Session.# Chat Turn #1 : move the red and green things together. reach(obj=‘red’, weight=1.0) min_L2_dist(obj l=‘red’, obj2=‘green’, weight=1.0)

[0016] In various implementations, an existing pre-trained generative model such as the PaLM 2 described in “PaLM 2 Technical Report” (arXiv:2305.10403) may be used. When used with prompts such as that described above, such a generative model yields non-zero initial task success rates given feedback from the user. The code generated within each chat turn can represent a single reward function or multiple reward functions to be sequenced one after the other. Upon terminating a chat session, the interaction data may be saved into a cached dataset to be used for slow adaptation. While the generative model may struggle with certain tasks - slow adaptation (described below) may unlock fast adaptation on these tasks.

[0017] Fast adaptation may benefit from fast inference runtime speeds, so that latencies do not negatively influence the human-robot interactions. Accordingly, in some implementations, the generative model inference may be applied at a rate such as 100 tokens per second, and may return robot reward code expressed with 200 - 300 tokens on average (which amounts to roughly 10-15 lines of code). The median duration for each chat turn is 56s, and the majority of user time is spent observing the robot performing the task in simulation.

[0018] With fine-tuning, by contrast, parameters of the foundation model are learned. Fine- tuning is useful for memorizing repeat phenomena and / or learning general concepts (using vast amounts of training data far exceeding a model’s context length) over time. Accordingly, finetuning is alternatively referred to herein as “night learning” or “slow adaptation.”

[0019] In various implementations, an HRI may be framed as a chat session or dialog between the human and a foundation model such as an LLM or VLM that at least partially controls operation of the robot. The human may provide an initial natural language input (e.g., spoken or typed) that expresses a robot command, such as “give me a high-five,” “do a pushup,” “pick up the chip bag,” etc. The foundation model may respond by generating robot control data that is usable, directly or indirectly, to control the robot. In some implementations, the robot control data may be rendered as output that is perceptible to the human, e.g., as chat output.Additionally or alternatively, in some implementations, the human may be presented e.g., as part of the chat interface) with a selectable element that is operable to cause a real or simulated robot to attempt to complete the robot task.

[0020] During each subsequent dialog turn, the human may provide feedback, e.g., as natural language expressing one or more modifications to the action(s) actually taken by the robot, or as natural language expressing a sub-task to piece together a more complex task. The foundation model may respond with updated robot control data, and the user may once again operate the selectable element to cause the robot to attempt to complete the now-modified task. This process may repeat through as many dialog turns as necessary until the affirmation is received, e.g., from the human or otherwise, that the initial robot command has been fulfilled, or in some cases, where the episode is labeled as a failure, e.g., where a threshold number of dialog turns is exceeded, or a human explicitly indicates episode failure.

[0021] To facilitate more efficient in-context learning, the dialog chat session may be used to formulate a Markov decision process (MDP) in which human inputs are state observations, generated / predicted robot control data are actions, and the set of goal states is where the human affirms that a robot command has been fulfilled. When human-robot interactions are formulated as a partially observable MDP, training a generative model such as a VLM or LLM can be viewed as training a transition dynamics model,

[0022] In some implementations, teachability of the foundation model may be improved using a shortest path planning problem. This shortest path planning problem may be addressed in some implementations by fusing foundation model(s) with other techniques, such as model predictive control (MPC), to form what will be referred to herein as language model predictive control (LMPC). With LMPC, foundation models can be fine-tuned (e.g., night learning) based on previous HRIs, e.g., in the form of logs of past dialog chat sessions between humans and foundation model(s) leveraged by robot(s) using in-context learning, in which previous interactions are kept as input context for subsequent ones.

[0023] In various implementations, an LMPC-trained foundation model may be used to predict future dialogs (or “imagined rollouts”) between humans and foundation models. Each future dialog or imagined rollout may include one or more turns of predicted robot control data, followed by a human’s expected feedback to robot operation based on the robot control data.Each time the human was required to modify the robot’s behavior by providing feedback, an additional dialog turn is added.

[0024] In some implementations, the foundation model may be applied to various data (e.g., the original robot command, robot control generated thus far, received human feedback(s)) to sample multiple different future dialogs that might occur based on the dialog so far. For example, techniques such as decoding with non-zero temperature sampling may be used to sample multiple different candidate future dialogs. Each candidate future dialog may be generated in parallel. Each candidate future dialog may have some number of turns. Each turn may include data indicative of (1) a predicted robot action, and (ii) a predicted human response to the predicted robot action. The predicted robot action may include, for instance, robot control data. The predicted human response to the predicted robot action may include, for instance, a natural language statement from the user instructing the robot how to alter its performance to better fulfill the original robot command. In various implementations, each candidate future dialog may end at an instance of predicted human feedback affirming fulfillment of the original robot command (e.g., “good job,” “looks good,” “task completed successfully,” etc.).

[0025] In various implementations, the candidate future dialog that best satisfies one or more criteria may be selected. In some implementations, these criteria may include the candidate future dialog having the fewest numbers of turns until predicted human affirmation of robot command fulfillment. Put another way, whichever future dialog is predicted to result in human satisfaction the fastest (e.g., with the fewest dialog turns) may be selected. Once selected, the predicted robot action (e.g., robot control code) may be obtained, as a next selected robot action, from a “next” turn of the selected future candidate dialog. In various implementations, a real or simulated robot may then be operated (e.g., automatically, or when the human operates a selectable element in the chat application) to perform this next selected robot action.

[0026] The human may provide feedback, and then the process may repeat, with the just- provided human feedback being processed using the foundation model to predict a next plurality of candidate future dialogs. This may continue until, for instance, a human response is received that expresses affirmation that the robot fulfilled the original robot command. By selecting the “best” candidate future dialog at each turn, the process of iteratively approaching fulfillment of the original robot command may be shortened. Moreover, the dialog may be captured as training data that can then be used during night learning to train the foundation model, so that itbecomes, over time, better able to fulfill novel robot commands in fewer turns of dialog. During training, the foundation model is fine-tuned to autoregressively complete the dialog. For example, the input to the foundation model is a prompt with the initial user instruction, and the target is the remaining dialog.

[0027] In the above-described examples, future dialogs with relatively long horizons-e.g. , multiple turns-are predicted and used to guide in-context learning. However, this is not meant to be limiting. In some implementations, a predicted candidate future dialog may have a shorter horizon, e.g., as little as one predicted action, or the single action and a single predicted response to the action (and the response may not necessarily be affirmation that the initial command was fulfilled). Dialogs built using these shorter prediction horizons may be recorded and used as training examples to fine-tune the generative model.

[0028] In the above-described example, a plurality of candidate future dialogs are predicted and used to guide in-context learning, with the resulting dialog captured as training data. However, this is not meant to be limiting. In some implementations, a single predicted action may be predicted by the generative model at each turn, with the resulting dialogs recorded and additionally or alternatively used as training examples to fine-tune the generative model.

[0029] In various implementations, supervised fine-tuning (SFT) may be applied to foundation models during slow adaptation to improve their teachability. One such fine-tuning approach mentioned previously is LMPC. Different variations of LMPC may be used in various implementations. The first is referred to herein as “LMPCRollouts,” where, given the current chat session (system prompt and current chat history), the foundation model is trained to autoregressively predict the rest of the chat session (sequence of otand af’s, until r). During training, the input to the foundation model is the system prompt with the initial user instruction. The target is the remaining chat session. The system prompt may be included in the inputs because different robot embodiments have different system prompts (robot APIs), and this allows the foundation model generation to support different robot APIs at inference time.

[0030] Since LLM training uses an auto-regressive training loss, training the LLM to predict the entire chat session effectively also trains it to predict the remaining portions of a chat session, conditioned on the current portion. LMPC-Rollouts may be trained on successful chat sessions. During inference, rollouts may be sampled from the foundation model some number (e.g., 8) of times with non-zero temperature sampling (next-token decoding). This process may beparallelizable across multiple processors. From these samples, the trajectory with the fewest predicted timesteps may be chosen until termination (r= 1). The first action of that trajectory may then be returned.

[0031] If no predicted chat session terminates, then af+i may be selected at random from the available trajectories. This process is then repeated given new human input for every chat turn, and it is similar to classic receding horizon control to search for the next best action. Intuitively, LMPC-Rollouts can be thought of as training the foundation model via human-robot interaction as a form of chain-of-thought — rather than cloning successful code, LMPC learns the process of getting to the correct code, and accelerating it via search at inference time.

[0032] Another variation of LMPC that may be used with selected implementations of the present disclosure will be referred to herein as “LMPC-Skip.” LMPC-Skip may have the same training input as LMPC-Rollouts, but different targets. LMPC-Skip may be trained to predict only the last action, skipping predictions of the interim trajectory, and may also be trained only on successful chat sessions. Skipping encourages the fine-tuned model to predict the final correct code as soon as possible. However, because LMPC-Skip is not trained on nor does it model intermediate interactions with the user, it may be less responsive to corrective feedback. During inference, LMPC-Skip’ s generation may be conditioned on the system prompt with the chat session so far, and only query the model once to generate a response.

[0033] Foundation models trained at the scale of the entire Internet often have hundreds of billions of parameters, and are continuing to grow larger. Accordingly, training and applying foundation models to data can be enormously costly in terms of computational resources, time, and / or energy expenditure. However, many of the tasks foundation models can perform, such as composing poetry, summarizing texts, etc., are irrelevant to robot control. Accordingly, in some implementations, a smaller “specialist” model having far fewer parameters — millions, tens of millions, hundreds of millions, billions, tens of billions, etc. — may be fine-tuned specifically for tasks related to controlling robots, e.g., in some cases being trained with less data associated with non-robot-related tasks than larger foundation models.

[0034] For example, the foundation model may be used during in-context learning as described above to generate the candidate future dialogs as described above. The specialist model (which may also be a generative model such as a LLM, transformer, etc.) may be fine-tuned based on the dialogs generated during in-context learning, instead of the larger foundation model. Themore the specialist model is fine-tuned in this manner, the more capable it becomes at adapting to new robot tasks, while also being less computationally expensive to apply than the larger foundation model.

[0035] Additionally or alternatively for example, a second generative model (which may also be a LLM, transformer, etc. may be used to generate dialogs using in-context learning, which are then used to fine-tune the generative model. In particular, the human may provide an initial natural language input (e.g., spoken or typed) that expresses a robot command to the second generative model. The second generative model may respond by generating robot control data that is usable, directly or indirectly, to control the robot. During each subsequent dialog turn, the human may provide feedback. The second generative model may respond with updated robot control data, and the user may once again provide feedback. This process may repeat through as many dialog turns as necessary until the human affirms that the initial robot command has been fulfilled, or in some cases, where the episode is labeled as a failure, e.g., where a threshold number of dialog turns is exceeded, or a human explicitly indicates episode failure. The resulting dialogs are captured as training data. The generative model (which may be a smaller model having fewer parameters than the second generative model) may then be fine-tuned based on the captured dialogs generated using the second generative model.

[0036] To further improve machine learning model teachability with fine-tuning, in some implementations, generative model outputs may be conditioned, during training and / or inference, on the specific user that is controlling the robot. For training, the input prompt may be assembled to include an identity (e.g., a unique identifier) of the user who generated the chat session. In some implementations, “top” or “expert” users may be autonomously identified from the training dataset and may be assigned a special ID “top-user" or “expert.” During inference, the machine learning model may be conditioned with the “top-user" labels.

[0037] Top-user conditioning in the present context may be interpreted as conditioning the machine learning model to generate the distribution of observations ot(expected human inputs) and actions at(expected code outputs) closest to the top Tth (e.g., 75th) percentile of users. Intuitively, if observations are viewed as a partial noisy representation of the true (user) state (or intent, during teaching), then different user proficiency levels can correspond to a varying amount of noise (e.g., higher proficiency is less noise), to which conditioning on top-users prompts the generative model to generate rollouts with less noise.

[0038] In various implementations, top or expert users may be identified as the top % (e.g., 10%, 20%, 25%, 30%, etc.) of users by their user performance score. This performance score may be, for instance, the average of a user’s task success rate weighted by task difficulty. Task difficulty may be the task’s failure rate across all users in some cases. Assume there are N tasks and K users. Let s(n, k) denote the success rate of the / / th task across for the th user, c(n, k) denote the number of times the Ath user performed the / / th task, and c(n, k) = l(c(n, k) > 1) indicate whether or not the Ath user has performed the / / th task. Due to practical constraints, c(n, k) = 1 for many user-task pairs.

[0039] In some implementations, a task difficulty rating d(ri) may be defined as one minus the average task success rate across all users:In various implementations, the user performance score may then be defined as an average success rate weighted by the following task difficulty rating:As noted previously, in some implementations, a top percentile (e.g., 75%) of users by these performance scores may be defined as top users.

[0040] While many examples described herein relate to operating robots, this is not meant to be limiting. Techniques described herein may be applicable in other scenarios in which foundation models may be taught using iterative human feedback. For example, instead of operating a robot to perform a high-level task, a user may seek out instructions for performing a complex task themselves (e.g., “show me how to rebuild this engine”). At each turn, the user may be presented with information for carrying out the complex task, such as one or more next steps. In the engine building example, for instance, the user may be presented with an animation (e.g., as an augmented reality (“AR”) overlay rendered using a pair of AR glasses or an AR headset) that suggests which engine component should be added / replaced / modified next, how components should be assembled together, etc. The user may provide feedback on these steps, either by performing them and then offering criticism or praise of the results, or by evaluating the presented steps themselves, without actually attempting them.

[0041] Fig. l is a schematic diagram of components that can cooperate to carry out selected aspects of the present disclosure, in accordance with various implementations. The various components depicted in Fig. 1, particularly those components forming an HRI system 120, arobotic planner system 140 and robot control data system 150, may be implemented using any combination of hardware and software. A robot 100 may be in communication with systems 120, 140, and / or 150, and / or all or parts of systems 120, 140, and / or 150 may be implemented onboard robot 100. The components of Fig. 1 may be communicatively coupled with each other via one or more networks 199, which may include one or more personal area networks, local area networks, and / or wide area networks (e.g., the Internet).

[0042] Robot 100 may take various forms, including but not limited to a telepresence robot (e.g., which may be as simple as a wheeled vehicle equipped with a display and a camera), a robot arm, a multi-pedal robot such as a “robot dog,” an aquatic robot, a wheeled device, a submersible vehicle, an unmanned aerial vehicle (“UAV”), and so forth. One non-limiting example of a mobile robot arm is depicted in Fig. 2. In various implementations, robot 100 may include logic 102. Logic 102 may take various forms, such as a real time controller, one or more processors, one or more field-programmable gate arrays (“FPGA”), one or more applicationspecific integrated circuits (“ASIC”), and so forth. In some implementations, logic 102 may be operably coupled with memory 103. Memory 103 may take various forms, such as randomaccess memory (“RAM”), dynamic RAM (“DRAM”), read-only memory (“ROM”), Magnetoresistive RAM (“MRAM”), resistive RAM (“RRAM”), NAND flash memory, and so forth. In some implementations, a robot controller may include, for instance, logic 102 and memory 103 of robot 100.

[0043] In some implementations, logic 102 may be operably coupled with one or more joints 104-1 to 104-N, one or more end effectors 106, and / or one or more sensors 108-1 to 108- M, e.g., via one or more buses 110. As used herein, “joint” 104 of a robot may broadly refer to actuators, motors (e.g., servo motors), shafts, gear trains, pumps (e.g., air or liquid), pistons, drives, propellers, flaps, rotors, or other components that may create and / or undergo propulsion, rotation, and / or motion. Some joints 104 may be independently controllable, although this is not required. In some instances, the more joints robot 100 has, the more degrees of freedom of movement it may have.

[0044] As used herein, “end effector” 106 may refer to a variety of tools that may be operated by robot 100 in order to accomplish various tasks. For example, some robots may be equipped with an end effector 106 that takes the form of a claw with two opposing “fingers” or “digits.” Such a claw is one type of “gripper” known as an “impactive” gripper. Other types of grippersmay include but are not limited to “ingressive” (e.g., physically penetrating an object using pins, needles, etc.), “astrictive” (e.g., using suction or vacuum to pick up an object), or “contigutive” (e.g., using surface tension, freezing or adhesive to pick up object). More generally, other types of end effectors may include but are not limited to drills, brushes, force-torque sensors, cutting tools, deburring tools, welding torches, containers, trays, and so forth. In some implementations, end effector 106 may be removable, and various types of modular end effectors may be installed onto robot 100, depending on the circumstances. Some robots, such as some telepresence robots, may not be equipped with end effectors. Instead, some telepresence robots may include displays to render visual representations of the users controlling the telepresence robots, as well as speakers and / or microphones that facilitate the telepresence robot “acting” like the user.

[0045] Sensors 108-1 to 108-M may take various forms, including but not limited to 3D laser scanners (e.g., light detection and ranging, or “LIDAR”) or other 3D vision sensors (e.g., stereographic cameras used to perform stereo visual odometry) configured to provide depth measurements, two-dimensional cameras (e.g., RGB, infrared), light sensors (e.g., passive infrared), force sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors (also referred to as “distance sensors”), depth sensors, torque sensors, barcode readers, radio frequency identification (“RFID”) readers, radars, range finders, accelerometers, gyroscopes, compasses, position coordinate sensors (e.g., global positioning system, or “GPS”), speedometers, edge detectors, Geiger counters, and so forth. While sensors 108-1 to 108-M are depicted as being integral with robot 100, this is not meant to be limiting.

[0046] In some implementations, HRI system 120, robotic planner system 140, and / or robotic control data system 150 may include one or more computing devices cooperating to perform selected aspects of the present disclosure. An example of such a computing device is depicted schematically in Fig. 6. In some implementations, one or more of systems 120, 140, and / or 150 may include one or more servers forming part of what is often referred to as a “cloud” infrastructure, or simply “the cloud.” Alternatively, one or more components of systems 120, 140, and / or 150 may be operated by logic 102 of robot 100.

[0047] HRI system 120 may be configured to facilitate HRI between a user 160 and robot 100. HRI system 120 may include a message exchange engine 122 and a transcript engine 124. Message exchange client 122 may facilitate an exchange of chat-style messages from user 160by interacting with an HRI client 164 executing on a client device 162. For example, user 160 may issue natural language commands to HRI client 164, e.g., by typing those commands or speaking the commands and having those utterances transcribed, e.g., at client device 162 or by transcription engine 124. These natural language commands may include initial robot commands to perform relatively high-level tasks, as well as additional natural language commands that respond to a real or simulated robot’s attempt to perform the relatively high- level tasks. For example, user 160 may approve of the robot’s performance, or may issue additional natural language commands that seek to modify the robot’s attempt to perform the high-level task.

[0048] Robotic planner system 140 may be configured to process natural language snippets (e.g., requests, queries, commands, etc.) received from HRI system 120, and / or other data, such as video data, images, etc., using one or more machine learning models 144. Based on this processing, robotic planner system 140, e.g., by way of robotic planner process 142, may generate, as output tokens, data that can be used to operate robot 100. In some implementations, robotic planner system 140 may generate natural language responses and / or other data that describe, for instance, mid-level actions, strategies, and / or rules for carrying out the robot command assigned to robot 100 by user 160. These natural language responses may be provided to robotic control data system 150, which as described below, generates robot control data that is used to control robot 100.

[0049] However, in other implementations, robotic planner system 140 may directly generate robot control data. “Robot control data” may include, for instance, low-level actuator commands (also referred to as “joint commands,” and may include torque commands) that directly control the actuators / joints 104-1 to 104-N of the robot, cartesian commands that specify directi on(s) for an end effector 106, a target robot pose, “reward function code” that specifies reward functions that a motion controller can optimize (e.g., using techniques such as receding horizon optimization) to find optimal low-level actuator commands, selected predefined robot primitives, and so forth. In some cases, robot logic 102 may be configured to convert between joint commands and Cartesian commands, e.g., using forward and / or inverse kinematics.

[0050] The following dialog demonstrates one non-limiting example of how HRI system 120 and robotic planner system 140 may facilitate an HRI between user 160 and robot 100. In thisexample, the robotic planner output takes the form of reward function code that can be optimized by a motion controller or similar mechanism of a robot.User input: Please sit down like a dog.Robotic planner output: set_torso_rewards(height=0.2, pitch=np.deg2rad(-45), ...) User input: No, you need to tilt your body the other way.Robotic planner output:set_torso_rewards(height=0.2, pitch=np.deg2rad(45), ...) User input: Great, now give me a high five!Robotic planner output: set_torso_rewards(height=0.2, pitch=np.deg2rad(45), ...) set_foot_rewards(‘front_left’, height=0.3, forward=0.1, ...) User input: That’s a bit too low for me, raise your paw higher.Robotic planner output: set_torso_rewards(height=0.2, pitch=np.deg2rad(45), ...) set_foot_rewards(‘front_left’, height=0.4, forward=0.1, ...)In this example, robot 100 may take the form of a simulated quadruped robot. User 160 provides the initial natural language command, “Please sit down like a dog.” Robotic planner system responds with reward function code. At this point, the simulated quadruped robot may operate in accordance with this reward function code, and the user may see this simulation rendered in HRI application 164. In response, the user issues another natural language command, “no, you need to tilt your body the other way.” This process may repeat, with previous interactions kept as inputs for subsequent ones, until the user explicitly (user expresses explicit approval) or implicitly (e.g., by issuing no more commands) approves of the robot operation.

[0051] Machine learning model(s) 144 may take various forms, including generative language model(s) such as PaLM, the PaLM 2 model referenced previously, Gemini, BERT, LaMDA, Meena, and / or any other generative language model, such as any other generative model that is encoder-only based, decoder-only based, sequence-to-sequence based and that optionally includes an attention mechanism or other memory. In generative language model form, machine learning model(s) 144 may have hundreds of millions, or even hundreds of billions of parameters. In some implementations, machine learning model(s) 144 may take the form of a multimodal generative model, including but not limited to VLMs such as Flamingo or othermultimodal models such as Gemini. The multimodal generative model may share characteristics with any of the aforementioned architectures, and may be used to process multiple modalities of data, particularly images and text, and / or images and audio for example, to generate one or more modalities of output, such as the aforementioned natural language responses, robot control data, etc.

[0052] Feedback engine 146 may be configured to receive feedback from various sources (human and / or otherwise) and take various actions based on that feedback. To train and / or finetune machine learning model(s) 144, for instance, feedback engine 146 may generate training examples based on recorded dialogs, such as the one presented above, and provide it for a training engine 148. Training engine 148 may use this feedback to train and / or fine-tune machine learning model(s) 144, e.g., using techniques such as stochastic gradient descent, adaptive learning rate algorithms, second-order optimization algorithms such as Newton’s method, regularization techniques such as 11 / L2 regularization or dropout, and so forth.

[0053] In various implementations, during slow adaptation / night learning, training engine 148 may train and / or fine-tune machine learning model(s) 144 based on training examples that are labeled as “expert” or “non-expert”, where training examples are labeled as “expert” or “nonexpert” based on, for example, a success rate. In some implementations, training examples may be labeled as “expert” and “non-expert manually or automatically, for example based on a success rate.

[0054] In some cases, training engine 148 may train the model 144 first using supervised learning and expert-labeled training examples, followed by non-expert-labeled training examples. Put another way, the training examples may include a subset that are labeled as “expert”, and the remainder of the training examples may be labeled as “non-expert”. In some implementations, to fine-tune model 144, the number of training steps may be set to some number of epochs of the available training data, such as two, three, four, five, ten, fifteen, twenty, or more. An adaptive moment estimation (ADAM) may be applied with a learning rate of, for instance, 5 x 10-3, a linear ramp up, and / or cosine decay learning rate scheduler, various batch sizes (e.g., one, two, four, six, eight, . . .) and various context lengths (including but not limited to 4096 tokens).

[0055] In one non-limiting implementation, to collect human teaching data and evaluate teaching performance, thirty -five non-expert users collected 350 chat sessions per day. As usedherein, a “non-expert” means the users are not researchers or engineers, and are not familiar with the underlying generative models (e.g., LLMs) or robot code. When a user interacted with a new chat session, a random robot embodiment and task was sampled, and the user was asked to teach the robot that task. Users were instructed to give natural language feedback on the behavior of the robot for each chat turn, instead of giving technical feedback or giving feedback on the code written by the LLM.

[0056] Data collection was separated into two phases: 1) initial data collection with the base model and 2) subsequent data collection with finetuned models. In phase 2, which model the user interacted with was randomly sampled, and the user did not know which model they were engaging with at the time. This allowed for blind A / B evaluations to minimize biases. For a given model, the data collected can be used for both downstream finetuning and evaluating the model.

[0057] Out of 78 possible tasks, 51 were training tasks (65%) while 27 were testing tasks (35%). While separating tasks into training and testing splits enabled measurement of model generalization performance, it also meant there was less data available for training. To address this and also to make the data distribution robust to user teaching noise, data collection and evaluation of models can be aggregated across multiple (e.g., two) days. Additional data filtering was performed to remove invalid and incorrect data. In total, 299 chat sessions from the initial data collection were made available for fine-tuning. All data collection and most evaluations were performed in simulations, and all models were trained on data obtained with simulation. Finetuned models were separately evaluated on real robots. Across chat sessions, the max total token length was 3900, with 1800 as the median. In total, the training set contained about 500k tokens. For evaluations, approximately 350 chat sessions were collected per model variant, split across all platforms and tasks.

[0058] During inference (e.g., in-context), all or some machine learning model 144 output may be conditioned based on expert labels. For instance, the “expert” label may be incorporated into the input prompt, for all inputs.

[0059] Robotic control data system 150 may be configured to generate robot control data that is operable to control robot 100, e.g., by transmitting robot control data to robot 100. Robotic control data system 150 may include a robot control data generation process 152 that processes natural language snippets, including those that express mid-level actions for carrying outportions of a higher level robot task, to generate robot control data. In some implementations, robot control data generation process 152 may use one or more robot control data machine learning models 154 to generate robot control data. Robot control data machine learning model(s) 154 may take various forms, similar to model(s) 144, such as PaLM, PaLM 2, BERT, LaMDA, Meena, and / or any other generative language model, such as any other generative model that is encoder-only based, decoder-only based, sequence-to-sequence based and that optionally includes an attention mechanism or other memory. An example of a robot control data machine learning model that may be used is described in “RT-1 : Robotics Transformer for Real-World Control at Scale” (arXiv:2212.06817).

[0060] As noted above, in some implementations, robotic planner system 140 may be configured to generate robot control data directly, particularly reward function code that can be used directly by a motion controller to control operation of robot 100. Accordingly, in some such implementations, robotic control data system 150 may be omitted. In some such implementations, an example of a robot control data machine learning model that may be used is described in “RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control” (arXiv:2307.15818).

[0061] Feedback engine 156 may obtain feedback, e.g., from human(s) (e.g., 160) and / or from outcomes of robot 100 being operated using robot control data generated by robot control data generation process 152. Feedback engine 156 may provide feedback data to another training engine 158. Similar to training engine 148, training engine 158 may be configured to train and / or fine-tune machine learning model(s) 154 based on feedback generated by feedback engine 156. In some implementations, training engine 158 and / or training engine 148 may jointly train machine learning model(s) 144 employed by robotic planner process 142 and machine learning model(s) 154 employed by robot control generation process 152, e.g., based on an outcome of robot 100 attempting to carrying out a high-level task.

[0062] Fig. 2 depicts a non-limiting example of a robot 200 in the form of a robot arm. An end effector 206 in the form of a gripper claw is removably attached to a sixth joint 204-6 of robot 200. In this example, six joints 204-1 to 204-6 are indicated. However, this is not meant to be limiting, and robots may have any number of joints. In some implementations, robot 200 may be mobile, e.g., by virtue of a wheeled base 255 or other locomotive mechanism. Robot 200 is depicted in Fig. 2 in a particular selected configuration or “pose.”

[0063] Fig. 3 depicts an example of how data may be exchanged between various components depicted in Fig. 1 to carry out selected aspects of the present disclosure during in-context learning, alternatively referred to herein as “fast adaptation” or “day learning.” In Fig. 3, time runs down the page. Starting at top left, HRI chat client 164 establishes a session with HRI system 120. Once the user (not depicted) issues an initial robot command, data indicative of the robot command (e.g., transcribed text, embeddings, etc.) is provided by HRI client 164 to HRI system 120. HRI system 120 may then provide that data to robotic planner system 140, e.g., as a first input prompt. As noted above, in some implementations, HRI system 120 and / or robotic planner system 142 may include a designation or label in the input prompt identifying the input as “expert.” In some such implementations, HRI system 120 and / or robotic planner system 142 may do the same for all episodes during inference (i.e. all inference-time inputs may be treated as expert).

[0064] Robotic planner system 140 may process the first input prompt to generate a plurality of I candidate future dialogs. Each candidate future dialog i may include some number (JI, J2, . . . , JI) of turns j that are predicted until human affirmation / approval is received that a high-level robot command has been satisfied. Within each turn, the candidate future dialog may include data indicative of a predicted robot action(where i = the candidate dialog and j = the turn) and a predicted human response Rij to the predicted robot action. If an “expert” label was included in the input prompt, then these candidate future dialogs may be conditioned on that expert label.

[0065] In some implementations, including that depicted in Fig. 3, these candidate future dialogs may be provided to HRI system 120. HRI system 120 may then select, from the plurality of candidate future dialogs, the candidate future dialog that best satisfies some criterion. For example, HRI system 120 may select the candidate dialog i having the fewest number j of turns needed until predicted affirmation / approval (e.g., by a human) that the robot command has been fulfilled is generated. In Fig. 3, for instance, HRI system 120 selects the second candidate dialog <A2,I, R2.I>, <A2,2, R2,2>, ■ ■ ■ , <A2,J2, R2,J2> . HRI system 120 may then obtain, as a first selected robot action, the predicted robot action from a next turn of the selected candidate future dialog. In Fig. 3, for instance, the predicted robot action from the next turn of the second candidate dialog is A2,I.

[0066] HRI system 120 may provide data indicative of this selected action A 1 (e.g., reward function code) to robot 100. For example, reward function code representing the selected action A2,I may be used as an intermediate interface between the model 144 of robotic planner system 140 and robotic control data system 150. Put another way, the reward function code bridges the gap between language and low-level robot actions. In some implementations, the reward function code generated by robotic planner system 140 may be converted to low-level robot actions / motions, e.g., by robot control data generation process 152 or by robot 100 itself, using techniques such as the MuJoCO predictive control (MJPC) described in “Predictive Sampling: Real-time Behaviour Synthesis with MuJoCo” (arXiv:2212.00541). MJPC runs a receding horizon trajectory optimization algorithm to find an action sequence that maximizes the reward in real-time. This enables near real-time interactions between users and robots.

[0067] Despite MJPC working well for simulation, deploying it directly in the real-world has several challenges, including high computation cost, and precise state estimation. To mitigate these challenges, in some implementations, a policy distillation pipeline may be implemented to train low-level end-to-end policies that are conditioned on cost terms generated by reward code. This can be thought of as training multi-task policies conditioned on latent task descriptors, which serve as an interface between high and low level control. In various implementations, these distilled policies may be transformer-based neural networks trained using offline (behavioral cloning) and online imitation learning algorithms to predict MJPC actions from proprioceptive observations.

[0068] Referring back to Fig. 1, robot 100 may then perform this selected action ^2,7, e.g, in simulation or in the real world as the case may be. Evidence of this performance (e.g., rendition of the robot’s simulation performance of the action) may be presented to the user via HRI system 120. For instance, HRI chat client 164 may include an animation of robot 100 that depicts robot 100 performing the action A2,I. The user may then operate HRI system 120 to provide a response. This response may include, for instance, additional natural language input that clarifies some aspect of the robot’s performance, provides additional detail about what the robot should have done, identified particular robot behavior as undesirable and / or incorrect, etc. This additional natural language input may be used by HRI system 120 to assemble a second input prompt, e.g., in combination with the original robot command and / or other data, such as sensor data of the robot, etc.

[0069] Similar to before, robotic planner system 140 may process the second input prompt using one or more machine learning model(s) 144 to generate a next plurality of candidate future dialogs. In Fig. 3, this next plurality of candidate future dialogs is depicted in a similar fashion as the first plurality of candidate future dialogs, except the individual predicted actions A and human responses R are modified with a single quotation (‘). HRI system 120 may once again select, as a next selected candidate future dialog, the candidate future dialog of the next plurality of candidate future dialogs that best satisfies the criterion. In this instance, HRI system 120 selects the first candidate future dialog <A ’u, R ’u>, <A ’1,2, R ’i,2>, ...,<AFrom the first candidate future dialog, HRI system 120 obtains, as a new selected robot action, the predicted robot action A’ 1,1 from a next turn of the selected candidate future dialog

[0070] HRI system 120 may then cause robot 100 to be operated to perform the predicted robot action A’ 1,1. Data indicative of robot performance may once again be returned to HRI system 120, which may cause it to be presented at HRI chat client 164. Assuming the user of HRI chat client 164 approves of the robot performance (e.g., by issuing no more commands, or providing explicitly approval such as “that works”), then HRI system 120 may generate, as a training example, a recorded dialog that includes the initial robot command, the first selected robot action ^2,1, the first received human response, and the second selected robot action A’ 1,1. While not shown in Fig. 3, in some implementations, this training example may then be used by training engine 148 as part of “night learning” or “slow adaptation” to fine-tune machine learning model(s) 144.

[0071] In various implementations, in a chat dialog between users and robot planner process 142, user messages may be free-form and up to the users’ discretion. User messages may or may not include instructions, preferences, feedback, etc. In response to each input user message, the robot planner process 142 may output robot control data, e.g., reward function code that can be optimized by a motion planner to control a real or simulated robot. The users can then provide feedback e.g., using natural language or by operating user interface elements indicating satisfaction / dissatisfaction, etc.) about observed robot behavior.

[0072] Each human-robot conversation e.g., chat session) may be goal-driven: users may be asked to teach one task per session and at the end of each session label “success” or “failure” conditioned on whether the user believes the robot to have completed the task. Chat sessions can consist of multiple chat turns (i.e., human-robot input-output pairs) before success. Users mayhave the option to rate individual robot responses as ‘good’ or ‘bad,’ the ratio between which may correlate with task success. User input messages can be corrections or broken-up step-by- step sub-tasks to piece together more complex tasks, and they may or may not be multi-round contextual.

[0073] In some implementations, the chat user interface (UI) (which may be rendered using a standalone application or via another application, such as a web browser) displays a user input box, the message history, as well as a simulated visualization of the robot and its surroundings, e.g., using the Multi-Joint Dynamics with Contact (MuJoCo) physics engine. The human user provides textual input and a generative model such as an LLM is applied to generate a reply to each subsequent user query with executable code. In various implementations, the user can then select a button to either run the code in the simulator to observe the resulting motion, or run it in a real robot. The user can continue to provide feedback (which can be multi-turn contextual) and keep modifying the behavior through the chat UI until the desired robot behavior is achieved. For each chat turn (e.g., language-input, LLM-output pair), the user can also rate the individual robot response as ‘good’ or ‘bad’. These single turn ratings help measure teachability across individual responses, and are strongly correlated to task success. In many cases, users can label the entire chat session as “success” by clicking a success button if the robot succeeded at the task during the conversation, or “failure” if the task does not succeed within some threshold number (e.g., seven) rounds of human input. After labeling a chat session, the chat history UI refreshes, the robot simulator is reset, and the user teaches the robot another task.

[0074] In some implementations, a chat session may be defined as a sequence that consists of the initial system prompt, followed by a sequence of chat turns. Each chat turn may have three parts — the user query, the LLM response, and a user rating of that response. For each chat session, the user teaches the robot one task via language e.g., teach this robot-dog to “sit down and give a high-five.”

[0075] In some implementations, when the chat UI is launched, one of multiple robot embodiments may be selected at random and the operator may be asked to teach a task. The reward code produced by the generative in response is presented to the operator who then has the option to evaluate the behavior in simulation. If a real robot is present, the same flow can be used to transmit the reward function to the robot’s hardware (e.g., for optimization by a motion controller). The next step may be for the operator to rate the reward code produced in responseto their query. This is done for each chat interaction. The teaching procedure continues by refining the task performance by means of additional chat turns. Once the operator is satisfied with the behavior or a maximum number of chat turns is reached, the operator can rate the overall teaching session and save it. At this point, the chat UI may reload and a new embodiment may be sampled.

[0076] As noted previously, “teachability” as used herein refers to the average number of human inputs (e.g., chat turns) n before the robot succeeds at the task. This metric measures how efficiently the robot adapts to human inputs, and n=l is equivalent to a standard zero-shot instruction. To improve teachability is to reduce the number of chat turns n before a desired success rate, and can be viewed as a m eta-1 earning objective. Intuitively, improving teachability of a model should encourage its responsiveness to feedback, as a means to maximize the likelihood of generating the correct behavior (according to the user). Teachability can also reflect how well a model adapts to preferences. For instance, user input “move a bit to the left” might yield different robot behavior modifications depending on the user - a strong meta-leamer (with respect to teachability) is one that can learn this difference to minimize / / , conditioned on who it interacts with.

[0077] Fig. 4 schematically depicts a non-limiting example of a chat dialog interface 450 that is operable by a user to communicate with robotic planner process 142 (represented by the robot icon) to generate reward function code. This reward function code may then be used to control a real or simulated robot 400, which in this example takes the form of a quadruped robot 400 depicted on the right hand side of chat dialog interface 450.

[0078] In the chat dialog interface 450, the user issued the first robot command, “Please sit down like a dog.” This results in robot planner process 142 generating, as a response, the depicted comment “#Reward Function Code” followed by the reward function code, “import numpy as np set_torso(0.2, np.deg2rad(-45)).” The real or simulated robot 400 is depicted on the right hand side of the interface (but could be a separate window and / or interface) having attempted the requested task. However, assuming the robot’s “head” is on the left hand side, the robot 400 has lowered its head towards the ground, instead of its hind end. Accordingly, the user issues the follow up command, “No, tilt the other way.” This results in the robot planner process 142 generating, as a response, the reward function code “set_torso(0.2, np.deg2rad(45)).” As shown at right, the robot 400 has performed this action so that its head is now lifted and its hindend is near the ground, as would be the case with a sitting dog. (It should be noted that in various implementations, only a single view of the robot 400 may be provided at a time, e.g., in a single viewing window, and the view may be updated at each turn to show the robot in a new state / pose. The multiple views shown in Fig. 4 are for illustrative purposes only).

[0079] Next, the user issues the command, “Great, now give me a high five.” This results in the robot planner process 142 generating, as a response, the reward function code “set_torso(0.2, np.deg2rad(-45)),” followed by additional reward function code, “set_feet(‘front_left’, 0.3, 0.1).” As shown at right, the robot 400 has raised its front left leg somewhat. Having judged the quadruped robot 400 as giving too low of a “high five,” the user issues another follow up command, “A bit too low, raise it higher.” While not shown in Fig. 4, the process may continue from this point, e.g., with the robot 400 raising its left “paw” higher to a more appropriate “high five” elevation.

[0080] Figs. 5A-C schematically depict another example chat dialog interface 550 that is operable by a user to communicate with robotic planner process 142 (represented by the robot icon) to generate reward function code. Once again, the reward function code may then be used to control a real or simulated robot 500. In the chat dialog interface 550, the user issued the first robot command, “pick up the apple using the right arm.” This results in robot planner process 142 generating, as a response, the depicted comments (prefaced by a “#”) and reward function code. In some implementations, the comments and reward function code may be generated together as a single Chain-of-Thought. In this instance, multiple reward function code snippets are generated, including one called “min_L2_disf ’ that is configured to cause a low level robot motion controller to generate low level commands selected to minimize the distance between the robot’s right arm and the apple. Subsequent reward functions such as set_target_pos are executed in the sequence they are provided until, as shown in Fig. 5B, the right robot arm has picked up the apple.

[0081] Next, the user instructs the robot to “now hand it over to the left hand.” This results in robotic planner process 142 generating a strategy comment (“# to hand over the apple, bring it close to the left gripper”) and the reward function code, “min_L2_dist,” with the apple and the left hand of robot 500 being the arguments. Execution of this reward function code, e.g., by a motion planner processor optimizing the reward function, eventually results in the right hand of robot 500 handing the apple to the left hand of robot 500, as depicted in Fig. 5C.

[0082] Referring now to Fig. 6, an example method 600 of practicing selected aspects of the present disclosure is described. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, including those depicted in Fig. 1. Moreover, while operations of method 600 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted or added.

[0083] At block 602, the system, e.g., by way of HRI system 120 or robotic planner system 140, may assemble, as a “current” input prompt for a generative model (e.g., 144), a natural language snippet that expresses a robot command. For example, a user of HRI client 164 may type, or speak and have transcribed using STT processing, a natural language snippet that expresses a robot command such as “give me a high five.” This current input prompt may be assembled with other data as well. In some implementations, for instance, the current input prompt may be assembled to include a description of the particular real or simulated robot being controlled, an available robot application programming interface (API) of the real or simulated robot being controlled, and / or some number of example episodes (e.g., chat sessions). If the current dialog session has advanced beyond the initial user command, then the current input prompt may also include data indicative of the current dialog session turns up to this point.

[0084] At block 604, the system, e.g., by way of robotic planner process 142, may process the current input prompt using the generative model (e.g., 144) to generate a plurality of candidate future dialogs. Each candidate future dialog may have some respective number (e.g., Jin Fig. 3) of turns. And in each turn, the candidate future dialog may include data indicative of: a predicted robot action (Ajj in Fig. 3); and a predicted response (Rij in Fig. 3, can be a predicted human response or another predicted response, e.g., from the robot) to the predicted robot action.

[0085] At block 606, the system, e.g., by way of HRI system 120 or robotic planner system 140, may select, from the plurality of candidate future dialogs, the candidate future dialog that best satisfies a criterion. For example, HRI system 120 or robotic planner system 140 may select the candidate future dialog that includes the fewest turns until human approval / affirmation is predicted. Intuitively, the desire is to find the predicted future dialog that results in human satisfaction most quickly. At block 608, the system, e.g., by way of HRI system 120 or robotic planner system 140, may obtain, as a “current” selected robot action, the predicted robot actionfrom a next turn of the selected candidate future dialog. Put another way, the action that would be performed first in the selected candidate future dialog is selected.

[0086] At block 610, the system, e.g., by way of HRI system 120, robotic planner system 140, and / or robot control data generation system 150, may cause (simulated and / or real) robot 100 to be operated to perform the current selected robot action. For example, in implementations in which the “current” selected action includes reward function code, robotic planner system 140 may provide the reward function code to a motion planner of robot 100, which may then optimize the reward function code to generate low level robot commands. If method 600 is being applied during inference with the purpose of operating the robot in a more efficient manner enabled by techniques described herein, then in some cases, method 600 may end.

[0087] However, if method 600 is being used during a training phase 699, then method 600 may continue to block 612. At block 612, the system, e.g., by way of HRI system 120, may receive a response (human or otherwise) to performance of the current selected robot action. For example, the user may provide natural language input that rejects or approves the performance, or may not provide input at all (which may indicate approval in some implementations). Additionally or alternatively, robot success or failure may be detected automatically, e.g., by prompting a VLM to process robotic vision data to determine whether the user’s stated goal appears to have been achieved, if anything in the environment was damaged (which may indicate failure), etc.

[0088] Notably, in various implementations, the remainder of the selected future candidate dialog, including the predicted human response to the current selected robotic action, and any downstream predicted robot actions and / or human responses, may be discarded. Instead, method may proceed to block 614 to determine whether the response (human or otherwise) expresses a request for additional action or affirmation that the robot command has been fulfilled. If the robot command is affirmed to have been fulfilled, e.g., by the human providing explicit approval (e.g., using natural language, clicking on a “thumbs up” symbol or similar) or by not rejecting the robot’s performance, or by success being detected via VLM-based processing of vision data, then method 600 may proceed to block 616.

[0089] At block 616, the system, e.g., by way of HRI system 120 or training engine 148, may generate, as a training example, recorded dialog that includes data indicative of the initial robot command, any human responses received at block 612, and any robot actions selected atblock 608. At block 618, the system, e.g., by way of training engine 148, may fine-tune the generative model using the training example generated at block 616.

[0090] It may be the case, however, that the robot’s performance fails to fulfill the initial robot command. This may be because the initial robot command is not something that has been trained into the machine learning model 144. Accordingly, at block 614 of Fig. 6, if the response (human or otherwise) requests additional action, then method 600 may proceed to block 620. At block 620, the system, e.g., by way of HRI system 120 or robotic planner system 140, may assemble, as the next “current” input prompt, a new input prompt that includes data indicative of a new natural language snippet expressed in the response that was received at block 612. This process may repeat until the user expresses satisfaction that the robot command has been fulfilled, in which case blocks 616-618 are performed and method 600 ends.

[0091] Fig. 7 is a block diagram of an example computer system 710. Computer system 710 typically includes at least one processor 714 which communicates with a number of peripheral devices via bus subsystem 712. These peripheral devices may include a storage subsystem 724, including, for example, a memory subsystem 725 and a file storage subsystem 726, user interface output devices 720, user interface input devices 722, and a network interface subsystem 716. The input and output devices allow user interaction with computer system 710. Network interface subsystem 716 provides an interface to outside networks and is coupled to corresponding interface devices in other computer systems.

[0092] User interface input devices 722 may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touch screen incorporated into the display, audio input devices such as voice recognition systems, microphones, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and ways to input information into computer system 710 or onto a communication network.

[0093] User interface output devices 720 may include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices. The display subsystem may include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual display such as via audio output devices. In general, use of theterm "output device" is intended to include all possible types of devices and ways to output information from computer system 710 to the user or to another machine or computer system.

[0094] Storage subsystem 724 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 724 may include the logic to perform selected aspects of method 600, and / or to implement one or more aspects of robot 100 or systems 120, 140, and / or 150. Memory 725 used in the storage subsystem 724 can include a number of memories including a main randomaccess memory (RAM) 730 for storage of instructions and data during program execution and a read only memory (ROM) 732 in which fixed instructions are stored. A file storage subsystem 726 can provide persistent storage for program and data files, and may include a hard disk drive, a CD-ROM drive, an optical drive, or removable media cartridges. Modules implementing the functionality of certain implementations may be stored by file storage subsystem 726 in the storage subsystem 724, or in other machines accessible by the processor(s) 714.

[0095] Bus subsystem 712 provides a mechanism for letting the various components and subsystems of computer system 710 communicate with each other as intended. Although bus subsystem 712 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0096] Computer system 710 can be of varying types including a workstation, server, computing cluster, blade server, server farm, smart phone, smart watch, smart glasses, set top box, tablet computer, laptop, or any other data processing system or computing device. Due to the everchanging nature of computers and networks, the description of computer system 710 depicted in Fig. 7 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computer system 710 are possible having more or fewer components than the computer system depicted in Fig. 7.

[0097] In some implementations, a computer implemented method may be provided that includes: assembling a natural language snippet that expresses a robot command into a first input prompt for a generative model; processing the first input prompt using the generative model to generate a plurality of candidate future dialogs, each candidate future dialog having a respective number of turns, wherein for each turn, the candidate future dialog includes data indicative of: a predicted robot action, and a predicted response (e.g., from a human or otherwise) to the predicted robot action; selecting, from the plurality of candidate future dialogs,the candidate future dialog that best satisfies a criterion; obtaining, as a first selected robot action, the predicted robot action from a turn (e.g., the next or first turn if there are multiple turns) of the selected candidate future dialog; and causing a robot to be operated to perform the first selected robot action.

[0098] In various implementations, the method may include: receiving a first human response to performance of the first selected predicted robot action by the robot, wherein the first human response comprises a new natural language snippet that expresses an additional action to be performed by the robot to fulfill the robot command; assembling a second input prompt that includes data indicative of the new natural language snippet; processing the second input prompt using the generative model to generate a next plurality of candidate future dialogs; selecting, as a next selected candidate future dialog, the candidate future dialog of the next plurality of candidate future dialogs that best satisfies the criterion; obtaining, as a second selected robot action, the predicted robot action from a turn (e.g., the first or next turn if there are multiple turns) of the next selected candidate future dialog; and causing the robot to be operated to perform the second selected robot action.

[0099] In various implementations, the method may include: receiving a second human response that expresses affirmation that performance of the second selected robot action by the robot fulfilled the robot command; and generating, as a training example, a recorded dialog that includes the robot command, the first selected robot action, the first received human response, and the second selected robot action. In various implementations, the method may further include fine-tuning a machine learning model based on the training example. In various implementations, the machine learning model may be the generative model or a separate specialist generative model having fewer parameters than the generative model.

[0100] In various implementations, the fine-tuning may be performed during a time interval in which the machine learning model is offline. In various implementations, the time interval may be overnight.

[0101] In various implementations, the method may include labeling the training example as “expert” based on, for example, a success rate. In various implementations, the machine learning model may be fine-tuned based on a plurality of training examples that includes the training example, and wherein a subset of the plurality of training examples are labeled as “expert” and the remainder of the plurality of training examples are labeled as “non-expert”. Invarious implementations, the method may include assembling an “expert” label into the first input prompt to condition the generative model.

[0102] In various implementations, the data indicative of the predicted robot actions may include reward function code. In some implementations, a receding horizon trajectory optimization algorithm may be used based on this reward function code to find an action sequence that maximizes the reward.

[0103] In various implementations, the criterion may include the fewest number of turns among the plurality of candidate future dialogs. In various implementations, the robot may be a simulated robot operating in a virtual environment. In various implementations, the simulated environment may be rendered in a graphical user interface (GUI) of an HRI chat application through which a human provided the natural language snippet.

[0104] In various implementations, the robot may be a physical robot operating in a physical environment. In various implementations, the method may include outputting the predicted robot action to a user that provided the natural language snippet.

[0105] In various implementations, the generative model may be an LLM and / or a VLM.

[0106] In another aspect, a method may be implemented using one or more processors, and may include: assembling a natural language snippet that expresses a command into a first input prompt for a generative model; processing the first input prompt using the generative model to generate a plurality of candidate future dialogs, each candidate future dialog having a respective number of turns, wherein for each turn, the candidate future dialog includes data indicative of a predicted action, and a predicted response e.g., human or otherwise) to the predicted action; selecting, from the plurality of candidate future dialogs, the candidate future dialog that best satisfies a criterion; obtaining, as a first selected action, the predicted action from a turn (e.g., the next turn if there are multiple turns) of the selected candidate future dialog; and causing data indicative of the first selected action to be rendered at one or more output devices.

[0107] In various implementations, the method may include: receiving a first human response to of the first selected action, wherein the first human response comprises a new natural language snippet that expresses an additional action to be performed to fulfill the command; assembling a second input prompt that includes data indicative of the new natural language snippet; processing the second input prompt using the generative model to generate a next plurality of candidate future dialogs; selecting, as a next selected candidate future dialog, the candidatefuture dialog of the next plurality of candidate future dialogs that best satisfies the criterion; obtaining, as a second selected action, the predicted action from a turn (e.g., the next or first turn if there are multiple turns) of the next selected candidate future dialog; and causing data indicative of the second selected action to be rendered at one or more of the output devices.

[0108] In another aspect, a method may be implemented using one or more processors and may include: assembling a natural language snippet that expresses a robot command into a first input prompt for a generative model; processing the first input prompt using the generative model to generate a plurality of candidate future dialogs, each candidate future dialog having a respective number of turns, wherein for each turn, the candidate future dialog includes data indicative of: a predicted robot action, and a predicted response (e.g., human or otherwise) to the predicted robot action; selecting, from the plurality of candidate future dialogs, the candidate future dialog that best satisfies a criterion; obtaining, as a first selected robot action, the predicted robot action from a turn (e.g., the next or first turn if there are multiple turns) of the selected candidate future dialog; receiving a first human response to the first selected predicted robot action, wherein the first human response comprises a new natural language snippet that expresses an additional action to be performed by the robot to fulfill the robot command; assembling a second input prompt that includes data indicative of the new natural language snippet; processing the second input prompt using the generative model to generate a next plurality of candidate future dialogs; selecting, as a next selected candidate future dialog, the candidate future dialog of the next plurality of candidate future dialogs that best satisfies the criterion; obtaining, as a second selected robot action, the predicted robot action from a turn (e.g., the next or first turn if there are multiple turns) of the next selected candidate future dialog; detecting approval by the human approved of the second selected robot action; and generating, as a training example, a recorded dialog that includes the robot command, the first selected robot action, the first received human response, and the second selected robot action.

[0109] In various implementations, the method may include fine-tuning a machine learning model based on the training example. In various implementations, the machine learning model may be the generative model. In various implementations, the machine learning model may include a separate specialist generative model having less parameters than the generative model.

[0110] Other implementations may include a non-transitory computer readable storage medium storing instructions executable by a processor to perform a method such as one or more of themethods described above. Yet another implementation may include a control system including memory and one or more processors operable to execute instructions, stored in the memory, to implement one or more modules or engines that, alone or collectively, perform a method such as one or more of the methods described above.[OHl] In another aspect, a method may be implemented using one or more processors and may include: assembling a natural language snippet that expresses a robot command into a first input prompt for a generative model; processing the first input prompt using the generative model to generate a first robot action; receiving a first human response to the first selected predicted robot action, wherein the first human response comprises a new natural language snippet that expresses an additional action to be performed by the robot to fulfill the robot command; assembling a second input prompt that includes data indicative of the new natural language snippet; processing the second input prompt using the generative model to generate a second selected robot action; detecting approval by the human approved of the second selected robot action; and generating, as a training example, a recorded dialog that includes the robot command, the first selected robot action, the first received human response, and the second selected robot action.

[0112] Other implementations may include a transitory or non-transitory computer readable storage medium storing instructions executable by a processor to perform a method such as one or more of the methods described above. Yet another implementation may include a control system including memory and one or more processors operable to execute instructions, stored in the memory, to implement one or more modules or engines that, alone or collectively, perform a method such as one or more of the methods described above.

[0113] It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.

[0114] While several implementations have been described and illustrated herein, a variety of other means and / or structures for performing the function and / or obtaining the results and / or one or more of the advantages described herein may be utilized, and each of such variations and / or modifications is deemed to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meantto be exemplary and that the actual parameters, dimensions, materials, and / or configurations will depend upon the specific application or applications for which the teachings is / are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific implementations described herein. It is, therefore, to be understood that the foregoing implementations are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, implementations may be practiced otherwise than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent, is included within the scope of the present disclosure.

Claims

CLAIMSWhat is claimed is:

1. A method implemented using one or more processors and comprising: assembling a natural language snippet that expresses a robot command into a first input prompt for a generative model; processing the first input prompt using the generative model to generate a plurality of candidate future dialogs, each candidate future dialog having a respective number of turns, wherein for each turn, the candidate future dialog includes data indicative of: a predicted robot action, and a predicted response to the predicted robot action; selecting, from the plurality of candidate future dialogs, the candidate future dialog that best satisfies a criterion; obtaining, as a first selected robot action, the predicted robot action from a next turn of the selected candidate future dialog; and causing a robot to be operated to perform the first selected robot action.

2. The method of claim 1, further comprising: receiving a first human response to performance of the first selected predicted robot action by the robot, wherein the first human response comprises a new natural language snippet that expresses an additional action to be performed by the robot to fulfill the robot command; assembling a second input prompt that includes data indicative of the new natural language snippet; processing the second input prompt using the generative model to generate a next plurality of candidate future dialogs; selecting, as a next selected candidate future dialog, the candidate future dialog of the next plurality of candidate future dialogs that best satisfies the criterion; obtaining, as a second selected robot action, the predicted robot action from a next turn of the next selected candidate future dialog; and causing the robot to be operated to perform the second selected robot action.

3. The method of claim 2, further comprising: receiving a second human response that expresses affirmation that performance of the second selected robot action by the robot fulfilled the robot command; andgenerating, as a training example, a recorded dialog that includes the robot command, the first selected robot action, the first received human response, and the second selected robot action.

4. The method of claim 3, further comprising fine-tuning a machine learning model based on the training example.

5. The method of claim 4, wherein the machine learning model comprises the generative model.

6. The method of claim 4 or 5, wherein the machine learning model comprises a separate specialist generative model having less parameters than the generative model.

7. The method of any of claims 4-6, wherein the fine-tuning is performed during a time interval in which the machine learning model is offline.

8. The method of claim 7, wherein the time interval is overnight.

9. The method of any of claims 4-8, further comprising labeling the training example based on a success rate.

10. The method of claim 9, wherein the machine learning model is fine-tuned based on a plurality of training examples that includes the training example, and wherein a subset of the plurality of training examples are labeled as “expert” and the remainder of the plurality of training examples are labeled as “non-expert”.

11. The method of claim 10, further comprising assembling an “expert” label into the first input prompt to condition the generative model.

12. The method of any of the preceding claims, wherein the data indicative of the predicted robot actions comprises reward function code.

13. The method of any of the preceding claims, wherein the criterion comprises the fewest number of turns among the plurality of candidate future dialogs.

14. The method of any of the preceding claims, wherein the robot is a simulated robot operating in a virtual environment.

15. The method of claim 14, wherein the simulated environment is rendered in a graphical user interface (GUI) of a human-robot-interaction (HRI) chat application through which a human provided the natural language snippet.

16. The method of any of the preceding claims, wherein the robot is a physical robot operating in a physical environment.

17. The method of any of the preceding claims, further comprising outputting the predicted robot action to a user that provided the natural language snippet.

18. The method of any of the preceding claims, wherein the generative model comprises a large language model (LLM).

19. The method of any of the preceding claims, wherein the generative model comprises a vision-language model (VLM).

20. A method implemented using one or more processors and comprising: assembling a natural language snippet that expresses a command into a first input prompt for a generative model; processing the first input prompt using the generative model to generate a plurality of candidate future dialogs, each candidate future dialog having a respective number of turns, wherein for each turn, the candidate future dialog includes data indicative of: a predicted action, and a predicted response to the predicted action; selecting, from the plurality of candidate future dialogs, the candidate future dialog that best satisfies a criterion; obtaining, as a first selected action, the predicted action from a next turn of the selected candidate future dialog; and causing data indicative of the first selected action to be rendered at one or more output devices.

21. The method of claim 20, further comprising: receiving a first human response to of the first selected action, wherein the first human response comprises a new natural language snippet that expresses an additional action to be performed to fulfill the command; assembling a second input prompt that includes data indicative of the new natural language snippet; processing the second input prompt using the generative model to generate a next plurality of candidate future dialogs; selecting, as a next selected candidate future dialog, the candidate future dialog of the next plurality of candidate future dialogs that best satisfies the criterion;obtaining, as a second selected action, the predicted action from a next turn of the next selected candidate future dialog; and causing data indicative of the second selected action to be rendered at one or more of the output devices.

22. A method implemented using one or more processors and comprising: assembling a natural language snippet that expresses a robot command into a first input prompt for a generative model; processing the first input prompt using the generative model to generate a plurality of candidate future dialogs, each candidate future dialog having a respective number of turns, wherein for each turn, the candidate future dialog includes data indicative of: a predicted robot action, and a predicted human response to the predicted robot action; selecting, from the plurality of candidate future dialogs, the candidate future dialog that best satisfies a criterion; obtaining, as a first selected robot action, the predicted robot action from a next turn of the selected candidate future dialog; receiving a first human response to the first selected predicted robot action, wherein the first human response comprises a new natural language snippet that expresses an additional action to be performed by the robot to fulfill the robot command; assembling a second input prompt that includes data indicative of the new natural language snippet; processing the second input prompt using the generative model to generate a next plurality of candidate future dialogs; selecting, as a next selected candidate future dialog, the candidate future dialog of the next plurality of candidate future dialogs that best satisfies the criterion; obtaining, as a second selected robot action, the predicted robot action from a next turn of the next selected candidate future dialog; detecting approval by the human approved of the second selected robot action; and generating, as a training example, a recorded dialog that includes the robot command, the first selected robot action, the first received human response, and the second selected robot action.

23. The method of claim 22, further comprising fine-tuning a machine learning model based on the training example.

24. The method of claim 23, wherein the machine learning model comprises the generative model.

25. The method of claim 23 or 24, wherein the machine learning model comprises a separate specialist generative model having less parameters than the generative model.

26. A method implemented using one or more processors, comprising: assembling a natural language snippet that expresses a robot command into a first input prompt for a generative model; processing the first input prompt using the generative model to generate a first robot action; receiving a first human response to the first selected predicted robot action, wherein the first human response comprises a new natural language snippet that expresses an additional action to be performed by the robot to fulfill the robot command; assembling a second input prompt that includes data indicative of the new natural language snippet; processing the second input prompt using the generative model to generate a second selected robot action; detecting approval by the human approved of the second selected robot action; and generating, as a training example, a recorded dialog that includes the robot command, the first selected robot action, the first received human response, and the second selected robot action.

27. The method of claim 26, further comprising fine-tuning a machine learning model based on the training example.

28. The method of claim 27, wherein the machine learning model comprises the generative model.

29. The method of claim 27 or 28, wherein the machine learning model comprises a separate specialist generative model having less parameters than the generative model.

30. A system comprising one or more processors and memory storing instructions that, in response to execution by the one or more processors, cause the one or more processors to perform the method of any of the preceding claims.

31. At least one non-transitory computer-readable medium comprising instructions that, in response to execution by one or more processors, cause the one or more processors to perform the method of any one of claims 1-29.

Images