Natural language control of robot

By combining language model output and robot skill evaluation and selecting suitable skills, the problem of insufficient real-world experience in FF NL input is solved, and the robot's task is successfully completed in a specific environment.

CN118900751APending Publication Date: 2024-11-05GOOGLE LLC
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202380031168.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-31
Filing Date
2023-03-30
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

Existing language models (LMs) lack real-world experience and environmental fit when dealing with freeform natural language (FF NL) input, resulting in robots being unable to respond effectively and perform tasks in a specific environment.

Method used

By combining language model (LM) output with the feasibility assessment of robot skills, a task fit metric and world fit metric are generated, and appropriate robot skills are selected to ensure successful completion of the task.

Benefits of technology

It realizes that the robot can effectively select and execute appropriate skills based on FF NL input, ensure that the task is successfully completed in a specific environment, and improves the practicality and feasibility of natural language control.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118900751A_ABST
    Figure CN118900751A_ABST
Patent Text Reader

Abstract

Embodiments process free-form natural language (NL) instructions using a large language model to generate to generate LLM outputs. Those embodiments generate a task coincidence metric based on the LLM output and the NL skill description of the robotic skill, the task coincidence metric reflecting the probability of the skill description in the probability distribution of the LLM output. Those implementations also generate a world agreement metric based on the robot skill and the current environmental state data, the world agreement metric reflecting a probability that the robot skill is successful based on the current environmental state data. Those implementations also determine whether to implement robotic skills based on both the task coincidence metric and the world coincidence metric.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] Many robots are programmed to perform certain tasks. For example, a robot on an assembly line may be programmed to recognize certain objects and perform specific actions on those objects.

[0002] Further, some robots may perform certain tasks in response to explicit user interface input corresponding to a certain task. For example, a vacuuming robot may perform a general vacuuming task in response to the spoken utterance "robot, clean". However, typically, a user interface input that causes a robot to perform a certain task must be explicitly mapped to the task. As a result, a robot may not be able to perform certain tasks in response to various free-form natural language inputs from a user attempting to control the robot. Summary of the invention

[0003] Language models (LMs) have been developed that can be used to process natural language (NL) content and / or other inputs to generate LM outputs that reflect the NL content and / or other content in response to the input. For example, large language models (LLMs) have been developed that can be trained on massive amounts of data and can be used to robustly process a wide range of NL inputs and generate corresponding LM outputs that reflect accurate and corresponding NL content in response to the NL input. The LLM may include at least hundreds of millions of parameters, and may typically include at least billions of parameters, such as 100 billion or more parameters. For example, the LLM may be a sequence-to-sequence model, Transformer-based, and / or include an encoder and / or decoder. A non-limiting example of an LLM is Google's Pathways Language Model (PaLM). Another non-limiting example of an LLM is Google's Language Model for Conversational Applications (LaMDA).

[0004] In addition, efforts have been made to attempt to achieve robust free-form (FF) NL control of robots. For example, enabling the robot to respond appropriately to any of a variety of different typed or spoken commands from a person to the robot. For example, in response to the FF NL command "put the blocks in the toy bin", a robot task including (a) navigating to "blocks", (b) picking up "blocks", (c) navigating to "toy bin", and (d) putting "blocks" into "toy bin" can be performed.

[0005] Embodiments disclosed herein recognize that LMs (such as LLMs) can encode rich semantic knowledge about the world, and such knowledge can be useful for robots to act according to high-level time-extended instructions expressed by FF NL instructions. Embodiments disclosed herein further recognize that the disadvantage of LLMs is that they lack real-world experience, such as real-world experience in robot control, and / or they lack any grounding for current real-world conditions. For example, prompting an LLM with "I spilled my drink on the table, can you help (I spilled my drink on the table, can you help)" can generate an LLM output that reflects the NL content of reasonable steps to clean up the spill. For example, the highest probability decoding of the LLM output can reflect the NL content "you could try using avacuum (you can try using a vacuum cleaner)". However, "you could try using a vacuum" may not be suitable for a specific agent, such as a robot, that needs to perform this task in a specific environment. For example, a robot may not have an integrated vacuum cleaner, and may not have authority over any separate vacuum cleaner in a specific environment, or may not be able to control a separate vacuum cleaner in a specific environment.

[0006] Embodiments disclosed herein utilize LM outputs to determine how to control a robot to perform a task in response to FF NL inputs that: describe the task at a high level; and fail to describe all (or even any) robot skills required to perform the task. However, in light of the recognized shortcomings of LM, embodiments described herein utilize one or more techniques to ground LM outputs so that the robot skills selected for execution in response to FF NL inputs based on LM outputs are robot skills that are both feasible (e.g., executable by the robot) and contextually appropriate (e.g., likely to succeed if executed by the robot in a particular environment). More specifically, embodiments ground LM outputs by considering not only LM outputs but also robot skills that can be executed by the robot (such as pre-trained robot skills).

[0007] As a non-limiting working example of some of those embodiments, assume that a user provides the FFNL instruction "I spilled my drink on the table, can you help". An LLM prompt can be generated based on the FF NL instruction. For example, the LLM prompt can strictly follow the FF NL instruction. As another example, and as described herein, the LLM prompt can be based on the FF NL instruction, but does not strictly follow the instruction. For example, the prompt can include some or all of the terms of the FF NL instruction, but can additionally include: a scene descriptor of the current environment (e.g., NL descriptors of objects detected in the environment); an explanation generated in a previous pass using the LLM (e.g., based on a previous LLM prompt "explain how you would help when a user says 'I spilled my drink on the table, can you help'); and / or terms used to encourage the LLM to predict steps (e.g., including "I would 1. " at the end of the prompt).

[0008] The generated LLM prompt can be processed using LLM to generate an LLM output that models a probability distribution over candidate word formations that is dependent on the instruction. Continuing with the working example, the highest probability decoding of the LLM output can be, for example, "use a vacuum". However, the embodiments disclosed herein do not simply blindly utilize the probability distribution of the LLM output to determine how to control the robot. Instead, the embodiments utilize the probability distribution of the LLM output while also considering the robot skills that can actually be performed by the robot, such as tens, hundreds, or thousands of pre-trained robot skills. Some versions of those embodiments generate a corresponding task-grounding measure and a corresponding world-grounding measure for each of the robot skills considered. Further, some of those versions select specific robot skills to implement in response to the FF NL prompt based on considering both the corresponding task-grounding measure and the corresponding world-grounding measure. For example, a corresponding overall metric may be generated for each of the robotic skills based on its corresponding world fit metric and task fit metric, and a specific robotic skill may be selected based on having the best corresponding overall metric.

[0009] When generating a task fit metric for a robot skill, the NL skill description of the robot skill can be compared with the LLM output to generate the task fit metric. For example, the task fit metric can reflect the probability of the NL skill description in the probability distribution modeled by the LLM output. For example, and continuing with the working example, the task fit metric of the first robot skill with the NL skill description "pick up a sponge" can reflect a higher probability than the task fit metric of the second robot skill with the NL skill description "pick up a banana". In other words, the probability distribution of the LLM output can characterize that the probability of "pick up a sponge" is higher than that of "pick up a banana". In addition, for example, and continuing with the working example, the task fit metric of the third robot skill with the NL skill description "pick up a squeegee" can reflect a probability similar to the first robot skill.

[0010] When generating a world-fitted metric for a robot skill, current state data, such as environment state data and / or robot state data, may optionally be considered. The environment state data reflects the state of one or more objects in the environment other than the robot, and may include, for example, visual sensor data and / or determinations made by processing visual sensor data (e.g., object detection and / or classification). The robot state data reflects the state of one or more components of the robot, and may include, for example, the current position of a component of the robot, the current velocity of a component of the robot, and / or other state data.

[0011] In some embodiments or for some robot skills, a description of the robot skill may also be considered. In some of those embodiments, a description of the robot skill (e.g., its word embedding) is processed using a trained value function model and current state data is processed to generate a value reflecting the probability of success of the robot skill based on the current state data, and a world fit metric is generated based on the value (e.g., conforms to the value). In some versions of those embodiments, a candidate robot action is also processed using a trained value function model and together with the description and current state data. In some of those versions, multiple values ​​are generated for a given robot skill, where each value is generated based on processing a different candidate robot action but using the same description and the same current state data. In those versions, a world fit metric may be generated based on, for example, the generated value reflecting the highest probability of success.

[0012] Continuing with the worked example, assume that the current environment state data includes an image captured by a camera of the robot, and that the image captures a nearby sponge and a nearby banana, but does not capture any squeegees. In this case, the world-fitting metric for a first robot skill with an NL skill description of "pick up a sponge" may reflect a high probability, the world-fitting metric for a second robot skill with an NL skill description of "pick up a banana" may also reflect a high probability, and the world-fitting metric for a third robot skill with an NL skill description of "pick up a squeegee" may reflect a low probability.

[0013] Continuing with this working example, a first robot skill with an NL skill description of "pick up a sponge" may be selected for implementation based on considering that both its task fit metric and its world fit metric reflect a high probability. Note that the second robot skill, despite having a high world fit metric, is not selected because it has a low task fit metric. Similarly, the third robot skill, despite having a high task fit metric, is not selected because it has a low world fit metric.

[0014] In these and other ways, embodiments disclosed herein can consider both the task fit metric and the world fit metric when selecting a robot skill to implement. This ensures that the selected robot skill is not only (a) likely to result in successful completion of the task reflected by the FF NL input (as reflected by the task fit metric), but also (b) likely to succeed if implemented by the robot in the current environment (as reflected by the world fit metric).

[0015] The above description is provided as an overview of only some of the embodiments disclosed herein. These and other embodiments are described in more detail herein, including in the detailed description and claims.

[0016] It should be understood that all combinations of the aforementioned concepts and additional concepts described in more detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1A An example of a person providing free-form (FF) natural language (NL) instructions to an example robot is shown.

[0018] Figure 1B Shows Figure 1AA simplified bird's-eye view of an example environment in which a person and robot are located, and shows example current visual data instances that can be captured in the environment.

[0019] Figure 2A shows how various example components may interact to select a response to Figure 1A The FF NL instruction and Figure 1B The process flow of the initial robotic skills to be implemented in the environment.

[0020] Figure 2B Shows how various example components may interact to select an implementation Figure 2A After (or during) the robot skill selected in and in response to Figure 1A The FF NL instruction and Figure 1B The process flow of the next robotic skill to be implemented in the environment.

[0021] Figure 3 is a flow chart illustrating an example method of implementing a robotic skill using an LLM to perform tasks reflected in FF NL instructions according to an embodiment disclosed herein.

[0022] Figure 4 is a flow chart illustrating an example method for generating a world-fitting metric for a robot skill based on processing current state data and a skill description of the robot skill.

[0023] Figure 5 An example architecture of a robot is schematically depicted.

[0024] Figure 6 An example architecture of a computer system is schematically depicted. DETAILED DESCRIPTION

[0025] Before turning to a detailed description of the drawings, a non-limiting overview of the various embodiments is provided.

[0026] In various embodiments disclosed herein, the robot is equipped with a learned robotic skill library for atomic behaviors capable of low-level visual motor control. Instead of simply prompting the LLM to simply interpret the FF NL high-level instructions, some of those embodiments utilize the LLM output generated by the prompt to generate a task fit metric, which each quantifies the likelihood that the corresponding robotic skill will make progress toward completing the high-level instruction. Further, the corresponding affordance function (e.g., the learned value function) of each robotic skill in the robotic skill can be used to generate a world fit metric for the robotic skill, which quantifies the likelihood that the robotic skill will succeed from the current state. Further, both the task fit metric and the world fit metric can be used to determine which robotic skill to implement next to complete the task reflected by the high-level instruction. In these and other ways, the embodiments utilize the LLM output to describe the probability that each skill contributes to completing the instruction, and the affordance function describes the probability that each skill will succeed, and the combination of the two provides the probability that each skill will successfully execute the instruction. In addition to task-fitting the LLM output, the affordance function enables consideration of real-world fit, and constraining completions to skill descriptions enables consideration of the LLM output in a way that understands the robot's capabilities (e.g., given its learned repertoire of robotic skills). Furthermore, this combination produces a fully interpretable sequence of steps that the robot will execute to complete the high-level instructions (e.g., a description of the robotic skill chosen for implementation) - an interpretable plan expressed in language.

[0027] Thus, embodiments utilize LLMs to provide task fit to determine useful actions for a high-level goal, and utilize affordance functions (e.g., learned functions) to provide world fit to determine which actions can actually be performed in achieving the high-level goal. Some of those embodiments utilize reinforcement learning (RL) as a way to learn a language-conditioned value function that provides affordances of what is possible in the real world.

[0028] Language models attempt to String The probability of a sequence of This is usually done by decomposing the probability into The algorithm is performed so that each successive string is predicted from the previous one. Recent breakthroughs initiated by attention architectures based on neural networks have enabled efficient scaling of so-called LLMs. Such LLMs include Transformer, LAMDA, PaLM, and others, each of which has shown increasingly large capacity (billions of parameters and terabytes of text) and the ability to subsequently generalize across tasks.

[0029] Embodiments disclosed herein utilize the extensive semantic knowledge contained in the LLM to determine useful tasks for solving high-level instructions. Some of those embodiments further attempt to accurately predict whether a skill (given by the NL descriptor) is feasible in the current state of the robot or other agent's environment. In some versions of those embodiments, or for at least some skills, reinforcement learning based on temporal differences (TD) is used to achieve this goal. In some of those versions, a Markov decision process is defined as (MDP) ,in and is the state and action space, is the state transition probability function, is the reward function, and is the discount factor. The goal of the TD method is to learn the state or state-action value function (Q function) , which means that when starting from state s and action a, followed by the policy The resulting action is the discounted sum of rewards, i.e. Q function It can be learned via an approximate dynamic programming method that optimizes the following loss: , where D is a dataset of states and actions, and are the parameters of the Q function.

[0030] For example, TD-based methods can be used to learn a value function, such as one conditioned on a natural language descriptor of a skill, and use that value function to determine whether a given skill is feasible in a given state. Notably, in the case of undiscounted sparse rewards, i.e., the agent receives a reward of 1.0 at the end of an episode if it succeeds and 0.0 otherwise, the value function trained via RL corresponds to an affordance function that specifies whether a skill is possible in a given state.

[0031] Various embodiments disclosed herein receive a natural language instruction i provided by a user, which describes a task that the robot should perform. The instruction can be long, abstract, and / or vague. Further, the embodiment has access to a set of robot skills II, where each skill Perform short tasks, such as picking up a specific object, with a brief verbal description (e.g., “find a sponge”) and affordance functions , this affordance function indicates successful completion from state s with description Intuitively, Meaning "ifI ask the robot to do , will it do it? (If I ask the robot to do , will it do it? )". In RL terms, if the reward is 1 for successful completion and 0 otherwise, then is the value function of the skill.

[0032] As mentioned above, Indicates skills The text label, and Indicates that if executed from state s, it has a text label Skills The probability of successful completion, where is a Bernoulli random variable. Using LLM to process the prompt based on the natural language instruction i generates an LLM output, which represents , i.e., the probability that the skill's text label is a valid next step for the user's instruction. However, embodiments disclosed herein recognize that it is important to consider the probability that a given skill successfully progresses toward actually completing the instruction, expressed as Assume that the probability of successful skill progress on i is (i.e., the probability that the skill is the correct skill) and the probability of progressing in a failed skill is zero, which can be decomposed into , which is sometimes referred to in this article as task fit or a task fit measure. Further, the probability that the skill is possible in the current state of the world can be decomposed into , which is sometimes referred to herein as world fitting or a world fitting metric.

[0033] Although LLM can take advantage of rich knowledge learned from large amounts of text, LLM will not necessarily decompose high-level commands into low-level instructions suitable for robot execution. For example, if you ask LLM "how would a robot bring me an apple", it may respond with "a robot could go to a nearby storeand purchase an apple for you". Although this response is a reasonable completion of the prompt, it is not necessarily actionable for a concrete agent (such as a robot) that may have a limited and fixed set of capabilities. Therefore, in order to adapt to LLM or other LMs, some embodiments attempt to implicitly tell LMs that high-level instructions should be decomposed into sequences of available low-level skills. One way to achieve this is careful prompt engineering, that is, a technique that induces LMs to specific response structures. Prompt design provides examples for LMs in context text ("prompts") that specify the tasks and response structures that LMs will simulate.

[0034] Scoring language models open up a path for constrained responses by outputting the probabilities assigned by LM to fixed outputs. LM represents the distribution of potential completions ,in is the word that appears at the kth position in the text. While typical generative applications (e.g., conversational agents) sample from this distribution or decode the maximum likelihood completion, embodiments disclosed herein may instead use this distribution to score candidate completions selected from a set of options (e.g., a candidate skill description selected from a set of candidate skill descriptions). More formally, given a set of low-level robot skills II, their language descriptions and instruction i, which can calculate the language description of the skill The probability of making progress towards executing instruction i: , which corresponds to querying the model within a potential completion. Compute the best skill based on the language model.

[0035] In some embodiments, the process is performed by iteratively selecting skills and attaching them to instructions. In practice, this can be viewed as a conversation between a user and a robot, where the user provides high-level instructions (e.g., "How would you bring me a coke can?"), and the language model responds with an explicit sequence ("I would: 1. (I will: 1.)"), for example, "I would: 1. find a coke can, 2. pick up the coke can, 3. bring it to you (I will: 1. find a coke can, 2. pick up the coke can, 3. bring it to you)". This has the benefit of increased interpretability because the generative response is generated along with the concept of possibility across multiple possible responses.

[0036] While this approach enables efficient extraction of knowledge from the language model, it leaves one major problem. That is, while the decoding of the instructions obtained in this way always contains the skills available to the robot, these skills may not necessarily be suitable for performing the desired high-level task in the specific situation the robot is currently in. For example, if the FF NL prompt is "bring me an apple", the optimal set of skills will change if there is no apple in the robot's field of view or if there is already an apple in the robot's gripper.

[0037] Therefore, embodiments attempt to fit large language models through a value function, namely an affordance function that captures the log-likelihood that a particular skill will be able to succeed in the current state. , its language description and provide (That is, in state s, the The corresponding value function of the skill described by This value function space captures the affordances across all skills. For each skill, the affordance function can be multiplied by the LLM probability, and the most likely skill is ultimately selected—that is, .

[0038] Once the most likely skill is selected, the agent executes the corresponding policy and revises the LLM query to include (the language description of the selected skill), and the process is run again until a terminal token is selected (e.g., "done"). This process is described in Algorithm 1 provided below. These two mirroring processes together lead to a probabilistic interpretation, where the LLM provides the probability that the skill is useful for the high-level instruction, and the affordance provides the probability of successfully executing each skill. Combining these two probabilities together provides the probability that the skill will further perform the high-level instruction commanded by the user. Algorithm 1 Given: high-level instruction i, state s0 and a set of skills II and their language descriptions

[0039] Embodiments disclosed herein utilize a set of skills, each of which has a policy (for use by the agent when performing the skill), a value function, and a short language description (e.g., "pick up the can"). These skills, value functions, and descriptions can be obtained in a variety of different ways. As an example, individual skills can be trained using image-based behavior cloning (e.g., following the BC-Z method) or reinforcement learning (e.g., following MT-Opt). Regardless of how the policy for the skill is obtained, a value function, such as one trained via TD backup, can be used for the skill. In order to amortize the cost of training multiple skills, multi-task BC and / or multi-task RL can be used for one or more skills. By using multi-task BC and / or multi-task RL, instead of training a separate policy and value function per skill, multi-task policies and models are trained that are conditioned on NL skill descriptions. However, it should be noted that this description only corresponds to low-level skills - LLM is still used to interpret high-level instructions and decompose them into separate low-level skill descriptions.

[0040] In some embodiments, in order to make the policy conditional on language, a pre-trained large sentence encoder language model can be utilized. The sentence encoder language model parameters can be frozen during training, and the embeddings generated by passing in the text description of each skill can be used as the embeddings used when conditioning the policy. The text embeddings are used as input to the policy and value functions that specify which skill should be executed. Since the language model used to generate the text embeddings is not necessarily the same as the language model used for planning, embodiments are able to utilize different language models suitable for different levels of abstraction - understanding planning about multiple skills, rather than expressing specific skills at a more fine-grained level.

[0041] In some embodiments, BC and / or RL policy training procedures may be used to obtain one or more of the language conditioning policies and one or more of the value functions, respectively. As mentioned above, for skill specifications, a set of short natural language descriptions represented as language model embeddings may be utilized. Further, a sparse reward function may be utilized, such as a sparse reward function that has a reward value of 1.0 at the end of a round if the language command has been successfully executed and a reward value of 0.0 otherwise. For example, the success of the execution of the language command may be evaluated by a human, where the evaluator is provided with a video of the robot performing the skill and the given instructions. If two out of three (or other threshold) evaluators agree that the skill has been successfully completed, the round will be marked as a positive reward. As an example, and for a round with Figure 1A For a robot similar to the one shown in , the action space of the policy can include the six degrees of freedom of the end effector pose as well as the opening and closing commands of the gripper, the xy position and yaw orientation increments of the robot's mobile base, and the terminal actions.

[0042] Various robotic skills can be utilized in the embodiments disclosed herein, such as manipulation and navigation skills using a mobile manipulator robot. Such skills can include picking up, placing, and rearranging objects, opening and closing drawers, navigating to various locations, and placing objects in a specific configuration.

[0043] The RL models utilized in some embodiments (such as RL policy models and / or RL value function models) can use an architecture similar to MT-Opt, optionally with slight changes to support natural language input. As a specific example, a camera image (e.g., from a robot's camera) can be processed through the convolutional layers of the image tower of the architecture to generate an image embedding. The skill description can be embedded through the LLM (or other embedding model) and then concatenated with the non-image parts of the robot's actions and states (such as gripper height). To support asynchronous control, reasoning can optionally occur while the robot is still moving from the previous action. The model is optionally provided with how much of the previous action is left to execute. The conditioned input undergoes the fully connected layers of the additional tower of the architecture and is then spatially tiled to generate an additional embedding that is added / concatenated to the image embedding before undergoing the additional convolutional layers. The output is gated by sigmoid so that the Q value is always in [0, 1].

[0044] The BC models utilized in some embodiments (such as the BC policy model) may use an architecture similar to BC-Z. As a specific example, the skill description may be embedded by a universal sentence encoder and then used to condition a Resnet-18 based architecture with FiLM. Unlike the RL model, the previous action or gripper height may not be provided. Multiple FC layers may be applied to the final visual features to output each action component (arm position, arm orientation, gripper, and terminal action).

[0045] It should be noted that the flexibility of the embodiments disclosed herein allows for mixing and matching strategies and affordances from different approaches. For example: for a pick manipulation skill, a single multi-task language conditional strategy can be used; for a place manipulation skill, a script strategy with affordances based on the gripper state can be used; for a navigation strategy, a planning-based approach can be used that knows where a specific object can be found and the corresponding distance metric. In some embodiments, to avoid the situation where a skill is selected but it has already been executed or will have no effect, an upper limit on the affordance can be set, which indicates that the skill has been completed and a reward has been received.

[0046] As described above, the embodiments disclosed herein enable many different strategies and / or affordance functions to be utilized through their probabilistic interface. Therefore, as skills become better performing or as new skills are learned, such skills can be directly incorporated into the embodiments disclosed herein.

[0047] As described above, embodiments utilize affordance functions , which indicates successful completion of the function from status s with description The probability of the skill. Some of the learned strategies that can be used in this paper generate Q functions Given an action a and a state s , obtained by optimizing the value through the cross entropy method , which is similar to MT-Opt. For brevity, some example value functions are described below with their skill text. and the affordance function is referenced as In some embodiments, due to training artifacts, the value function needs to apply the calibration directly as a probability. The parameters used for calibration can be determined, for example, empirically. In addition, and as mentioned above, the following logic can be enforced: if the skill has been completed and a reward received (e.g., navigating to a table that the robot is already in front of), then the skill should not be performed.

[0048] The following is an example affordance function for the "pick" skill. The trained value function for the pick skill typically has a minimum when the skill is impossible and a maximum when the skill is successful, and so the value function is normalized to obtain an affordance function where .

[0049] The following is an example affordance function for a "go to" / "navigate" skill. The affordance function for the go to skill is based on the distance d (in meters) to the location. .

[0050] An example affordance function that terminates a skill could be to always set it to be A small value for , such as 0.1, is used to ensure that the planning process terminates when no viable skills are available.

[0051] Now turning to the attached figure, Figure 1A An example is shown of a person 101 providing a free-form (FF) natural language (NL) instruction 105 to an example robot 110, “bring me a snack from the table.”

[0052] Figure 1A The robot 110 shown is a specific mobile robot. However, additional and / or alternative robots may be utilized with the techniques disclosed herein, such as robots that are similar in one or more aspects to Figure 1A The robot 110 shown is different from additional robots. For example, in the technology described herein, instead of or in addition to the robot 110, a mobile forklift robot, an unmanned aerial vehicle ("UAV"), a non-mobile robot and / or a humanoid robot can also be utilized.

[0053] The robot 110 includes a base 113 having wheels on opposite sides thereof for movement of the robot 110. The base 113 may include, for example, one or more motors for driving the wheels of the robot 110 to achieve a desired direction, speed, and / or acceleration of movement of the robot 110. The robot 110 also includes a robot arm 114 having an end effector 115 in the form of a gripper having two opposing "fingers" or "digits".

[0054] The robot 110 also includes a vision component 111 that can generate visual data (e.g., images) related to the shape, color, depth, and / or other features of objects within the line of sight of the vision component 111. The vision component 111 can be, for example, a monocular camera, a stereo camera (active or passive), and / or a 3D laser scanner.

[0055] A 3D laser scanner may include one or more lasers that emit light and one or more sensors that collect data related to the reflection of the emitted light. The 3D laser scanner may generate visual component data, which is a 3D point cloud, wherein each point in the 3D point cloud defines the position of a point of a surface in 3D space. A monocular camera may include a single sensor (e.g., a charge coupled device (CCD)), and generates images based on physical properties sensed by the sensor, each of which includes a plurality of data points defining a color value and / or a grayscale value. For example, a monocular camera may generate an image including red, blue, and / or green channels. Each channel may define a value for each of a plurality of pixels of an image, such as defining a value from 0 to 255 for each of a plurality of pixels of an image. A stereo camera may include two or more sensors each located at a different vantage point. In some of those embodiments, a stereo camera generates an image based on properties sensed by two sensors, each of which includes a plurality of data points defining a depth value and a color value and / or a grayscale value. For example, a stereo camera may generate an image including a depth channel and a red, blue, and / or green channel.

[0056] The robot 110 also includes one or more processors that, for example: use LLM to process LLM prompts based on the FF NL input 105 to generate LLM outputs; determine a description of a robot skill and a value function of the robot skill based on the LLM output, the robot skill to be implemented when performing a robot task; control the robot 110 based on the determined robot skill during the performance of the robot task; etc. For example, one or more processors of the robot 110 can implement all or multiple aspects of the methods 300 and / or 400 described herein. Additional descriptions of some examples of the structure and function of various robots are provided herein.

[0057] Now go to Figure 1B , showing Figure 1A A simplified bird's-eye view of an example environment in which a person 101 and a robot 110 are located. Figure 1B , the person 101 and the robot 110 are represented as circles. Further, Figure 1B 1, 192, 193, and 194 are shown in FIG. Environmental features 191, 192, 193, and 194 show the outlines of various landmarks in the environment. For example, the environment may be an office kitchen or a work kitchen, and features 191 and 192 may be countertops, feature 193 may be a sink, and feature 194 may be a round table.

[0058] Figure 1BAlso shown is an example current visual data instance 180 that can be captured in an environment and used, for example, to generate a world-fitting metric for a robot skill. For example, robot 110 can capture current visual data instance 180 using vision component 111. Visual data instance 180 captures a pear 184A and a key 184B, both of which are present on a round table represented by feature 194. Note that in the bird's-eye view, both the pear and the key are shown as points for simplicity.

[0059] Turning now to FIG. 2 , various example components may interact to select a response to Figure 1A The FF NL instruction is 150 and in Figure 1B The process flow of the initial robotic skills to be implemented in the environment. Figure 2A The example components shown include an LLM engine 130, an LLM 150, a task conformation engine 132, a world conformation engine 134, a value function model, a selection engine 130, and an implementation engine 136. One or more of the components shown may be implemented by the robot 110 (e.g., using a processor and / or its memory) and / or using a remote computing device (e.g., a cloud-based server) in network communication with the robot 110.

[0060] exist Figure 2A , the LLM engine 130 generates an LLM prompt 205A based on the FF NL input 105 (“bring me a snack from the table”). The LLM engine 130 may generate the LLM prompt 205A such that it strictly complies with the FF NL input 105, or may generate the LLM prompt 205A such that it is based on the FF NL input 105 but does not strictly comply with the FF NL input. For example, as shown by LLM prompt 205A1, a non-limiting example of LLM prompt 205A, the LLM prompt may be “How would you bring me a snack from the table? I would 1.” Such an LLM prompt 205A1 includes “How would you” as a prefix and “I would 1.” as a suffix, either or both of which may encourage prediction of steps in the LLM output related to achieving the high-level task specified by the FF NL input 105.

[0061] In some implementations, the LLM engine 130 may optionally further generate the LLM prompt 205A based on one or more of the context descriptor 202A, the prompt examples 203A, and / or the interpretation 204A of the robot's 110 current environment.

[0062] The scene descriptors 202A may include NL descriptors of objects currently or recently detected by the robot 110 in the environment, such as descriptors of objects determined based on processing images or other visual data using an object detection and classification machine learning model. For example, the scene descriptors 202A may include "pear", "keys", "human", "table", "sink", and "countertops", and the LLM engine 130 may generate an LLM prompt 205A to incorporate one or more of such descriptors. For example, the LLM prompt 205A may be "A pear, keys, a human, a table, a sink, and countertops are nearby. How would you bring the human a snack from the table. I would 1. (There are pears, keys, people, tables, sinks, and countertops nearby. How would you bring me a snack from the table? I would 1.)".

[0063] Prompt examples 203A may optionally include manually designed examples of desired output styles. For example, they may include "in a step-by-step format" or "in a numbered list format" or "in 1. first step, 2. second step, 3. third step, etc." Prompt examples 203A may be prepended to LLM prompts 205A or otherwise incorporated into LLM prompts 205A to encourage prediction of content in the output style in the LLM output. Explanation 204A may be an explanation generated based on processing a previous LLM prompt (also based on FF NL input), in a previous pass, and using LLM 150. For example, a previous LLM prompt may be "explain how you would bring me a snack from the table", and explanation 204A may be generated based on the highest probability decoding from a previous LLM output. For example, explanation 204A may be "I would find the table, then find a snack on the table, then bring it to you". Interpretation 204A may be prepended to LLM prompt 205A, replace the term of FF NL input 105 in LLM prompt 205A, or otherwise incorporated into LLM prompt 205A.

[0064] The LLM engine 130 processes the generated LLM hints 205A using the LLM 150 to generate an LLM output 206A. As described herein, the LLM output 206A may model a probability distribution over candidate word formations and rely on the LLM hints 205A.

[0065] The task fit engine 132 generates a task fit metric 208A, and generates the task fit metric 208A based on the LLM output 206A and the skill description 207. Each skill description in the skill description 207 describes a corresponding skill that the robot 110 is configured to perform. For example, "go to the table" can describe a "navigate to table" skill that the robot can perform using a trained navigation strategy with "table" (or a location corresponding to "table") as a navigation target. As another example, "go to the sink" can describe a "navigate to sink" skill that the robot can perform using a trained navigation strategy with "sink" (or a location corresponding to "sink") as a navigation target. As yet another example, "pick up a bottle" can describe a "grasp a bottle" skill that the robot can perform using a grasping heuristic fine-tuned for bottles and / or using a trained grasping network. As yet another example, “pick up keys” may describe a “grasp keys” skill that a robot may perform using a grasping heuristic fine-tuned for keys and / or using a trained grasping network. Although skill description 207 includes Figure 2A AG in the skill description 207, but the skill description 207 may include descriptors of additional skills in various embodiments (as indicated by ellipses). Such additional skills may correspond to substitute objects and / or may be used for different types of robot actions (e.g., "place", "push", "open", "close").

[0066] Each task fit metric in task fit metric 208A is generated based on the probability of the corresponding skill description in LLM output 206A. For example, task fit metric A "0.85" reflects the probability of the word sequence "go to the table" in LLM output 206A. As another example, task fit metric B "0.20" reflects the probability of the word sequence "go to the sink" in LLM output 206A.

[0067] The world adaptation engine 134 generates a world adaptation metric 211A for the robot skills. In generating the world adaptation metric 211A for at least some of the robot skills, the world adaptation engine 134 may generate the world adaptation metric based on the environment state data 209A, and optionally, further based on the robot state data 210A and / or the corresponding skill description in the skill description 207. Further, in generating the world adaptation metric 211A for at least some of the robot skills, the world adaptation engine 134 may utilize one or more value function models in the value function model 152.

[0068] In some embodiments, the world conformity engine 134 may generate fixed world conformity metrics for some robot skills. For example, any "place" robot skill may always have a fixed metric, such as 1.0 or 0.9, or a "terminate" robot skill (i.e., indicating that a task is completed) may always have a fixed metric, such as 0.1 or 0.2. In some embodiments, the world conformity engine 134 may additionally or alternatively generate world conformity metrics for some robot skills based on corresponding value function models in the value function model 152 that are not based on machine learning (e.g., not a neural network and / or not trained). For example, the value function model may define that when the environment state data 209A and / or the robot state data 210A indicates that the robot 110 is grasping an object, any "place" robot skill should have a fixed metric, such as 1.0 or 0.9, otherwise it should have another fixed metric, such as 0.0 or 0.1. As another example, a value function for a "navigate to [object / location]" robot skill may define a world conformity metric that is a function of the distance between the robot and the object / location, as determined based on the environment state data 209A and the robot state data 210A. Or a "terminate" robot skill (i.e., indicating that the task is complete) may always have a fixed metric, such as 0.1 or 0.2.

[0069] In some embodiments, the world fitting engine 134 may additionally or alternatively generate a world fitting metric for some robot skills based on a corresponding value function model in the value function model 152 as a trained value function model. In some embodiments of those embodiments, the trained value function model may be a language-conditioned model. For example, the world fitting engine 134 may use the language-conditioned model to process a corresponding skill description in the skill description 207 of the robot skill together with the environment state data 209A and optionally the robot state data 210A when generating a world fitting metric for the robot skill to generate a value reflecting the probability of success of the robot skill based on the current state data. The world fitting engine may generate a world fitting metric based on the generated value (e.g., conforming to the value). In some versions of those embodiments, the language-conditioned model is also used and the candidate robot action is processed together with the corresponding skill description in the skill description 207, the environment state data 209A, and optionally the robot state data 210A. In some of those versions, multiple values ​​are generated, where each value is generated based on processing a different candidate robot action but using the same corresponding skill description in the skill description 207, the same environment state data 209A, and optionally the same robot state data 210A. In those versions, the world fitting engine 134 can generate a world fitting metric based on, for example, the generated values ​​reflecting the highest probability of success. In some of those versions, different candidate robot actions can be selected using, for example, a cross-entropy method. For example, N robot actions can be initially randomly sampled and a value generated for each action, and then N additional robot actions can be sampled around one of the initial N robot actions based on the initial robot action with the highest generated value. The trained value function model used by the world fitting engine 134 when generating a world fitting metric for a robot skill can also be used for the actual implementation of the robot skill by the robot 110.

[0070] The world fit metric 211A is based on Figure 1B184A and the key 184B. Therefore, the world fit metrics F and G for "pick up apear" and "pick up keys", respectively, are both relatively low ("0.10"). This reflects the low probability of a successful attempted grasp of either item, which is attributed to the large distance between the robot 110 and the pear 184A and the key 184B. The world fit metric A "0.80" for "go to the table" reflects a lower probability than the world fit metric B "0.85" for "go to the sink". This can be based on the fact that the robot 111 is closer to the sink 193 than to the table 194.

[0071] The selection engine 136 considers both the world fit metric 211A and the task fit metric 208A when selecting robot skill A ("go to the table"), and sends an indication 213A of the selected robot skill A to the implementation engine 136. In response, the implementation engine 136 controls the robot 110 based on the selected robot skill A. For example, the implementation engine 136 may control the robot using a navigation strategy with "table" (or a location corresponding to "table") as a navigation target.

[0072] exist Figure 2A , the selection engine 136 generates an overall metric 212A by multiplying the world fit metric 211A by the task fit metric 208A, and selects robot skill A based on robot skill A having the highest overall metric in overall metrics 212A. It should be noted that although robot skill A does not have the highest world fit metric, it has the highest overall metric. Figure 2A The overall metric 212A is shown to be generated by multiplying the world alignment metric 211A and the task alignment metric 208A, but other techniques may be used to generate the overall metric 212A. For example, different weights may be applied to the world alignment metric 211A and the task alignment metric 208A when multiplying. For example, the weight of the world alignment metric 211A may be 90%, and the weight of the task alignment metric 208A may be 100%.

[0073] Now turn to Figure 2B , Figure 2A How the various example components may interact to select an implementation Figure 2A After (or during) the robot skill A selected in and in response to Figure 1A The FF NL instruction and Figure 1B The process flow of the next robotic skill to be implemented in the environment.

[0074] exist Figure 2B , the LLM engine 130 generates an LLM prompt 205B for robot skill A based on the FF NL input 105 (“bring me a snack from the table”) and further based on the selected skill descriptor 201B (“go to the table”). The LLM engine 130 can generate the LLM prompt 205B so that it strictly adheres to the FF NL input 105 and the selected skill descriptor 201B, or can generate the LLM prompt 205B so that it is based on but does not strictly adhere to the FF NL input 105 and the selected skill descriptor 201B. For example, as shown by LLM prompt 205B1, a non-limiting example of an LLM prompt 205B, the LLM prompt can be “How would you bring me a snack from the table? I would 1. Go to the table. 2.” Such an LLM prompt 205B1 includes "How would you" as a prefix and "2." as a suffix, either or both of which may encourage prediction of steps in the LLM output related to achieving the high-level task specified by the FF NL input 105. Further, such an LLM prompt 205B1 includes the selected skill descriptor 201B "go to the table" preceded by "1.", which may encourage prediction of steps in the LLM output related to occurring after the "go to the table" step.

[0075] In some embodiments, the LLM engine 130 may optionally further generate a scene descriptor 202B (which may be relative to the current environment of the robot 110) based on the scene descriptor 202B of the current environment of the robot 110. Figure 2A ), prompt example 203B (which can optionally be combined with Figure 2A ) and / or explanation 204B (which may optionally be different from the prompt example in Figure 2A ) to generate LLM prompt 205B.

[0076] The LLM engine 130 processes the generated LLM hints 205B using the LLM 150 to generate an LLM output 206B. As described herein, the LLM output 206B may model a probability distribution over candidate word formations and rely on the LLM hints 205B.

[0077] The task alignment engine 132 generates task alignment metrics 208B, and generates task alignment metrics 208B based on the LLM output 206A and the skill description 207. Each task alignment metric in the task alignment metrics 208B is generated based on the probability of the corresponding skill description in the LLM output 206B. For example, the task alignment metric A "0.00" reflects the probability of the word sequence "go to the table" in the LLM output 206B. As another example, the task alignment metric B "0.10" reflects the probability of the word sequence "go to the sink" in the LLM output 206B.

[0078] The world adaptation engine 134 generates a world adaptation metric 211B for the robot skills. When generating the world adaptation metric 211B for at least some of the robot skills, the world adaptation engine 134 may generate the world adaptation metric based on the environment state data 209B, and optionally, further based on the robot state data 210B and / or the corresponding skill description in the skill description 207. Figure 2B The environment state data 209B and the robot state data 210B are at least partially implemented at the time of Figure 2A Further, the world adaptation engine 134 may utilize one or more value function models in the value function models 152 when generating world adaptation metrics 211B for at least some of the robot skills.

[0079] In some embodiments, the world fitting engine 134 may generate a fixed world fitting metric for some robot skills. In some embodiments, the world fitting engine 134 may additionally or alternatively generate a world fitting metric for some robot skills based on a corresponding value function model in the value function model 152 that is not based on machine learning (e.g., not a neural network and / or is not trained). In some embodiments, the world fitting engine 134 may additionally or alternatively generate a world fitting metric for some robot skills based on a corresponding value function model in the value function model 152 that is a trained value function model. In some of those embodiments, the trained value function model may be a language-conditioned model.

[0080] The world fit metric 211B is generated based on the state of the robot 110 after the robot skill A has been at least partially implemented (i.e., the robot 110 is more Figure 1B The bird's eye view is closer to the table 194). Therefore, Figure 2A184A and key 184B (e.g., within reach of both). Therefore, the world fit metrics F and G for “pick up a pear” and “pick up keys,” respectively, are both relatively high (“0.90”).

[0081] The selection engine 136 considers both the world-fitting metric 211B and the task-fitting metric 208B in selecting a robot skill F ("pick up the pear"), and sends an indication 213B of the selected robot skill F to the implementation engine 136. In response, the implementation engine 136 controls the robot 110 based on the selected robot skill F. For example, the implementation engine 136 may control the robot using a grasping strategy, optionally with parameters tuned for grasping a pear.

[0082] exist Figure 2B , the selection engine 136 generates an overall metric 212B by multiplying the world fit metric 211B by the task fit metric 208B, and selects robot skill F based on the robot skill F having the highest overall metric in the overall metrics 212B. It should be noted that although robot skill A does not have the highest world fit metric and does not have the highest task fit metric, it has the highest overall metric. Figure 2B The overall metric 212B is shown to be generated by multiplying the world alignment metric 211B and the task alignment metric 208B, but other techniques may be used to generate the overall metric 212B. For example, different weights may be applied to the world alignment metric 211B and the task alignment metric 208B when multiplying. For example, the weight of the world alignment metric 211B may be 90%, and the weight of the task alignment metric 208B may be 100%.

[0083] Figure 3 is a flow chart illustrating an example method 300 for implementing a robot skill using an LLM to perform a task reflected in an FF NL instruction according to an embodiment disclosed herein. For convenience, the operations of method 300 are described with reference to a system that performs the operations. The system may include one or more components of a robot, such as a robot processor and / or a robot control system of robot 110, robot 520, and / or other robots, and / or may include one or more components of a computer system (such as computer system 610). In addition, although the operations of method 300 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0084] At block 352, the system recognizes a FF NL instruction.

[0085] At block 354, the system processes the LLM hint based on the FF NL instruction using the LLM to generate an LLM output. Block 354 optionally includes sub-block 354A and / or sub-block 354B. At sub-block 354A, the system includes the scene descriptor in the LLM hint. At sub-block 354B, the system generates an explanation and includes the explanation in the LLM hint.

[0086] At block 356, the system generates a corresponding task fit metric based on the LLM output of block 354 and the corresponding NL skill description for each of the plurality of candidate robot skills. Block 356 optionally includes subblock 356A, in which the system generates the task fit metric for the candidate robot skill based on the probability of the NL skill description as reflected in the probability distribution of the LLM output.

[0087] At block 358, the system generates a corresponding world-fitting metric for each candidate robot skill in the plurality of candidate robot skills based on the current state data. Figure 3 358 is shown as being positionally below block 356, but it should be noted that in various implementations, block 358 may occur in parallel with block 356, or even before block 356.

[0088] Block 358 optionally includes sub-block 358A and / or sub-block 358B. Optionally, sub-block 358A and / or sub-block 358B are performed only for certain candidate robot skills.

[0089] At subblock 358A, the system uses the value function model to process current environment state data and / or current robot state data when generating a world-fitting metric for a candidate robot skill.

[0090] At sub-block 358B, the system also processes the NL description of the robot skill (e.g., its embedding) when generating a world-fitting metric for the candidate robot skill using the multi-skill and language-conditioned value function model.

[0091] At block 360, the system selects a robot skill to implement or selects a termination condition (which may be considered a specific robot skill) based on the task fit metric of block 356 and the world fit metric of block 358. For example, a termination condition may be selected when a task fit metric of "done," "complete," or other termination description corresponding to the termination condition meets a threshold, and / or when the task fit metric, world fit metric, and / or overall metric of all robot skills fail to meet the same or different thresholds.

[0092] At block 362, the system determines whether a robot skill is selected for implementation at block 360. If not, the system proceeds to block 364 and completes controlling the robot based on the FF NL instructions of block 352. If yes, the system proceeds to blocks 366 and 368.

[0093] At block 366, the system implements the selected robot skill. At block 368, the system modifies the most recent LLM hint based on the skill description of the implemented skill. The system then returns to block 354 and processes the modified LLM hint at block 368. The system also performs another iteration of blocks 356, 358, 360, and 362, and optionally another iteration of blocks 366 and 368 (depending on the decision of block 362). This general process may continue until a termination condition is selected in an iteration of block 360.

[0094] Figure 4 is a flow chart illustrating an example method for generating a world fit metric for a robot skill based on processing current state data and a skill description of the robot skill. For convenience, the operations of method 400 are described with respect to a system that performs the operations. The system may include one or more components of a robot, such as a robot processor and / or a robot control system of robot 110, robot 520, and / or other robots, and / or may include one or more components of a computer system (such as computer system 610). In addition, although the operations of method 400 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0095] At block 452, the system generates an image embedding by processing a current image using an image tower of the trained language-conditioned value function model. The image may be a current image captured by a vision component of the robot.

[0096] At block 454, the system selects a candidate action. For example, the system can select a candidate action by sampling from the action space.

[0097] At block 456 , the system generates additional embeddings by processing the candidate action selected at block 454 , the skill description of the robot skill (e.g., its embedding), and optionally the robot state data using an additional tower of the trained language-conditioned value function model.

[0098] At block 458, the system generates a value based on processing the concatenation of the image embedding and the additional embedding using an additional layer of the trained language conditioned value function model.

[0099] At box 460, the system determines whether to generate additional values ​​for the additional candidate actions. If not, the system proceeds to box 464. If yes, the system proceeds to box 462 and selects the additional candidate actions. In some iterations of box 462, this may include sampling from the action space without regard to the values ​​generated so far. In some subsequent iterations of box 462, this may include sampling from the action space using a cross-entropy method and based on the values ​​generated so far. For example, the sampling may be centered around or otherwise biased toward a portion of the action space corresponding to the candidate action with the highest generated value so far. The system then returns to box 456 using the newly selected candidate action but the same skill description and optionally the same robot state.

[0100] At box 464 , the system generates a world-fitting metric for the robot skill corresponding to the skill description (used in the iteration of box 456 ) based on the maximum of the values ​​generated in the iteration of box 458 .

[0101] Figure 5 An example architecture of a robot 520 is schematically depicted. The robot 520 includes a robot control system 560, one or more operating components 540a-540n, and one or more sensors 542a-542m. The sensors 542a-542m may include, for example, vision sensors, light sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors, accelerometers, gyroscopes, thermometers, barometers, etc. Although the sensors 542a-542m are depicted as being integrated with the robot 520, this is not meant to be limiting. In some embodiments, the sensors 542a-542m may be located external to the robot 520, for example, as a separate unit.

[0102] The operating components 540a-540n may include, for example, one or more end effectors and / or one or more servo motors or other actuators to achieve movement of one or more components of the robot. For example, the robot 520 may have multiple degrees of freedom, and each of the actuators may control actuation of the robot 520 within one or more of the degrees of freedom in response to a control command. As used herein, the term "actuator" encompasses a mechanical or electrical device (e.g., a motor) that produces motion, in addition to any driver that may be associated with an actuator and converts a received control command into one or more signals for driving the actuator. Thus, providing a control command to an actuator may include providing a control command to a driver that converts the control command into an appropriate signal to drive an electrical or mechanical device to produce the desired motion.

[0103] The robotic control system 560 may be implemented in one or more processors, such as a CPU, GPU, and / or other controller of the robot 520. In some embodiments, the robot 520 may include a "brain box" that may include all or multiple aspects of the control system 560. For example, the brain box may provide real-time data bursts to the operating components 540a-540n, wherein each of the real-time bursts includes a set of one or more control commands that specify, among other things, motion parameters (if any) for each of one or more of the operating components 540a-540n. In some embodiments, the robotic control system 560 may perform one or more aspects of the methods described herein, such as Figure 3 Method 300 and / or Figure 4 Method 400.

[0104] As described herein, in some embodiments, when controlling a robot during performance of a robotic task, all or multiple aspects of the control commands generated by the control system 560 may be generated based on a robot skill determined to be relevant to the robotic task based on the world fit metrics and task fit metrics described herein. Figure 5 5, the control system 560 is shown as an integral part of the robot 520, but in some embodiments, all or multiple aspects of the control system 560 may be implemented in a component separate from but in communication with the robot 520. For example, all or multiple aspects of the control system 560 may be implemented on one or more computing devices (such as the computing device 610) that are in wired and / or wireless communication with the robot 520.

[0105] Figure 6 6 is a block diagram of an example computing device 610 that may optionally be used to perform one or more aspects of the techniques described herein. The computing device 610 generally includes at least one processor 614 that communicates with a number of peripheral devices via a bus subsystem 612. These peripheral devices may include a storage subsystem 624 (which includes, for example, a memory subsystem 625 and a file storage subsystem 626), a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices allow for user interaction with the computing device 610. The network interface subsystem 616 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0106] The user interface input device 622 may include a keyboard, a pointing device (such as a mouse, a trackball, a touch pad or a graphics tablet, a scanner, a touch screen incorporated into a display), an audio input device (such as a voice recognition system, a microphone), and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computing device 610 or onto a communication network.

[0107] The user interface output device 620 may include a display subsystem, a printer, a fax machine, or a non-visual display, such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device (such as a liquid crystal display (LCD)), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. Generally speaking, the use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 610 to a user or another machine or computing device.

[0108] The storage subsystem 624 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 624 may include a Figure 3 Method 300 and / or Figure 4 The logic of selected aspects of method 400 .

[0109] These software modules are typically executed by the processor 614 alone or in combination with other processors. The memory 625 used in the storage subsystem 624 may include multiple memories, including a main random access memory (RAM) 630 for storing instructions and data during program execution and a read-only memory (ROM) 632 in which fixed instructions are stored. The file storage subsystem 626 can provide persistent storage for program and data files, and can include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Functional modules implementing certain embodiments can be stored by the file storage subsystem 626 in the storage subsystem 624, or stored in other machines accessible by the processor 614.

[0110] The bus subsystem 612 provides a mechanism for the various components and subsystems of the computing device 610 to communicate with each other as intended. Although the bus subsystem 612 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0111] The computing device 610 can be of different types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the Figure 6 The description of computing device 610 depicted in FIG. 6 is intended only as a specific example for illustrating some embodiments. Many other configurations of computing device 610 are possible, and these configurations are similar to those of FIG. Figure 6 The computing devices depicted in FIG. 1 may have more or fewer components than those depicted in FIG.

[0112] In some embodiments, a method implemented by one or more processors is provided, the method comprising identifying an instruction and processing the instruction using a language model (LM) (e.g., a large language model (LLM)) to generate an LM output. The instruction may be a free-form natural language instruction generated based on a user interface input provided by a user via one or more user interface input devices. The generated LM output may model an instruction-dependent probability distribution regarding candidate word formation. The method also includes identifying a robot skill that can be performed by the robot and a skill description that is a natural language description of the robot skill. The method also includes generating a task fit metric for the robot skill based on the LM output and the skill description. The task fit metric may reflect the probability of the skill description in the probability distribution of the LM output. The method also includes generating a world fit metric for the robot skill based on the robot skill and current environment state data. The world fit metric may reflect the probability of success of the robot skill based on the current environment state data. The current environment state data may include sensor data captured by one or more sensor components of the robot in the current environment of the robot. The method also includes determining to implement the robot skill to replace the additional robot skills that can be performed by the robot based on both the task fit metric and the world fit metric. The method also includes, in response to determining to implement the robot skill, causing the robot to implement the robot skill in the current environment.

[0113] These and other implementations of the technology disclosed herein may include one or more of the following features.

[0114] In some embodiments, the method further includes: identifying an additional robot skill in the additional robot skills and an additional skill description as a natural language description of the additional robot skill; and generating an additional task fit metric for the additional robot skill based on the LM output and the additional skill description. The additional task fit metric may reflect the additional probability of the additional skill description in the probability distribution. In those embodiments, the method further includes generating an additional world fit metric for the additional robot skill based on the additional robot skills and the current environment state data. The additional world fit metric may reflect the additional probability of success of the additional robot skill based on the current environment state data. Further, in those embodiments, determining to implement the robot skill to replace each additional robot skill executable by the robot based on both the task fit metric and the world fit metric may include: determining to implement the robot skill based on the task fit metric, the world fit metric, the additional task fit metric, and the additional world fit metric. For example, it may include: generating an overall metric for the robot skill based on both the task fit metric and the world fit metric; generating an additional overall metric for the additional robot skill based on both the additional task fit metric and the additional world fit metric; and determining to implement the robot skill to replace the additional robot skill based on a comparison of the overall metric with the additional overall metric.

[0115] In some embodiments, the method further includes, in response to determining to implement the robot skill: using the LM to process the instruction and the skill description of the robot skill to generate a further LM output, the LM output modeling a further probability distribution of candidate word formations that depends on the instruction and the skill description; identifying the additional robot skills in the additional robot skills and the additional skill description as a natural language description of the additional robot skills; generating an additional task fit metric for the additional robot skill based on the further LM output and the additional skill description; generating an additional world fit metric for the additional robot skill based on the additional robot skill and the updated current environment state data; determining to implement the additional robot skill to replace the robot skill or other additional robot skills that can be performed by the robot based on both the additional task fit metric and the additional world fit metric; and in response to determining to implement the additional robot skill: causing the robot to implement the additional robot skill in the current environment. The updated current environment state data may include updated sensor data captured by one or more sensor components in the current environment and after the implementation of the robot skill.

[0116] In some embodiments, the method further includes, in response to determining to implement the robot skill: using the LM to process the instruction and the skill description of the robot skill to generate a further LM output, the LM output modeling a further probability distribution of candidate word formations that depends on the instruction and the skill description; identifying a terminal skill in the additional robot skill and a terminal description that is a natural language description of the execution of the completed instruction; generating a terminal task fit metric for the terminal skill based on the further LM output and the terminal description; generating a terminal world fit metric for the terminal skill based on the terminal skill and the updated current environment state data; determining to implement the terminal skill to replace the robot skill or other additional robot skills that can be performed by the robot based on both the additional task fit metric and the additional world fit metric; and in response to determining to implement the terminal skill: causing the robot to stop further implementing any control commands of the propulsion instruction. The updated current environment state data may include updated sensor data captured by one or more sensor components in the current environment and after the implementation of the robot skill.

[0117] In some embodiments, generating a world fit metric based on a robot skill and current environment state data includes processing the robot skill and current environment state data using a trained value function to generate a value function output including a world fit metric. In some versions of those embodiments, the current environment state data includes visual data (e.g., a multi-channel image) in sensor data, which is captured by one or more visual components in one or more sensor components of the robot. In some additional or alternative versions of those embodiments, the trained value function is a language-conditioned value function, and processing the robot skill using the trained value function includes processing a skill description of the robot skill. In some additional or alternative versions of those embodiments, the trained value function is trained to correspond to an affordance function, and the value function output specifies whether the robot skill is possible based on the current environment state data. In some additional or alternative versions of those embodiments, the value function is a machine learning model trained using reinforcement learning.

[0118] In some embodiments, determining to implement the robot skill in place of the additional robot skill based on both the task fit metric and the world fit metric includes: generating an overall metric based on the task fit metric and the world fit metric; comparing the overall metric with corresponding additional metrics, each of which is for a corresponding one of the additional robot skills; and determining to implement the robot skill based on the comparison. In some versions of those embodiments, the overall metric is a weighted or non-weighted combination of the task fit metric and the world fit metric. In some additional or alternative versions of those embodiments, the task fit metric is a task fit probability, the world fit metric is a world fit probability, and generating the overall metric includes generating a product based on multiplying the task fit probability by the world fit probability, and using the product as the overall metric.

[0119] In some embodiments, causing the robot to perform the robot skill in the current environment includes causing execution of a language-conditioned robot control policy conditioned on a skill description of the robot skill. In some versions of those embodiments, the language-conditioned robot control policy includes a machine learning model. In some versions of those embodiments, the language-conditioned robot control policy is trained using reinforcement learning and / or imitation learning.

[0120] In some implementations, the instructions strictly adhere to natural language input provided by a user via a user interface input.

[0121] In some embodiments, the instructions do not strictly adhere to the natural language input provided by the user via the user interface input. In some of those embodiments, the method further comprises determining that the natural language input is not in the form of a question, and generating the instructions by modifying the natural language input into the form of a question in response to determining that the natural language input is not in the form of a question.

[0122] In some implementations, the LM is a large language model (LLM).

[0123] In some embodiments, a method implemented by one or more processors is provided, the method comprising recognizing an instruction, the instruction being a free-form natural language instruction generated based on a user interface input provided by a user via one or more user interface input devices. The method also includes processing the instruction using a language model (LM) to generate an LM output, such as an LM output that models a probability distribution that depends on the instruction regarding candidate word formation. The method also includes, for each of a plurality of robot skills that can be performed by the robot: generating a task fit metric for the robot skill based on the LM output and the skill description of the robot skill (e.g., a metric reflecting the probability of the skill description in the probability distribution); generating a world fit metric for the robot skill based on the robot skill and current environment state data (e.g., a metric reflecting the probability of the robot skill being successful based on the current environment state data); and generating an overall metric for the robot skill based on the task fit metric and the world fit metric. The current environment state data includes sensor data captured by one or more sensor components of the robot in the current environment of the robot. The method also includes selecting a given skill in the robot skill based on the overall metric of the robot skill. The method also includes, in response to selecting a given robot skill, causing the robot to implement the given robot skill in the current environment.

[0124] In some embodiments, a method implemented by a processor is provided, and the method includes identifying an instruction that is a free-form natural language instruction, such as a free-form natural language instruction generated based on a user interface input provided by a user via one or more user interface input devices. The method also includes processing an LM prompt generated based on the instruction using a language model (LM) (e.g., a large language model (LLM)) to generate an LM output, which models a probability distribution that depends on the LM prompt about the candidate word formation. The method also includes identifying a robot skill that can be performed by the robot and a skill description that is a natural language description of the robot skill. The method also includes generating a task fit metric for the robot skill based on the LM output and the skill description, the task fit metric reflecting the probability of the skill description in the probability distribution of the LM output. The method also includes generating a world fit metric for the robot skill based on the robot skill and the current environment state data, the world fit metric reflecting the probability of the robot skill success based on the current environment state data. The current environment state data optionally includes sensor data captured by one or more sensor components of the robot in the current environment of the robot. The method also includes determining to implement the robot skill to replace the additional robot skills respectively executable by the robot based on both the task fit metric and the world fit metric. The method also includes, in response to determining to implement the robot skill: causing the robot to implement the robot skill in the current environment.

[0125] In some embodiments, a method implemented by a processor is provided, and the method includes identifying an instruction that is a free-form natural language instruction, such as a free-form natural language instruction generated based on a user interface input provided by a user via one or more user interface input devices. The method also includes generating a large language model (LLM) prompt based on the instruction. The method also includes processing the LLM prompt using the LLM to generate an LLM output that models a probability distribution that depends on the LLM prompt about the candidate word formation. The method also includes, for each of a plurality of robot skills that can be performed by the robot: generating a task fit metric for the robot skill based on the LM output and the skill description of the robot skill, which reflects the probability of the skill description in the probability distribution; generating a world fit metric for the robot skill based on the robot skill and the current environment state data, which reflects the probability of the robot skill being successful based on the current environment state data; and generating an overall metric for the robot skill based on the task fit metric and the world fit metric. The method also includes selecting a given skill in the robot skill based on the overall metric of the robot skill. The method also includes, in response to selecting a given robot skill, causing the robot to implement the given robot skill in the current environment.

[0126] Other embodiments may include a non-transitory computer-readable storage medium storing instructions that are executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform a method, such as one or more of the methods described herein. Other embodiments may include a system of one or more computers and / or one or more robots that include one or more processors that are operable to execute stored instructions to perform a method, such as one or more of the methods described herein.

Claims

1. A method implemented by one or more processors, the method comprising: recognizing instructions that are free-form natural language instructions generated based on user interface input provided by a user via one or more user interface input devices; Processing, using a language model LM, LM prompts generated based on the instructions to generate LM outputs, the LM outputs modeling a probability distribution over candidate word formations that depends on the LM prompts; identifying robot skills that can be performed by the robot and skill descriptions that are natural language descriptions of the robot skills; generating a task fit metric for the robot skill based on the LM output and the skill description, the task fit metric reflecting a probability of the skill description in the probability distribution of the LM output; generating a world fit metric for the robot skill based on the robot skill and current environment state data, the world fit metric reflecting a probability of success of the robot skill based on the current environment state data, wherein the current environment state data comprises sensor data captured by one or more sensor components of the robot in the current environment of the robot; Determining to implement the robotic skill in place of additional robotic skills each capable of being performed by the robot based on both the task fit metric and the world fit metric; and In response to determining to implement the robot skill: The robot is caused to implement the robot skill in the current environment.

2. The method of claim 1, further comprising: identifying an additional robotic skill among the additional robotic skills and an additional skill description that is a natural language description of the additional robotic skill; generating an additional task fit metric for the additional robot skill based on the LM output and the additional skill description, the additional task fit metric reflecting an additional probability of the additional skill description in the probability distribution; generating an additional world fit metric for the additional robot skill based on the additional robot skill and the current environment state data, the additional world fit metric reflecting an additional probability of success of the additional robot skill based on the current environment state data; Wherein, determining to implement the robot skill to replace the additional robot skills that can be performed by the robot based on both the task fit metric and the world fit metric comprises: generating an overall metric for the robot skill, the overall metric being based on both the task-fitting metric and the world-fitting metric; generating an additional overall metric for the additional robotic skill, the additional overall metric being based on both the additional task adaptation metric and the additional world adaptation metric; and A determination is made to implement the robotic skill in place of the additional robotic skill based on a comparison of the overall metric to the additional overall metric.

3. The method of claim 1, further comprising: In response to determining to implement the robot skill: processing, using the LM, further LM prompts based on the instructions and the skill description of the robot skill to generate further LM outputs, the further LM outputs modeling further probability distributions over the candidate word formations that depend on the further LM prompts; identifying an additional robotic skill among the additional robotic skills and an additional skill description that is a natural language description of the additional robotic skill; generating an additional task fitment metric for the additional robot skill based on the further LM output and the additional skill description, the additional task fitment metric reflecting a probability of the additional skill description in the further probability distribution; generating an additional world-fitting metric for the additional robotic skill based on the additional robotic skill and the updated current environment state data, the additional world-fitting metric reflecting a probability of success of the additional robotic skill based on the updated current environment state data, wherein the updated current environment state data comprises updated sensor data captured by the one or more sensor components in the current environment and after performing the robot skill; Determining to implement the additional robotic skill to replace the robotic skill or other of the additional robotic skills that are each executable by the robot based on both the additional task fit metric and the additional world fit metric; and In response to determining to implement the additional robotic skill: The robot is caused to implement the additional robot skill in the current environment.

4. The method of claim 1, further comprising: In response to determining to implement the robot skill: processing, using the LM, further LM prompts based on the instructions and the skill description of the robot skill to generate further LM outputs, the further LM outputs modeling further probability distributions over the candidate word formations that depend on the further LM prompts; identifying a termination skill in the additional robot skills and a termination description that is a natural language description of completing the execution of the instruction; generating a termination task fit metric for the termination skill based on the further LM output and the termination description, the termination task fit metric reflecting a probability of the termination skill description in the further probability distribution; Determining, based on the additional task fit metric, to implement the termination skill to replace the robot skill or other additional robot skills among the additional robot skills that can be performed by the robot; as well as In response to determining to implement the termination skill: Causes the robot to stop further implementation of any control commands that advance the instructions.

5. A method as claimed in any preceding claim, wherein: Generating the world-fitting metric based on the robot skill and the current environment state data includes: The robot skill and the current environment state data are processed using the trained value function to generate a value function output including the world conformity metric.

6. The method of claim 5, wherein: The current environment state data includes visual data in the sensor data, and the visual data is captured by one or more visual components in the one or more sensor components of the robot.

7. The method of claim 6, wherein: The visual data includes a multi-channel image.

8. The method of claim 5, wherein: The trained value function is a language-conditioned value function, and wherein processing the robotic skill using the trained value function includes processing the skill description of the robotic skill.

9. The method of claim 5, wherein: The trained value function is trained to correspond to an affordance function, and the value function output specifies whether the robot skill is possible based on the current environment state data.

10. The method of claim 5, wherein: The value function is a machine learning model trained using reinforcement learning.

11. A method as claimed in any preceding claim, wherein: Determining to implement the robotic skill to replace the additional robotic skill based on both the task fit metric and the world fit metric includes: generating an overall metric based on the task fit metric and the world fit metric; comparing the overall metric to corresponding additional metrics, each for a corresponding one of the additional robotic skills; and The robot skill is performed based on the comparison.

12. The method of claim 11, wherein: The overall metric is a weighted or unweighted combination of the task fit metric and the world fit metric.

13. The method of claim 12, wherein: The task fit metric is a task fit probability, the world fit metric is a world fit probability, and generating the overall metric includes generating a product based on multiplying the task fit probability by the world fit probability, and using the product as the overall metric.

14. A method as claimed in any preceding claim, wherein: Enabling the robot to perform the robot skill in the current environment includes: A language is used to condition the execution of a robot control strategy on the skill description of the robot skill.

15. The method of claim 14, wherein: The language-conditioned robotic control strategy includes a machine learning model.

16. The method of claim 15, wherein: The language-conditioned robotic control strategy is trained using reinforcement learning and / or imitation learning.

17. A method as claimed in any preceding claim, wherein: The LM prompts strictly adhere to the natural language input provided by the user via the user interface input.

18. The method according to any one of claims 1 to 16, wherein: The LM prompt does not strictly adhere to the natural language input provided by the user via the user interface input, and the method further comprises: determining that the natural language input is not in the form of a question; and In response to determining that the natural language input is not in the form of a question, the LM prompt is generated by modifying the natural language input into the form of a question.

19. The method according to any one of claims 1 to 16, wherein: The LM prompt does not strictly adhere to the natural language input provided by the user via the user interface input, and the method further comprises: The LM prompt is generated to include as a suffix the content requesting the next step of generation.

20. The method of any one of claims 1 to 16, wherein: The LM prompt does not strictly adhere to the natural language input provided by the user via the user interface input, and the method further comprises: The LM prompt is generated to include one or more scene descriptors describing a current environment of the robot.

21. A method as claimed in any preceding claim, wherein the LM is a Large Language Model (LLM).

22. A method implemented by one or more processors, the method comprising: recognizing instructions that are free-form natural language instructions generated based on user interface input provided by a user via one or more user interface input devices; Generate a large language model (LLM) prompt based on the instruction; processing the LLM prompt using an LLM to generate an LLM output that models a probability distribution over candidate word formations that is dependent on the LLM prompt; For each of a plurality of robot skills each capable of being performed by the robot: generating a task fit metric for the robot skill based on the LLM output and the skill description of the robot skill, the task fit metric reflecting a probability of the skill description in the probability distribution; generating a world fit metric for the robot skill based on the robot skill and current environment state data, the world fit metric reflecting a probability of success of the robot skill based on the current environment state data, wherein the current environment state data comprises sensor data captured by one or more sensor components of the robot in the current environment of the robot; and generating an overall metric for the robot skill based on the task-fitting metric and the world-fitting metric; selecting a given one of the robot's skills based on the overall measure of the robot's skills; and In response to selecting the given robot skill: The robot is caused to perform the given robot skill in the current environment.

23. A robot comprising: one or more actuators; End effector; a memory storing instructions; One or more processors operable to execute the instructions to perform a method as claimed in any preceding claim.

24. A system comprising a memory storing instructions and one or more processors operable to execute the instructions to perform the method of any one of claims 1 to 22.

Citation Information

Cited By

  • Method and device for generating passable area of humanoid robot and humanoid robot

    CN119772880A

  • Human-centered visual perception method facing human-machine cooperation scene

    CN120299081A

  • Scene perception and task reasoning system oriented to man-machine cooperation

    CN120764662A

  • Control method and device based on semantic perception and related equipment

    CN121500788A