Natural language control of robots

Abstract

Embodiments use a large language model to process free-form natural language (NL) instructions and generate an LLM output. Those embodiments generate a task grounding measure that reflects the probability of the skill description in the probability distribution of the LLM output, based on the LLM output and the NL skill description of the robot skills. Those embodiments further generate a world grounding measure that reflects the probability that the robot skill is successful, based on the current environmental state data, based on the robot skills and the current environmental state data. Those embodiments further determine whether to execute the robot skill, based on both the task grounding measure and the world grounding measure.

Description

[Background technology]

[0001] Many robots are programmed to perform specific tasks. For example, a robot on an assembly line may be programmed to recognize certain objects and perform specific operations on those objects.

[0002] Furthermore, some robots can perform specific tasks in response to explicit user interface input corresponding to those tasks. For example, a cleaning robot can perform a general cleaning task in response to the utterance, "Robot, clean." However, typically, user interface input that causes a robot to perform a task must be explicitly mapped to that task. Therefore, a robot may not be able to perform a specific task in response to various free-form natural language inputs from a user attempting to control it. [Overview of the project]

[0003] Language models (LMs) have been developed that can be used to process natural language (NL) content and / or other inputs to produce LM outputs that reflect the NL content and / or other content in response to the inputs. For example, large-scale language models (LLMs) have been developed that are trained on large amounts of data and can be used to robustly process a wide range of NL inputs and to produce corresponding LM outputs that accurately reflect the corresponding NL content in response to the NL inputs. An LLM may contain at least several hundred million parameters, and often at least several billion, such as more than 100 billion parameters. An LLM may include, for example, a sequence-to-sequence model, which may be Transformer-based, and / or an encoder and / or decoder. One non-restrictive example of an LLM is Google's Pathways Language Model (PaLM). Another non-restrictive example of an LLM is Google's Language Model for Dialogue Applications (LaMDA).

[0004] Separately, efforts are being made to enable robust free-form (FF)NL control of robots. For example, this involves enabling robots to respond appropriately to any one of several different types of verbal commands directed at them by humans. For instance, in response to the FF NL command "Put the block in the toy box," the robot could perform a task that includes (a) navigating to the "block," (b) picking up the "block," (c) navigating to the "toy box," and (d) placing the "block" into the "toy box."

[0005] The embodiments disclosed herein recognize that LMs such as LLMs can encode rich semantic knowledge about the world, and that such knowledge can be useful to a robot when acting on high-level, temporally extended instructions expressed in FF NL instructions. The embodiments disclosed herein further recognize that a drawback of LLMs is that they lack real-world experience, such as real-world experience in robot control, and / or grounding to the current real-world state(s). For example, prompting an LLM with "I spilled a drink on the table, please help" may result in the generation of an LLM output that reflects NL content describing reasonable steps(s) to clean up the spill. For example, the most likely decoding of the LLM output might reflect NL content such as "Why not use the vacuum cleaner?" However, "Why not use the vacuum cleaner?" may not be applicable to certain agents, such as a robot that needs to perform this task in a specific environment. For example, a robot may not have an integrated vacuum cleaner, and in certain environments, no separate vacuum cleaner may be available, or it may not be able to control a separate vacuum cleaner.

[0006] The embodiments disclosed herein describe tasks at a high level and utilize LM output to determine how to control a robot to execute a task in response to a FF NL input that does not describe all (or any) of the robot skill(s) necessary to execute the task. However, in view of the recognized drawback(s) of the LM, the embodiments described herein ground the LM output using one or more techniques such that the robot skill(s) selected based on the LM output for execution in response to the FF NL input are both executable (e.g., executable by the robot) and contextually appropriate (e.g., likely to succeed if executed by the robot in a particular environment). More specifically, the embodiments ground the LM output considering not only the LM output but also robot skills executable by the robot such as pre-trained robot skills.

[0007] As some non-limiting examples of the operation of those embodiments, assume that a user gives a FF NL command of "I spilled a drink on the table, please help". An LLM prompt can be generated based on the FF NL command. For example, the LLM prompt can strictly conform to the FF NL command. As another example, as described herein, the LLM prompt may be based on the FF NL command but may not strictly conform. For example, the prompt may include some or all of the terms of the FF NL command, but additionally, a scene descriptor(s) of the current environment (e.g., NL descriptor(s) of the object(s) detected in the environment) and an explanation generated in a previous pass using the LLM (e.g., based on a previous LLM prompt that explains how to help when the user says "I spilled a drink on the table, please help") and / or terms(s) that prompt the prediction of step(s) by the LLM (e.g., including "I will do 1." at the end of the prompt).

[0008] The generated LLM prompt may be processed using LLM to produce an LLM output that models the probability distribution of candidate word constructs dependent on the instruction. Continuing with the example, the most probable decoding of the LLM output might be, for example, "Use the vacuum cleaner." However, the embodiments disclosed herein do not simply rely on the probability distribution of the LLM output when determining how to control the robot. Rather, the embodiments leverage the probability distribution of the LLM output while also considering robot skills that the robot can actually perform, such as tens, hundreds, or thousands of pre-trained robot skills. Some versions of those embodiments generate a corresponding task grounding scale and a corresponding world grounding scale for each robot skill considered. Furthermore, some of those versions select a particular robot skill to perform in response to an FF NL prompt based on considering both the corresponding task grounding scale and the corresponding world grounding scale. For example, a corresponding global scale may be generated for each robot skill as a function of the corresponding world grounding and task grounding scales for the robot skill, and a particular robot skill may be selected based on having the best corresponding global scale.

[0009] When generating a task grounding metric for a robotic skill, the NL skill description of the robotic skill can be compared with the LLM output to generate a task grounding metric. For example, the task grounding metric can reflect the probability of the NL skill description in the probability distribution modeled by the LLM output. For example, continuing with the example of actions, the task grounding metric for a first robotic skill with the NL skill description of "pick up a sponge" can reflect a higher probability than the task grounding metric for a second robotic skill with the NL skill description of "pick up a banana". In other words, the probability distribution of the LLM output can characterize that the probability of "pick up a sponge" is higher than that of "pick up a banana". Also, for example, continuing with the example of actions, the task grounding metric for a third robotic skill with the NL skill description of "pick up a squeegee" can reflect a probability similar to that of the first robotic skill.

[0010] When generating a world grounding metric for a robotic skill, current state data such as environmental state data and / or robotic state data can optionally be considered. Environmental state data reflects the state of one or more objects (multiple possible) that exist in the environment in addition to the robot, and can include, for example, visual sensor data and / or decisions made by processing visual sensor data (e.g., object detection (multiple possible) and / or classification (multiple possible)). Robotic state data reflects the state of one or more components of the robot and can include, for example, the current position of the robot's components (multiple possible), the current speed of the robot's components (multiple possible), and / or other state data.

[0011] In some embodiments, or for some robot skills, a description of the robot skill may also be considered. In some of those embodiments, the description of the robot skill (e.g., its word embeddings) is processed, and current state data is processed using a trained value function model to generate a value that reflects the probability that the robot skill is successful based on the current state data. A world grounding scale is generated based on that value (e.g., conforming to that value). In some versions of those embodiments, candidate robot actions are also processed using the trained value function model along with the description and current state data. In some versions of those embodiments, multiple values ​​are generated for a given robot skill, each processing a different candidate robot action but utilizing the same description and the same current state data. In those versions, a world grounding scale may be generated based on the generated value that reflects the highest probability of success, for example.

[0012] Continuing with the example, let's assume that the current environmental state data includes images captured by the robot's camera, and that these images capture a nearby sponge and a nearby banana, but not a squeegee. In this situation, the world grounding scale for the first robot skill with the NL skill description "pick up the sponge" would reflect a high probability, the world grounding scale for the second robot skill with the NL skill description "pick up the banana" would also reflect a high probability, and the world grounding scale for the third robot skill with the NL description "pick up the squeegee" might reflect a low probability.

[0013] Continuing with the example, the first robot skill, described as "picking up a sponge," can be selected for implementation based on considering both its task grounding scale and its world grounding scale, both of which reflect high probabilities. Note that the second robot skill will not be selected because it has a low task grounding scale despite having a high world grounding scale. Similarly, the third robot skill will not be selected because it has a low world grounding scale despite having a high task grounding scale.

[0014] In these and other methods, embodiments disclosed herein may consider both the Task Grounding Scale and the World Grounding Scale when selecting robot skills to perform. This ensures that the selected robot skills are both (a) likely to lead to successful task completion as reflected by the FF NL input (as reflected by the Task Grounding Scale), and (b) likely to succeed when performed by the robot in the current environment (as reflected by the World Grounding Scale).

[0015] The above description is provided as an overview of only some of the embodiments disclosed herein. These and other embodiments, including modes for carrying out the invention and claims, are described in more detail herein.

[0016] It should be understood that all combinations of the aforementioned concepts and any additional concepts described in more detail herein are intended to be part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are intended to be part of the subject matter disclosed herein. [Brief explanation of the drawing]

[0017] [Figure 1A]This example shows how a human can give free-form (FF) natural language (NL) commands to an exemplary robot. [Figure 1B] Figure 1A shows a simplified bird's-eye view of an exemplary environment in which the human and robot are located, and illustrates an exemplary instance of current visual data that can be captured within the environment. [Figure 2A] This diagram illustrates the process flow of how various exemplary components can interact when selecting initial robot skills to perform in the environment shown in Figure 1B in response to the FF NL instruction in Figure 1A. [Figure 2B] The process flow shows how various exemplary components can interact when selecting the next robot skill to perform in the environment of Figure 1B, in response to the FF NL command in Figure 1A, following (or during) the execution of the robot skill selected in Figure 2A. [Figure 3] This flowchart illustrates an exemplary method of utilizing LLM in the execution of a robotic skill(s) that performs a task reflected in an FF NL instruction, according to embodiments disclosed herein. [Figure 4] This is a flowchart illustrating an exemplary method for generating a global grounding scale for robot skills based on processing current state data and skill descriptions of robot skills. [Figure 5] A schematic diagram of an exemplary robot architecture is shown. [Figure 6] This provides a schematic diagram of an exemplary computer system architecture. [Modes for carrying out the invention]

[0018] Before delving into a detailed description of the drawings, we provide a non-limiting overview of various embodiments.

[0019] In various embodiments disclosed herein, the robot has a repertoire of learned robot skills for atomic behaviors that are capable of low-level visuomotor control. Some of these embodiments not only prompt the LLM to interpret FF NL high-level instructions, but also utilize the LLM output generated by the prompting to generate a task grounding scale that quantifies the likelihood that the corresponding robot skill will progress toward completing the high-level instruction. Furthermore, a corresponding affordance function (e.g., a learned value function) for each robot skill may be used to generate a world grounding scale for the robot skills that quantifies the probability that the robot skill will succeed from the current state. Furthermore, both the task grounding scale and the world grounding scale may be used to determine which robot skill to perform next in achieving the task(s) reflected by the high-level instructions. In these and other methods, the embodiments leverage the fact that the LLM output describes the probability that each skill contributes to completing the instruction, the affordance function describes the probability that each skill succeeds, and by combining these two, the probability that each skill will succeed in executing the instruction is obtained. The affordance function allows for the consideration of real-world grounding in addition to task grounding of the LLM output, and by constraining completion to skill descriptions, it becomes possible to consider the LLM output in a way that recognizes the robot's capabilities (e.g., in terms of the repertoire of learned robot skills). Furthermore, this combination yields an interpretable plan expressed through language, which is a fully explainable sequence of steps that the robot will perform to complete a high-level instruction (e.g., a description of the robot skills selected for implementation).

[0020] Therefore, embodiments leverage LLM to provide task grounding for determining useful actions for high-level goals and leverage affordance functions (e.g., trained functions) to provide world grounding for determining which actions are actually feasible to achieve high-level goals. Some of these embodiments utilize reinforcement learning (RL) as a method for learning linguistic conditional value functions that provide affordances for what is possible in the real world.

[0021] The language model is a sequence of strings w, namely text W = {w0, w1, w2, ..., w n We attempt to model the probability p(W) of}. This is usually done so that each consecutive string is predicted from the previous string.

number

[0022] Embodiments disclosed herein utilize the vast semantic knowledge contained in an LLM to determine useful tasks for resolving high-level instructions. Some of these embodiments further attempt to accurately predict whether a skill (given by an NL descriptor) is executable in the current state of the environment of the robot or other agent. In some versions of these embodiments, or for at least some skills, time-difference-based (TD) reinforcement learning is used to achieve this goal. In some versions of these embodiments, the Markov decision process is defined as (MDP)M=(S,A,P,R,γ), where S and A are the state space and action space, respectively.

number

number

number

number

[0023] For example, TD-based methods may be used to learn a value function such as a value function that additionally conditions on natural language descriptors of skills, and the value function may be utilized to determine whether a given skill is executable from a given state. Notably, in the case of undiscounted sparse rewards, the agent receives a reward of 1.0 at the end of an episode if successful and 0.0 otherwise, and the value function trained by RL corresponds to an affordance function that specifies whether a skill is possible in a given state.

[0024] As disclosed herein, various embodiments receive a user-provided natural language instruction i that describes a task for a robot to perform. The instruction may be long, abstract, and / or ambiguous. Further, the embodiments can utilize a set of robot skills II, where each skill π ∈ II performs a short task such as picking up a particular object and has a short language description l π (e.g., "find the sponge") and an affordance function p(c π │s,l π ). The affordance function indicates the probability of successfully completing a skill with description l from state s. Intuitively, p(c π │s,l π │s,l π ) means "if asked the robot to do l, will the robot do so?". In RL terms, p(c π │s,l π │s,l π ) is the value function of the skill where the reward for completion success is 1 and 0 otherwise.

[0025] As described above, l π denotes the text label of skill π, and p(c π │s,l π ) indicates the probability of successfully completing when skill π with text label l is executed from state s. Here, c π denotes the completion of the skill, and c πis a Bernoulli random variable. By processing prompts based on natural language commands i using LLM, the probability p(l) of the skill's text label being a valid next step for the user's command can be determined. π An LLM output is generated that characterizes |i). However, the embodiments disclosed herein are p(c i │i,s,l π It is important to consider the probability that a given skill will successfully complete the command, which is expressed as p(l). π If we assume that the process proceeds with i) (i.e., the probability that the skill is appropriate) and that the skill that fails proceeds with a probability of zero, then this is p(c i │i,s,l π )∝p(c π │s,l π )p(l π It can be factorized as │i), which is sometimes referred to herein as the task grounding or task grounding measure. Furthermore, the probability that a skill is possible in the current state of the world is given by p(c π │s,l π It can be factorized into ), which may be referred to herein as the World Grounding or World Grounding Scale.

[0026] While LLMs can draw upon a wealth of knowledge learned from large amounts of text, they do not always break down high-level commands into lower-level instructions suitable for robot execution. For example, if an LLM is asked, "How does the robot bring the apple?", it might respond, "The robot can go to a nearby store and buy the apple." While this response is a reasonable completion of the prompt, it is not always effective for embodying agents such as robots, which may have a narrow, fixed set of capabilities. Therefore, to adapt LLMs or other LMs, some embodiments attempt to implicitly inform the LM that high-level instructions should be broken down into a sequence of available lower-level skills. One technique to accomplish this is careful prompt engineering, a technique that guides the LM to a specific response structure. Prompt engineering provides an example in contextual text ("prompt") for the LM that specifies the task and response structure that the LM should emulate.

[0027] The scoring language model opens a means to a constrained response by outputting probabilities assigned by the LM to a fixed output. The LM is the distribution of potential completion p(w k │w <k ) represents, and here, w k is the word appearing at the k-th position in the text. Typical generative applications (e.g., conversational agents) sample from this distribution or decode maximum likelihood completes, but embodiments disclosed herein may instead use the distribution to score candidate completes selected from an optional set (e.g., candidate skill descriptions from a set of candidate skill descriptions). More formally, a set of lower-level robot skills II, their language descriptions l II Assuming that command i, the skill l progresses toward the execution of command i. π ∈l II The probability p(l) of the language description π │i) can be calculated. This corresponds to querying the model for potential completion. The optimal skill by the language model is,

number

[0028] In some embodiments, the process proceeds by iteratively selecting skills and adding them to commands. In practice, this can be viewed as a user-robot interaction, where the user gives a high-level command (e.g., "Can you bring me a can of cola?"), and the language model responds in an explicit order ("I, 1.", e.g., "I, 1. find a can of cola, 2. pick up a can of cola, 3. bring it to you"). This has the added advantage of interpretability, as generative responses are generated along with the concept of likelihood across many possible responses.

[0029] While such methods allow for the effective extraction of knowledge from language models, a major problem remains: the decoding of instructions obtained in this way always consists of the skills available to the robot, but these skills may not always be appropriate for performing the desired high-level task in the specific situation the robot is currently in. For example, if the FF NL prompt is "bring me an apple," the optimal set of skills will change depending on whether the apple is not within the robot's field of view or if the robot already has the apple in its gripper.

[0030] Therefore, the embodiment attempts to ground a large-scale language model through a value function, i.e., an affordance function that captures the log-likelihood of a particular skill succeeding in the current state. Skill π∈II, its language description l π , and in state s, l π The completion probability p(c) of the skill described by π │s,l π Assuming the corresponding value function gives, the affordance space {p(c π │s,l π)}π∈II can be formed. This value function space captures affordances across all skills. For each skill, the affordance function can be multiplied by the probability of LLM, and finally the most likely skill, i.e., π=argmax π∈II p(c π │s,l π )p(l π │i) is selected.

[0031] When the most likely skill is selected, the corresponding policy is executed by the agent, and the LLM query is l π The process is modified to include (the language description of the selected skill) and executed again until an exit token (e.g., "end") is selected. This process is described in Algorithm 1, given below. Together these two mirrored processes lead to a probabilistic interpretation, where the LLM provides the probability of a useful skill for a high-level instruction, and the affordances provide the probability of success for the execution of each skill. Combining these two probabilities gives the probability that this skill will further advance the execution of the high-level instruction commanded by the user. Algorithm 1 Assumption: High-level instruction i, state s0, and set of skills II and their language descriptions l π 1: n=0, π=Φ 2: while l πn-1 ≠done do 3: C=Φ 4: for π∈II and l π ∈l II do 5:

number

number

number

number

[0032] Embodiments disclosed herein utilize a set of skills, each of which has a policy, a value function, and a short linguistic description (e.g., "Pick up the can") (for use by an agent when performing the skill). These skills, value functions, and descriptions can be acquired in a variety of different ways. For example, individual skills may be trained using image-based behavior cloning (e.g., following BC-Z) or reinforcement learning (e.g., following MT-Opt). Regardless of how the skill policy is acquired, a value function, such as a value function trained by TD backup, may be used for the skill. To amortize the cost of training many skills, multitask BC and / or multitask RL may be used for one or more skills. In multitask BC and / or multitask RL, instead of training separate policies and value functions for each skill, a multitask policy and model conditioned on the NL skill description is trained. However, it should be noted that this description only addresses low-level skills, and LLM is still used to interpret high-level instructions and break them down into individual low-level skill descriptions.

[0033] In some embodiments, a pre-trained large-scale sentence encoder language model may be used to condition the policy on language. The parameters of the sentence encoder language model may be frozen during training, and the embeddings generated by passing text descriptions of each skill may be the embeddings used to condition the policy. The text embeddings are used as input to the policy and value function that specify which skills should be performed. Since the language model used to generate the text embeddings is not necessarily the same as the language model used for planning, embodiments can utilize different language models that are better suited to different levels of abstraction, and understand the plan with respect to many skills rather than representing specific skills at a higher granularity.

[0034] In some embodiments, BC and / or RL policy training procedures may be used to acquire one or more linguistic conditional policies and one or more value functions, respectively. As previously mentioned, for skill specifications, a set of short natural language descriptions may be available, represented as embeddings of a language model. Furthermore, a sparse reward function may be available, such as having a reward value of 1.0 at the end of an episode if the execution of a linguistic command is successful, and a reward value of 0.0 otherwise. The success of the execution of a linguistic command may be rated, for example, by a human, who is given a video of the robot performing the skill along with a given command. If two out of three raters (or other thresholds) agree that the completion of the skill was successful, the episode is labeled with a positive reward. The action space of the policy may include, for a robot similar to the one shown in Figure 1A as an example, six-degree-of-freedom end-effector posture and gripper open / close commands, the xy position and yaw delta of the robot's movement base, and a termination action.

[0035] Various robotic skills, such as manipulative and navigation skills using a mobile manipulator robot, may be utilized in the embodiments disclosed herein. Such skills may include picking, placing, and rearranging objects, opening and closing drawers, navigating to various locations, and arranging objects in specific configurations.

[0036] RL models, used in several embodiments such as RL policy models and / or RL value function models, may use an architecture similar to MT-Opt, with optional modifications to support natural language input. In one particular example, camera images (e.g., from a robot's camera) may be processed by convolutional layers of the architecture's image tower to generate image embeddings. Skill descriptions may be embedded by an LLM (or other embedding model) and then concatenated with non-image portions of the state, such as robot behavior and gripper height. To support asynchronous control, inference may be performed while the robot is still moving from a previous action. The model is optionally given the remaining amount to be performed from the previous action. Adjustment inputs pass through fully connected layers of an additional tower of the architecture and are then spatially tiled to generate additional embeddings. The additional embeddings are appended / concatenated to the image embeddings before passing through an additional convolutional layer. The output is gated through a sigmoid so that the Q value is always [0,1].

[0037] BC models, used in several embodiments such as BC policy models, may employ architectures similar to BC-Z. In one specific example, a skill description may be embedded by a universal sentence encoder, which is then used to condition a Resnet-18 based architecture using FiLM. Unlike RL models, previous actions or gripper height may not be provided. Multiple FC layers may be applied to the final visual features to output each action component (arm position, arm direction, gripper, and ending action).

[0038] It should be noted that the flexibility of the embodiments disclosed herein allows for the mixing and matching of policies and affordances from different methods. For example, for picking skills, a single multitasking linguistic conditional policy can be used; for placement skills, a scripted policy using affordances based on the gripper state can be used; and for navigation policies, a plan-based approach can be used that recognizes the location(s) where a particular object(s) may be found and the corresponding distance(s). In some embodiments, an upper limit may be set on the affordance indicating that a skill is completed and a reward has been received, in order to avoid situations where a skill is selected but already performed, or where it would be ineffective.

[0039] As described above, the embodiments disclosed herein, through their probabilistic interface, enable the use of many different policies and / or affordance functions. Therefore, as skills become more advanced or as new skills are learned, it is easy to incorporate such skills into the embodiments disclosed herein.

[0040] As described above, the embodiment is described from state s to l π The affordance function p(c) shows the probability of successfully completing the skill using this function. π │s,l π ) is used. Some of the trained policies that may be used in this specification are the Q function Q π Create (s,a). Q is created with action a and state s. π Given (s,a), similar to MT-Opt, the value v(s) = max can be obtained through optimization using the cross-entropy method. a Q π (s,a) is found. For simplicity, some example value functions are described below in their skill text descriptions. π and

number

[0041] An exemplary affordance function for the "pick" skill is as follows: The trained value function for the pick skill generally has a minimum value when the skill is impossible and a maximum value when the skill is successful, and therefore, by normalizing the value function, we obtain the following affordance function.

number

[0042] An example affordance function for the "goto" / "navigate" skill is as follows: The affordance function for the goto skill is based on the distance d (meters) to the location.

number

[0043] An example affordance function for the terminate skill is:

number

[0044] Referring to the figure, Figure 1A shows an example in which a human 101 gives an exemplary robot 110 a free-form (FF) natural language (NL) command 105, "Bring me a snack from the table."

[0045] The robot 110 shown in Figure 1 is a specific mobile robot. However, additional and / or alternative robots, such as additional robots that differ in one or more respects from the robot 110 shown in Figure 1A, may be available in conjunction with the technologies disclosed herein. For example, in the technologies described herein, mobile forklift robots, unmanned aerial vehicles ("UAVs"), non-mobile robots, and / or humanoid robots may be used instead of or in addition to robot 110.

[0046] The robot 110 includes a base 113 on which wheels are provided on its opposing sides for the movement of the robot 110. The base 113 may include, for example, one or more motors for driving the wheels of the robot 110 to achieve movement of the robot 110 in a desired direction, at a desired speed, and / or acceleration. The robot 110 also includes a robot arm 114 having an end effector 115 in the form of a gripper having two opposing "finger" or "digit" parts.

[0047] The robot 110 also includes a visual component 111 that can generate visual data (e.g., images) relating to the shape, color, depth, and / or other features of an object(s) within the line of sight of the visual component 111. The visual component 111 may be, for example, a monocular camera, a stereographic camera (active or passive), and / or a 3D laser scanner.

[0048] A 3D laser scanner may include one or more lasers that emit light and one or more sensors that collect data related to the reflection of the emitted light. A 3D laser scanner may generate visual component data, which is a 3D point cloud, where each point in the 3D point cloud defines the position of a point on a surface in 3D space. A monocular camera may include a single sensor (e.g., a charge-coupled device (CCD)) and generate an image in which each of several data points defines color and / or grayscale values ​​based on physical properties sensed by the sensor. For example, a monocular camera may generate an image in which red, blue, and / or green channels are included. Each channel may define a value for each of several pixels in the image, such as a value between 0 and 255 for each pixel in the image. A stereographic camera may include two or more sensors, each at a different viewpoint. In some of these embodiments, a stereographic camera generates an image in which each of several data points defines depth values ​​and color and / or grayscale values ​​based on properties sensed by the two sensors. For example, a stereographic camera may produce an image that includes a depth channel and red, blue, and / or green channels.

[0049] The robot 110 may also include one or more processors. One or more processors may, for example, use LLM to process LLM prompts based on the FF NL input 105 and generate LLM outputs, determine a description of robot skills and a value function(s) for the robot skills based on the LLM outputs, determine that the robot skills(s) are to be implemented when performing robot tasks, control the robot 110 during the execution of robot tasks based on the determined robot skills(s), etc. For example, one or more processors of the robot 110 may implement all or aspects of methods 300 and / or 400 described herein. Additional descriptions of some examples of various robot structures and functionalities are provided herein.

[0050] Referring now to Figure 1B, a simplified bird's-eye view of an exemplary environment in which the human 101 and robot 110 are located in Figure 1A is shown. The human 101 and robot 110 are represented by circles in Figure 1B. Furthermore, environmental features 191, 192, 193, and 194 are shown in Figure 1B. Environmental features 191, 192, 193, and 194 outline various landmarks in the environment. For example, the environment may be an office kitchen or a workshop kitchen, features 191 and 192 may be countertops, feature 193 may be a sink, and feature 194 may be a round table.

[0051] Figure 1B also shows an example of a current visual data instance 180 that is captured in the environment and can be used, for example, when generating a world grounding scale for robot skills. For example, robot 110 may capture the current visual data instance 180 using visual component 111. Visual data instance 180 captures a pear 184A and a key 184B, both of which reside on a round table represented by feature 194. Note that in the bird's-eye view, the pear and key are shown as points for simplification.

[0052] Referring here to Figure 2A, the process flow of how various exemplary components may interact in response to the FF NL instruction 150 in Figure 1A and in selecting initial robot skills to perform within the environment of Figure 1B. The exemplary components shown in Figure 2A include the LLM engine 130, LLM 150, task grounding engine 132, world grounding engine 134, value function model(s), selection engine 130, and execution engine 136. One or more of the illustrated components may be performed by the robot 110 (e.g., using a processor(s) and / or its memory) and / or using a remote computing device(s) (e.g., cloud-based server(s)) that communicates with the robot 110 over a network.

[0053] In Figure 2A, the LLM engine 130 generates an LLM prompt 205A based on the FF NL input 105 ("Bring me a snack from the table"). The LLM engine 130 may generate an LLM prompt 205A that strictly conforms to the FF NL input 105, or it may generate an LLM prompt 205A that is based on the FF NL input 105 but does not strictly conform to it. For example, as shown by LLM prompt 205A1, a non-restrictive example of LLM prompt 205A, the LLM prompt could be "How would you bring me a snack from the table? I would 1." Such an LLM prompt 205A1 includes the prefix "How would you" and the suffix "I would 1." Either or both of these may facilitate the prediction of steps(s) related to achieving the high-level task specified by the FF NL input 105 in the LLM output.

[0054] In some embodiments, the LLM engine 130 may optionally generate an LLM prompt 205A based further on one or more of the scene descriptors 202A, prompt examples 203A, and / or descriptions 204A of the robot 110's current environment.

[0055] Scene descriptors 202A may include NL descriptors of currently or recently detected objects in the environment having the robot 110, such as descriptors of objects determined based on processed images or other visual data using object detection and classification machine learning models. For example, scene descriptors 202A may include "pear," "key," "human," "table," "sink," and "countertop," and the LLM engine 130 may generate an LLM prompt 205A to incorporate one or more of such descriptors. For example, LLM prompt 205A may be "Pear, key, human, table, sink, and countertop are nearby. How do I get a snack from the table? I'll go with 1."

[0056] Prompt example(s) 203A may include manually designed examples(s) of any desired output style(s). For example, they may include “in step-by-step format,” “in numbered list format,” or “in the style of 1. Step 1, 2. Step 2, 3. Step 3, etc.” Prompt example(s) 203A may be added before LLM prompt 205A or otherwise incorporated into LLM prompt 205A to facilitate prediction of content in output style(s) in the LLM output. Description 204A may be a description generated based on processing the previous LLM prompt, based on the FF NL input in the previous pass, and utilizing LLM150. For example, the previous LLM prompt may be “Describe how to get the snack from the table,” and description 204A may be generated based on the most probable decoding from the previous LLM output. For example, description 204A may be “Find the table, then find the snack on the table, then take it.” Description 204A may be added before LLM prompt 205A, replace the term(s) of FF NL input 105 within LLM prompt 205A, or otherwise incorporated into LLM prompt 205A.

[0057] The LLM engine 130 processes the generated LLM prompt 205A using LLM 150 to produce LLM output 206A. As described herein, the LLM output 206A may model the probability distribution of candidate word constructs and depends on the LLM prompt 205A.

[0058] The task grounding engine 132 generates a task grounding scale 208A, which is generated based on the LLM output 206A and the skill descriptions 207. Each of the multiple skill descriptions 207 describes a corresponding skill that the robot 110 is configured to perform. For example, "Go to table" may describe the skill "navigate to table" that the robot can perform by utilizing a trained navigation policy at the navigation target "table" (or a location corresponding to "table"). As another example, "Go to sink" may describe the skill "navigate to sink" that the robot can perform by utilizing a trained navigation policy at the navigation target "sink" (or a location corresponding to "sink"). As yet another example, "Pick up bottle" may describe the skill "grasp bottle" that the robot can perform by utilizing a gripping heuristic finely tuned for bottles and / or using a trained gripping network. As yet another example, “Pick up the key” could describe the “grasp the key” skill that a robot could perform using key-tuned grasping heuristics and / or a trained grasping network. Skill description 207 includes descriptors for skills A-G in Figure 2A, but skill description 207 may include descriptors for additional skills in various embodiments (as indicated by ellipses). Such additional skills may correspond to alternative objects and / or to different types of robot actions (e.g., “place,” “push,” “open,” “close”).

[0059] Each of the Task Grounding Scales 208A is generated based on the probability of the corresponding skill description in LLM Output 206A. For example, Task Grounding Scale A, "0.85", reflects the probability of the word sequence "go to table" in LLM Output 206A. As another example, Task Grounding Scale B, "0.20", reflects the probability of the word sequence "go to sink" in LLM Output 206A.

[0060] The world grounding engine 134 generates a world grounding scale 211A for robot skills. When generating a world grounding scale 211A for at least some robot skills, the world grounding engine 134 may generate the world grounding scale based on environmental state data 209A and, optionally, further on robot state data 210A and / or corresponding skill descriptions 207. Furthermore, when generating a world grounding scale 211A for at least some robot skills, the world grounding engine 134 may utilize one or more of the value function models 152.

[0061] In some embodiments, the world grounding engine 134 may generate fixed world grounding measures for several robot skills. For example, any “place” robot skill may always have a fixed measure such as 1.0 or 0.9, or a “finish” robot skill (i.e., indicating that the task is complete) may always have a fixed measure such as 0.1 or 0.2. In some embodiments, the world grounding engine 134 may additionally or alternatively generate world grounding measures for several robot skills based on corresponding non-machine learning-based (e.g., non-neural network and / or untrained) value function models 152. For example, a value function model may define that if environmental state data 209A and / or robot state data 210A indicate that an object is being held by the robot 110, then any “place” robot skill should have a fixed measure such as 1.0 or 0.9, and otherwise should have another fixed measure such as 0.0 or 0.1. As another example, the value function of the robot skill "navigate to [object / location]" could be defined as a function of the distance between the robot and the object / location, such that the world grounding measure is determined based on the environmental state data 209A and the robot state data 210. Or, the robot skill "complete" (i.e., indicating that the task is completed) could have a fixed measure, such as 0.1 or 0.2.

[0062] In some embodiments, the world grounding engine 134 may additionally or alternatively generate a world grounding scale for several robot skills based on corresponding values ​​from a number of trained value function models 152. In some of these embodiments, the trained value function models may be linguistic conditional models. For example, when generating a world grounding scale for a robot skill, the world grounding engine 134 may use a linguistic conditional model to process corresponding values ​​from a number of skill descriptions 207 for a robot skill, along with environmental state data 209A and optionally robot state data 210A, to generate a value that reflects the probability of the robot skill being successful, based on the current state data. The world grounding engine may generate a world grounding scale based on the generated value (e.g., to conform to the value). In some versions of these embodiments, candidate robot behaviors are also processed using a linguistic conditional model, along with corresponding values ​​from a number of skill descriptions 207, environmental state data 209A, and optionally robot state data 210A. In some of these versions, multiple values ​​are generated, each handling a different candidate robot action, based on the use of the same corresponding skill description 207, the same environmental state data 209A, and optionally the same robot state data 210A. In these versions, the world grounding engine 134 may generate a world grounding scale based on the generated value that reflects, for example, the highest probability of success. In some of these versions, different candidate robot actions may be selected using, for example, a cross-entropy method. For example, N robot actions may be initially sampled randomly, each for which a value may be generated, and then N additional robot actions are sampled from about one of the initial N robot actions based on that first robot action with the highest generated value.The trained value function model used by the world grounding engine 134 to generate the world grounding scale(s) for robot skills can also be used to actually perform robot skills with the robot 110.

[0063] The World Grounding Scale 211A is generated based on the state of robot 110, as reflected in the bird's-eye view in Figure 1B. That is, when robot 110 is still quite far from pear 184A and key 184B. Therefore, the World Grounding Scale F and G for “Pick up pear” and “Pick up key” are both relatively low ("0.10"). This reflects a low probability of success in grasping either of the attempted items due to the long distance between robot 110 and pear 184A and key 184B. The World Grounding Scale A of "0.80" for “Go to table” reflects a lower probability than the World Grounding Scale B of "0.85" for “Go to sink.” This may be based on robot 111 being closer to sink 193 than table 194.

[0064] When the selection engine 136 selects robot skill A ("Go to the table"), it considers both the world grounding scale 211A and the task grounding scale 208A and sends the instruction 213A for the selected robot skill A to the execution engine 136. In response, the execution engine 136 controls the robot 110 based on the selected robot skill A. For example, the execution engine 136 may control the robot using a navigation policy at the navigation target of "table" (or a location corresponding to "table").

[0065] In Figure 2A, the selection engine 136 generates an overall scale 212A by multiplying the world grounding scale 211A and the task grounding scale 208A, and selects robot skill A based on the fact that robot skill A has the highest overall scale 212A. Note that robot skill A has the highest overall scale despite not having the highest world grounding scale. Figure 2A shows that the overall scale 212A is generated by multiplying the world grounding scale 211A and the task grounding scale 208A, but other techniques may be used when generating the overall scale 212A. For example, different weights may be applied to the world grounding scale 211A and the task grounding scale 208A in the multiplication. For example, the world grounding scale 211A may be weighted 90%, and the task grounding scale 208A may be weighted 100%.

[0066] Referring now to Figure 2B, the process flow of how the various exemplary components in Figure 2A can interact when selecting the next robot skill to perform in the environment of Figure 1B in response to the FF NL command in Figure 1A, following (or during) the performance of robot skill A selected in Figure 2A.

[0067] In Figure 2B, the LLM engine 130 generates an LLM prompt 205B based on the FF NL input 105 ("Bring snacks from the table") and further on the selected skill descriptor 201B ("Go to the table") for robot skill A, based on the fact that robot skill A has been selected and is provided for implementation. The LLM engine 130 may generate the LLM prompt 205B to strictly conform to the FF NL input 105 and the selected skill descriptor 201B, or it may generate the LLM prompt 205B based on the FF NL input 105 and the selected skill descriptor 201B, but not to strictly conform to them. For example, as shown by LLM prompt 205B1, a non-restrictive example of LLM prompt 205B, the LLM prompt may be "How do I bring snacks from the table? 1. I'll do 1. Go to the table. 2." Such an LLM prompt 205B1 includes the prefix "how" and the suffix "2." Either or both of these may facilitate the prediction in the LLM output of steps(s) related to the achievement of a high-level task specified by the FF NL input 105. Furthermore, such an LLM prompt 205B1 may include a selected skill descriptor 201B of “Go to table” preceded by “1.”, which may facilitate the prediction in the LLM output of steps(s) related to occurring after the “Go to table” step.

[0068] In some embodiments, the LLM engine 130 may optionally generate an LLM prompt 205B based further on one or more of the following: a scene descriptor 202B of the robot 110's current environment (which may vary from Figure 2A), an example prompt 203B (which may vary from those in Figure 2A), and / or a description 204B (which may vary from those in Figure 2A).

[0069] The LLM engine 130 processes the generated LLM prompt 205B using LLM 150 to produce LLM output 206B. As described herein, the LLM output 206B may model the probability distribution of candidate word constructs and depends on LLM prompt 205B.

[0070] The task grounding engine 132 generates task grounding scales 208B, which are generated based on the LLM output 206A and skill descriptions 207. Each of the task grounding scales 208B is generated based on the probability of the corresponding skill description in the LLM output 206B. For example, task grounding scale A, "0.00", reflects the probability of the word sequence "go to table" in the LLM output 206B. As another example, task grounding scale B, "0.10", reflects the probability of the word sequence "go to sink" in the LLM output 206B.

[0071] The world grounding engine 134 generates a world grounding scale 211B for robot skills. When generating the world grounding scale 211B for at least some robot skills, the world grounding engine 134 may generate the world grounding scale based on environmental state data 209B, and optionally further based on robot state data 210B and / or corresponding skill descriptions 207. Note that the environmental state data 209B and robot state data 210B change from their correspondings in Figure 2A due to robot skill A being at least partially implemented at the time of Figure 2B. Furthermore, when generating the world grounding scale 211B for at least some robot skills, the world grounding engine 134 may utilize one or more value function models 152.

[0072] In some embodiments, the world grounding engine 134 may generate a fixed world grounding scale for several robot skills. In some embodiments, the world grounding engine 134 may additionally or alternatively generate a world grounding scale for several robot skills based on a corresponding value function model 152 that is not machine learning-based (e.g., not a neural network and / or untrained). In some embodiments, the world grounding engine 134 may additionally or alternatively generate a world grounding scale for several robot skills based on a corresponding value function model 152 that is a trained value function model. In some of those embodiments, the trained value function model may be a linguistic conditional model.

[0073] The World Grounding Scale 211B is generated based on the state of robot 110 after robot skill A has been performed at least partially (i.e., robot 110 is closer to Table 194 than reflected in the bird's-eye view of Figure 1B). Thus, robot 110 is closer to the pear 184A and key 184B at the time of Figure 2A (e.g., both are within reach). Therefore, the World Grounding Scale F and G for “picking up the pear” and “picking up the key” are both relatively high ("0.90").

[0074] When the selection engine 136 selects robot skill F ("pick up the pear"), it considers both the World Grounding Scale 211B and the Task Grounding Scale 208B and sends the instruction 213B for the selected robot skill F to the execution engine 136. In response, the execution engine 136 controls the robot 110 based on the selected robot skill F. For example, the execution engine 136 may control the robot using a grasping strategy with optionally fine-tuned parameters for grasping the pear.

[0075] In Figure 2B, the selection engine 136 generates an overall scale 212B by multiplying the world grounding scale 211B and the task grounding scale 208B, and selects robot skill F based on the fact that robot skill F has the highest overall scale 212B. Note that robot skill A has the highest overall scale even though it does not have the highest world grounding scale or the highest task grounding scale. Figure 2B shows that the overall scale 212B is generated by multiplying the world grounding scale 211B and the task grounding scale 208B, but other techniques may be used when generating the overall scale 212B. For example, different weights may be applied to the world grounding scale 211B and the task grounding scale 208B in the multiplication. For example, the world grounding scale 211B may be weighted 90%, and the task grounding scale 208B may be weighted 100%.

[0076] Figure 3 is a flowchart illustrating exemplary method 300 of utilizing LLM in the implementation of robot skills(s) that perform tasks reflected in FF NL instructions, according to embodiments disclosed herein. For convenience, the operation of method 300 is described with respect to the system that performs the operation. This system may include one or more components of robots, such as robot processors and / or robot control systems of robots 110, robot 520, and / or other robots, and / or one or more components of a computer system, such as computer system 610. Furthermore, the operation of method 300 is shown in a particular order, but this is not intended to limit it. One or more operations may be reordered, omitted, or added.

[0077] In block 352, the system identifies the FF NL instruction.

[0078] In block 354, the system uses LLM to process LLM prompts based on FF NL instructions and generate LLM output. Block 354 optionally includes subblocks 354A and / or 354B. In subblock 354A, the system includes the scene descriptor(s) in the LLM prompt. In subblock 354B, the system generates a description and includes it in the LLM prompt.

[0079] In block 356, the system generates a corresponding task grounding scale based on the LLM output of block 354 and the corresponding NL skill description for each of the multiple candidate robot skills. Block 356 optionally includes subblock 356A, in which the system generates a task grounding scale for the candidate robot skills based on the probabilities of the NL skill descriptions reflected in the probability distribution of the LLM output.

[0080] In block 358, the system generates a corresponding world grounding scale for each of several candidate robot skills based on the current state data. Although shown below block 356 in Figure 3, it should be noted that in various embodiments, block 358 may occur in parallel with or even before block 356.

[0081] Block 358 optionally includes subblock 358A and / or subblock 358B. Optionally, subblock 358A and / or subblock 358B are executed only for certain robot skills among the candidate robot skills.

[0082] In subblock 358A, the system processes current environmental state data and / or current robot state data using a value function model when generating a global grounding scale for candidate robot skills.

[0083] In subblock 358B, the system uses a multi-skill and language-conditioned value function model to generate a world grounding scale for candidate robot skills, while also processing the NL descriptions (e.g., their embeddings) of the robot skills.

[0084] In block 360, the system selects a robotic skill to perform or a termination condition (which may be considered a specific robotic skill) based on the task grounding scale in block 356 and the world grounding scale in block 358. For example, a termination condition may be selected if the task grounding scale for "finished," "completed," or other termination descriptions corresponding to the termination condition meets a threshold, and / or if the task grounding scale, world grounding scale, and / or overall scale for all robotic skills do not meet the same or different thresholds.

[0085] In block 362, the system determines whether a robot skill was selected for execution in block 360. If not, the system proceeds to block 364 and terminates controlling the robot based on the FF NL instruction in block 352. If selected, the system proceeds to blocks 366 and 368.

[0086] In block 366, the system performs the selected robot skill. In block 368, the system modifies the latest LLM prompt based on the skill description of the performed skill. The system then returns to block 354 and processes the LLM prompt as modified in block 368. The system also performs blocks 356, 358, 360, and 362, and optionally another iteration of blocks 366 and 368 (depending on the decision in block 362). This general process may continue until a termination condition is selected in the iteration of block 360.

[0087] Figure 4 is a flowchart illustrating an exemplary method for generating a global grounding scale for robot skills based on processing current state data and skill descriptions of robot skills. For convenience, the operations of Method 400 are described with respect to a system that performs the operations. This system may include one or more components of robots, such as robot processors and / or robot control systems of robots 110, robot 520, and / or other robots, and / or one or more components of a computer system, such as computer system 610. Furthermore, the operations of Method 400 are shown in a particular order, but this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0088] In block 452, the system generates an image embedding by processing the current image using an image tower of a trained linguistic conditional value function model. The image may be the current image captured by the robot's visual component.

[0089] In block 454, the system selects a candidate action. For example, the system may select a candidate action by sampling from the action space.

[0090] In block 456, the system generates additional embeddings by processing candidate actions selected in block 454, skill descriptions of robot skills (e.g., their embeddings), and optionally robot state data, using additional towers of a trained linguistic conditional value function model.

[0091] In block 458, the system generates values ​​based on processing the concatenation of image embeddings and additional embeddings using an additional layer of the trained linguistic conditional value function model.

[0092] In block 460, the system decides whether to generate additional values ​​for an additional candidate action. If not, the system proceeds to block 464. If it does generate additional values, the system proceeds to block 462 and selects an additional candidate action. In some iterations of block 462, this may involve sampling from the action space without considering the values ​​generated so far. In some later iterations of block 462, this may involve sampling from the action space based on the values ​​generated so far, using the cross-entropy method. For example, sampling may be around the portion of the action space corresponding to the candidate action with the highest value generated so far, or may be biased towards that portion in other ways. The system then returns to block 456 with the newly selected candidate action, but with the same skill description and, optionally, the same robot state.

[0093] In block 464, the system generates a world grounding scale for robot skills corresponding to the skill description (used in the iteration(s) of block 456) based on the maximum value of the values ​​generated in the iteration(s) of block 458.

[0094] Figure 5 schematically shows an exemplary architecture of robot 520. Robot 520 includes a robot control system 560, one or more motion components 540a-540n, and one or more sensors 542a-542m. Sensors 542a-542m may include, for example, a vision sensor, a light sensor, a pressure sensor, a pressure wave sensor (e.g., a microphone), a proximity sensor, an accelerometer, a gyroscope, a thermometer, a barometer, and the like. Although sensors 542a-542m are shown as being integrated with robot 520, this is not meant to be limiting. In some embodiments, sensors 542a-542m may be located outside of robot 520, for example, as standalone units.

[0095] The motion components 540a to 540n may include, for example, one or more end effectors and / or one or more servo motors or other actuators to realize the movement of one or more components of the robot. For example, the robot 520 may have multiple degrees of freedom, and each actuator may, in response to a control command, control the operation of the robot 520 within the range of one or more degrees of freedom. As used herein, the term actuator includes any driver(s) that may be associated with the actuator and convert received control commands into one or more signals for driving the actuator, as well as mechanical or electrical devices that produce motion (e.g., motors). Thus, providing a control command to an actuator may include providing a control command to a driver that converts the control command into appropriate signals for driving an electrical or mechanical device to produce the desired motion.

[0096] The robot control system 560 may be implemented in one or more processors of the robot 520, such as the CPU, GPU, and / or other controllers. In some embodiments, the robot 520 may include a “brain box” which may include all or aspects of the control system 560. For example, the brain box may bring real-time bursts of data to the operating components 540a-n, each of which includes one or more sets of control commands that, in particular, instruct motion parameters (if any) for each of one or more of the operating components 540a-n. In some embodiments, the robot control system 560 may perform one or more aspects of the methods described herein, such as method 300 and / or method 400 in Figure 3.

[0097] As described herein, in some embodiments, all or aspects of the control commands generated by the control system 560 when controlling the robot during the execution of a robot task may be generated based on robot skills(s) determined to be relevant to the robot task based on the World Grounding Scale and Task Grounding Scale described herein. In some embodiments, the control system 560 is shown as an integral part of the robot 520 in Figure 5, but all or aspects of the control system 560 may be implemented on a component that is separate from the robot 520 but communicates with the robot 520. For example, all or aspects of the control system 560 may be implemented on one or more computing devices that communicate with the robot 520 by wire and / or wirelessly, such as a computing device 610.

[0098] Figure 6 is a block diagram of an exemplary computing device 610 that may be optionally used to perform one or more embodiments of the technology described herein. The computing device 610 typically includes at least one processor 614 that communicates with several peripheral devices via a bus subsystem 612. These peripheral devices may include, for example, a storage subsystem 624 including a memory subsystem 625 and a file storage subsystem 626, a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices enable user interaction with the computing device 610. The network interface subsystem 616 provides an interface to an external network and is coupled to a corresponding interface device in another computing device.

[0099] The user interface input device 622 may include pointing devices such as keyboards, mice, trackballs, touchpads, or graphics tablets, audio input devices such as scanners, touchscreens integrated into displays, speech recognition systems, microphones, and / or other types of input devices. Generally, the use of the term “input device” is intended to include all possible types of devices and methods for inputting information into the computing device 610 or a communication network.

[0100] The user interface output device 620 may include a non-visual display such as a display subsystem, printer, fax machine, or audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT) or liquid crystal display (LCD), a projection device, or any other mechanism for producing a visible image. The display subsystem may also provide a non-visual display, such as through an audio output device. In general, the use of the term “output device” is intended to include all possible types of devices and methods for outputting information from the computing device 610 to a user or another machine or computing device.

[0101] The storage subsystem 624 stores programming and data structures that provide some or all of the functionality of the modules described herein. For example, the storage subsystem 624 may include logic for performing selected embodiments of method 300 in Figure 3 and / or method 400 in Figure 4.

[0102] These software modules are typically executed on processor 614 alone or in combination with other processors. The memory 625 used by the storage subsystem 624 may include several memories, including main random access memory (RAM) 630 for storing instructions and data during program execution, and read-only memory (ROM) 632 for storing fixed instructions. The file storage subsystem 626 can provide persistent storage of program files and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of a particular embodiment may be stored by the file storage subsystem 626 within the storage subsystem 624, or on another machine accessible by processor 614.

[0103] The bus subsystem 612 provides a mechanism for various components and subsystems of the computing device 610 to communicate with each other as intended. Although the bus subsystem 612 is schematically shown as a single bus, alternative embodiments of the bus subsystem may use multiple buses.

[0104] The computing device 610 can be of various types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing systems or computing devices. Because computers and networks are constantly changing, the description of the computing device 610 shown in Figure 6 is intended only as a specific example to illustrate several embodiments. Many other configurations of the computing device 610 may have more or fewer components than the computing device shown in Figure 6.

[0105] In some embodiments, methods are provided that are implemented by one or more processors, including identifying instructions and processing the instructions using a language model (LM) (e.g., a large language model (LLM)) to generate LM outputs. Instructions may be free-form natural language instructions generated based on user interface inputs provided by a user via one or more user interface input devices. The generated LM outputs may model a probability distribution of candidate word constructs that depend on the instructions. The method further includes identifying robot skills that can be performed by the robot and skill descriptions, which are natural language descriptions of the robot skills. The method further includes generating a task grounding scale for the robot skills based on the LM outputs and skill descriptions. The task grounding scale may reflect the probability of the skill descriptions in the probability distribution of the LM outputs. The method further includes generating a world grounding scale for the robot skills based on the robot skills and current environmental state data. The world grounding scale may reflect the probability that the robot skills are successful, based on the current environmental state data. The current environmental state data may include sensor data captured by one or more sensor components of the robot in the robot's current environment. The method further includes determining, based on both the Task Grounding Scale and the World Grounding Scale, to perform a robotic skill instead of several additional robotic skills that the robot can perform. In response to the decision to perform a robotic skill, the method further includes having the robot perform the robotic skill in the current environment.

[0106] These and other embodiments of the technology disclosed herein may include one or more of the following features:

[0107] In some embodiments, the method further includes identifying one additional robot skill from among several additional robot skills, and an additional skill description which is a natural language description of the one additional robot skill, and generating an additional task grounding scale for the one additional robot skill based on the LM output and the additional skill description. The additional task grounding scale may reflect the additional probability of the additional skill description in a probability distribution. In those embodiments, the method further includes generating an additional world grounding scale for the one additional robot skill based on the one additional robot skill and current environmental state data. The additional world grounding scale may reflect the additional probability that the one additional robot skill is successful, based on the current environmental state data. Furthermore, in those embodiments, deciding to perform a robot skill on behalf of several additional robot skills, each of which is performable by the robot, based on both the task grounding scale and the world grounding scale, may include deciding to perform a robot skill based on the task grounding scale, the world grounding scale, the additional task grounding scale, and the additional world grounding scale. For example, the method may include generating an overall scale for robot skills based on both a task grounding scale and a world grounding scale, generating an additional overall scale for additional robot skills based on both an additional task grounding scale and an additional world grounding scale, and deciding to implement robot skills instead of additional robot skills based on a comparison between the overall scale and the additional overall scale.

[0108] In some embodiments, the method further includes: using LM to process the instruction and skill description of the robot skill in response to a decision to perform a robot skill to generate a further LM output that models a further probability distribution of candidate word constructs dependent on the instruction and skill description; identifying one additional robot skill among a plurality of additional robot skills and an additional skill description which is a natural language description of the one additional robot skill; generating an additional task grounding metric for the one additional robot skill based on the further LM output and the additional skill description; generating an additional world grounding metric for the one additional robot skill based on the one additional robot skill and updated current environment state data; deciding to perform one additional robot skill in place of the robot skill or any other additional robot skill among a plurality of additional robot skills each of which can be performed by the robot, based on both the additional task grounding metric and the additional world grounding metric; and having the robot perform one additional robot skill in the current environment in response to the decision to perform one additional robot skill. The updated current environment state data may include updated sensor data captured by one or more sensor components after the robot skill has been performed in the current environment.

[0109] In some embodiments, the method further includes: using LM to process the skill description of the instruction and the robot skill in response to a decision to perform a robot skill to generate a further LM output that models a further probability distribution of candidate word constructs dependent on the instruction and the skill description; identifying a termination skill among a plurality of additional robot skills and a termination description which is a natural language description that the execution of the instruction is complete; generating a termination task grounding scale for the termination skill based on the further LM output and the termination description; generating a termination world grounding scale for the termination skill based on the termination skill and updated current environment state data; deciding to perform the termination skill in place of the robot skill, or other additional robot skills among a plurality of additional robot skills, each of which can be performed by the robot, based on both the additional task grounding scale and the additional world grounding scale; and in response to the decision to perform the termination skill, causing the robot to stop further execution of any control commands that advance the instruction. The updated current environment state data may include updated sensor data captured by one or more sensor components after the execution of the robot skill in the current environment.

[0110] In some embodiments, generating a world grounding scale based on robot skills and current environmental state data involves processing the robot skills and current environmental state data using a trained value function to generate a value function output that includes a world grounding scale. In some versions of those embodiments, the current environmental state data includes visual data (e.g., multi-channel images) from sensor data, and the visual data is captured by one or more visual components of one or more sensor components of the robot. In some additional or alternative versions of those embodiments, the trained value function is a linguistic conditional value function, and processing robot skills using the trained value function involves processing a skill description of the robot skills. In some additional or alternative versions of those embodiments, the trained value function is trained to correspond to an affordance function, and the value function output specifies whether the robot skill is possible based on the current environmental state data. In some additional or alternative versions of those embodiments, the value function is a machine learning model trained using reinforcement learning.

[0111] In some embodiments, deciding to perform a robot skill instead of multiple additional robot skills, based on both a task grounding scale and a world grounding scale, includes generating an overall scale as a function of the task grounding scale and the world grounding scale, comparing the overall scale to the corresponding multiple additional scales, each for the corresponding of the multiple additional robot skills, and deciding to perform the robot skill based on the comparison. In some versions of those embodiments, the overall scale is a weighted or unweighted combination of the task grounding scale and the world grounding scale. In some additional or alternative versions of those embodiments, the task grounding scale is the task grounding probability, the world grounding scale is the world grounding probability, and generating the overall scale includes generating a product based on multiplying the task grounding probability and the world grounding probability, and using the product as the overall scale.

[0112] In some embodiments, having a robot perform a robot skill in its current environment involves having it execute a linguistically conditioned robot control policy, which is conditioned by a skill description of the robot skill. In some versions of these embodiments, the linguistically conditioned robot control policy includes a machine learning model. In some versions of these embodiments, the linguistically conditioned robot control policy is trained using reinforcement learning and / or imitation learning.

[0113] In some embodiments, the instructions strictly conform to natural language input provided by the user via user interface input.

[0114] In some embodiments, the instructions do not strictly conform to the natural language input provided by the user via the user interface input. In some of those embodiments, the method further includes determining that the natural language input is not in the form of a question, and generating instructions by modifying the natural language input to be in the form of a question in response to determining that the natural language input is not in the form of a question.

[0115] In some embodiments, the LM is a Large-Scale Language Model (LLM).

[0116] In some embodiments, a method is provided that is implemented by one or more processors, which includes identifying instructions, which are free-form natural language instructions generated based on user interface inputs provided by a user via one or more user interface input devices. The method further includes processing the instructions using a language model (LM) to generate LM outputs, such as LM outputs that model a probability distribution of candidate word constructs dependent on the instructions. The method further includes, for each of a plurality of robot skills, each of which is performable by the robot, generating a task grounding scale for the robot skill (e.g., reflecting the probability of the skill description in a probability distribution) based on the LM output and the skill description of the robot skill; generating a world grounding scale for the robot skill (e.g., reflecting the probability that the robot skill is successful, based on the current environment state data) based on the robot skill and current environment state data; and generating an overall scale for the robot skill based on the task grounding scale and the world grounding scale. The current environment state data includes sensor data captured by one or more sensor components of the robot in the robot's current environment. The method further includes selecting a given skill from among the robot skills based on the overall scale for the robot skills. The method further includes, in response to selecting a given robot skill, causing the robot to perform a given robot skill in the current environment.

[0117] In some embodiments, a method is provided, implemented by a processor(s), that includes identifying instructions, which are free-form natural language instructions, such as those generated based on user interface inputs provided by a user via one or more user interface input devices. The method further includes processing the language model (LM) prompts generated based on the instructions using an LM (e.g., a Large Language Model (LLM)) to produce an LM output that models a probability distribution of candidate word constructs dependent on the LM prompt. The method further includes identifying robot skills that can be performed by the robot and skill descriptions, which are natural language descriptions of the robot skills. The method further includes generating a task grounding measure for the robot skills, based on the LM output and skill descriptions, that reflects the probability of the skill descriptions in the probability distribution of the LM output. The method further includes generating a world grounding measure for the robot skills, based on the robot skills and current environmental state data, that reflects the probability of the robot skills being successful, based on the current environmental state data. The current environmental state data optionally includes sensor data captured by one or more sensor components of the robot in the robot's current environment. The method further includes determining, based on both the Task Grounding Scale and the World Grounding Scale, to perform a robotic skill instead of several additional robotic skills that the robot can perform. In response to the decision to perform a robotic skill, the method further includes having the robot perform the robotic skill in the current environment.

[0118] In some embodiments, a method is provided, implemented by a processor(s), which includes identifying instructions that are free-form natural language instructions, such as those generated based on user interface input provided by a user via one or more user interface input devices. The method further includes generating a Large Language Model (LLM) prompt based on the instructions. The method further includes processing the LLM prompt using the LLM to generate an LLM output that models a probability distribution of candidate word constructs dependent on the LLM prompt. The method further includes, for each of several robot skills, each performable by the robot, generating a task grounding scale for the robot skill, based on the LLM output and the skill description of the robot skill, which reflects the probability of the skill description in a probability distribution; generating a world grounding scale for the robot skill, based on the robot skill and current environmental state data, which reflects the probability of the robot skill being successful based on current environmental state data; and generating an overall scale for the robot skill, based on the task grounding scale and the world grounding scale. The method further includes selecting a given skill from among the robot skills based on the overall scale for the robot skills. The method further includes, in response to selecting a given robot skill, causing the robot to perform a given robot skill in the current environment.

[0119] Other embodiments may include a non-temporary computer-readable storage medium that stores instructions that can be executed by one or more processors (e.g., a central processing unit (CPU(or more)), a graphics processing unit (or more) (GPU(or more)), and / or tensor processing units (or more) (TPU(or more))) in order to perform one or more of the methods described herein. Further embodiments may include a system of one or more computers and / or one or more robots, each including one or more processors capable of operating to execute the stored instructions in order to perform one or more of the methods described herein.

Claims

1. A method carried out by one or more processors, Identifying an instruction, wherein the instruction is a free-form natural language instruction generated based on user interface input provided by a user via one or more user interface input devices, The process involves using a language model (LM) to process LM prompts generated based on the instructions and producing an LM output that models the probability distribution of candidate word constructs, where the LM is a large-scale language model (LLM). Identifying robot skills that can be performed by a robot and skill descriptions that are natural language descriptions of said robot skills, Based on the LM output and the skill description, a task grounding scale for the robot skill is generated that reflects the probability of the skill description in the probability distribution of the LM output. Based on the robot skills and current environmental state data, a global grounding scale for the robot skills is generated that reflects the probability of success based on the current environmental state data, The current environmental state data includes sensor data captured by one or more sensor components of the robot in the robot's current environment. To generate, Based on both the Task Grounding Scale and the World Grounding Scale, it is determined that the robot will perform the robot skill instead of any additional robot skills that the robot can perform, In response to deciding to perform the aforementioned robot skills, The robot is to perform the robot skills in the current environment, Methods that include...

2. Identifying one additional robot skill from among the aforementioned multiple additional robot skills, and an additional skill description which is a natural language description of the aforementioned one additional robot skill, Based on the LM output and the additional skill description, an additional task grounding scale is generated for the one additional robot skill, which reflects the additional probability of the additional skill description in the probability distribution. Based on the aforementioned additional robot skill and the current environmental state data, an additional world grounding measure is generated for the additional robot skill, which reflects the additional probability that the aforementioned additional robot skill is successful, based on the current environmental state data. It further includes, Based on both the Task Grounding Scale and the World Grounding Scale, it is determined that the robot will perform the robot skill instead of any additional robot skills that the robot can perform. Based on both the Task Grounding Scale and the World Grounding Scale, an overall scale for the robot skills is generated. Based on both the aforementioned additional task grounding scale and the aforementioned additional world grounding scale, an additional overall scale is generated for the one additional robot skill. Based on a comparison of the overall scale and the additional overall scale, it is decided to perform the robot skill instead of the one additional robot skill, The method according to claim 1, including the method described in claim 1.

3. In response to deciding to perform the aforementioned robot skills, Using the LM, process further LM prompts based on the instruction and the skill description of the robot skill to generate further LM outputs that model the further probability distribution of the candidate word constructs, depending on the further LM prompts. Identifying one additional robot skill from among the aforementioned multiple additional robot skills, and an additional skill description which is a natural language description of the aforementioned one additional robot skill, Based on the further LM output and the additional skill description, an additional task grounding measure is generated for the one additional robot skill, which reflects the probability of the additional skill description in the further probability distribution. To generate an additional world grounding measure for the additional robot skill, which reflects the probability that the additional robot skill is successful, based on the updated current environmental state data, based on the additional robot skill and updated current environmental state data, The updated current environmental state data includes updated sensor data captured by one or more sensor components after the robot skill is performed in the current environment. The above-mentioned generation, Based on both the aforementioned additional task grounding scale and the aforementioned additional world grounding scale, it is decided to perform the robot skill, or one of the additional robot skills among the plurality of additional robot skills that each of the robots can perform, In response to the decision to perform the aforementioned additional robot skill, The robot is to perform the one additional robot skill in the current environment, The method according to claim 1, including the method described in claim 1.

4. In response to deciding to perform the aforementioned robot skills, Using the LM, process further LM prompts based on the instruction and the skill description of the robot skill to generate further LM outputs that model the further probability distribution of the candidate word constructs, depending on the further LM prompts. Identifying the termination skill among the multiple additional robot skills, and the termination description which is a natural language description of the completion of the command execution, Based on the aforementioned further LM output and the termination description, a termination task grounding scale for the termination skill is generated that reflects the probability of the termination skill description in the aforementioned further probability distribution, Based on the aforementioned completion task grounding scale, it is decided to perform the completion skill instead of the robot skill, or any of the additional robot skills among the plurality of additional robot skills that can be performed by the robot. In response to the decision to perform the aforementioned completion skills, To cause the robot to cease further execution of any control commands that promote the aforementioned instruction, The method according to claim 1, including the method described in claim 1.

5. Based on the robot skills and the current environmental state data, the world grounding scale can be generated. The method according to claim 1, comprising processing the robot skills and the current environmental state data using a trained value function to generate a value function output that includes the world grounding measure.

6. The method according to claim 5, wherein the current environmental state data includes visual data of the sensor data, and the visual data is captured by one or more visual components of the one or more sensor components of the robot.

7. The method according to claim 6, wherein the visual data includes a multi-channel image.

8. The aforementioned trained value function is a linguistic conditional value function, The method according to claim 5, wherein processing the robot skill using the trained value function includes processing the skill description of the robot skill.

9. The method according to claim 5, wherein the trained value function is trained to correspond to an affordance function, and the value function output specifies whether the robot skill is possible based on the current environmental state data.

10. The method according to claim 5, wherein the value function is a machine learning model trained using reinforcement learning.

11. Based on both the Task Grounding Scale and the World Grounding Scale, it is decided to perform the robot skills instead of the multiple additional robot skills. The overall scale is generated as a function of the aforementioned task grounding scale and the aforementioned world grounding scale, The overall scale is compared with the corresponding additional scales for each of the additional robot skills, Based on the above comparison, it is decided to implement the robot skills, The method according to claim 1, including the method described in claim 1.

12. The method according to claim 11, wherein the overall scale is a weighted or unweighted combination of the task grounding scale and the world grounding scale.

13. The method according to claim 12, wherein the task grounding scale is the task grounding probability, the world grounding scale is the world grounding probability, and generating the overall scale involves generating a product based on multiplying the task grounding probability and the world grounding probability, and using the product as the overall scale.

14. Having the robot perform the robot skills in the current environment is The method according to claim 1, comprising causing the robot to execute a linguistically conditioned robot control policy conditioned by the skill description of the robot skill.

15. The method according to claim 14, wherein the language-conditional robot control strategy includes a machine learning model.

16. The method according to claim 15, wherein the language-conditional robot control policy is trained using reinforcement learning and / or imitation learning.

17. The method according to claim 1, wherein the LM prompt strictly conforms to the natural language input provided by the user via the user interface input.

18. A method carried out by one or more processors, Identifying an instruction, wherein the instruction is a free-form natural language instruction generated based on user interface input provided by a user via one or more user interface input devices, Using a language model (LM), the system processes LM prompts generated based on the instructions to produce an LM output that models the probability distribution of candidate word constructs, which depends on the LM prompts. Identifying robot skills that can be performed by a robot and skill descriptions that are natural language descriptions of said robot skills, Based on the LM output and the skill description, a task grounding scale for the robot skill is generated that reflects the probability of the skill description in the probability distribution of the LM output. Based on the robot skills and current environmental state data, a global grounding scale for the robot skills is generated that reflects the probability of success based on the current environmental state data, The current environmental state data includes sensor data captured by one or more sensor components of the robot in the robot's current environment. To generate, Based on both the Task Grounding Scale and the World Grounding Scale, it is determined that the robot will perform the robot skill instead of any additional robot skills that the robot can perform, In response to deciding to perform the aforementioned robot skills, The robot is to perform the robot skills in the current environment, Includes, The LM prompt does not strictly conform to the natural language input provided by the user via the user interface input. The natural language input is determined not to be in question format, In response to determining that the natural language input is not in question format, the LM prompt is generated by modifying the natural language input to be in question format. Methods that further include this.

19. A method carried out by one or more processors, Identifying an instruction, wherein the instruction is a free-form natural language instruction generated based on user interface input provided by a user via one or more user interface input devices, Using a language model (LM), the system processes LM prompts generated based on the instructions to produce an LM output that models the probability distribution of candidate word constructs, which depends on the LM prompts. Identifying robot skills that can be performed by a robot and skill descriptions that are natural language descriptions of said robot skills, Based on the LM output and the skill description, a task grounding scale for the robot skill is generated that reflects the probability of the skill description in the probability distribution of the LM output. Based on the robot skills and current environmental state data, a global grounding scale for the robot skills is generated that reflects the probability of success based on the current environmental state data, The current environmental state data includes sensor data captured by one or more sensor components of the robot in the robot's current environment. To generate, Based on both the Task Grounding Scale and the World Grounding Scale, it is determined that the robot will perform the robot skill instead of any additional robot skills that the robot can perform, In response to deciding to perform the aforementioned robot skills, The robot is to perform the robot skills in the current environment, Includes, The LM prompt does not strictly conform to the natural language input provided by the user via the user interface input. A method further comprising generating the LM prompt to include as a suffix content requesting the generation of the next step.

20. The LM prompt does not strictly conform to the natural language input provided by the user via the user interface input. The method according to claim 1, further comprising generating the LM prompt to include one or more scene descriptors describing the current environment of the robot.

21. A method carried out by one or more processors, Identifying an instruction, wherein the instruction is a free-form natural language instruction generated based on user interface input provided by a user via one or more user interface input devices, To generate a Large-Scale Language Model (LLM) prompt based on the aforementioned instruction, The process involves using LLM to process the LLM prompt and generating an LLM output that models the probability distribution of candidate word constructs, depending on the LLM prompt. For each of the multiple robot skills that can be performed by the robot, Based on the LLM output and the skill description of the robot skill, a task grounding scale is generated for the robot skill that reflects the probability of the skill description in the probability distribution. Based on the robot skills and current environmental state data, a global grounding scale for the robot skills is generated that reflects the probability of success based on the current environmental state data, The current environmental state data includes sensor data captured by one or more sensor components of the robot in the robot's current environment. The above-mentioned generation, Based on both the Task Grounding Scale and the World Grounding Scale, an overall scale for the robot skills is generated. Based on the overall scale for the aforementioned robot skills, a given skill is selected from among the aforementioned robot skills. In response to selecting the aforementioned given robot skills, The robot is made to perform the given robot skills in the current environment, Methods that include...

22. It is a robot, One or more actuators, End effectors and, Memory for storing instructions, One or more processors operable to execute the instructions for performing the method according to any one of claims 1 to 21, A robot equipped with [the following features].

23. Memory for storing instructions, One or more processors operable to execute the instructions for performing the method according to any one of claims 1 to 21, A system that includes these features.

24. A computer program for causing a computer to perform the method described in any one of claims 1 to 21.

Images