Robot control based on natural language commands and descriptors of objects in the robot environment

Abstract

Some embodiments relate to generating regions of interest, as well as estimated map locations and region embedding(s) for each region of interest, based on processing captured visual data instances throughout an environment. Some embodiments additionally or alternatively relate to determining object descriptors that describe objects relevant to performing the task and likely to be present in the environment based on (1) free-form (FF) natural language (NL) instructions for a robot to perform a task and (2) region embedding(s) generated for identified regions of interest in the environment. Some embodiments additionally or alternatively relate to utilizing a subset of the object descriptor(s) determined to describe object(s) relevant to performing the task of the FF NL instructions and likely to be included in the environment when determining robot skill(s) to be performed by the robot(s) to perform the task specified in the FF NL instructions.

Description

[Background technology]

[0001] Many robots are programmed to perform certain tasks, for example, robots on an assembly line may be programmed to recognize certain objects and perform certain operations on those objects.

[0002] Furthermore, some robots may be able to perform a particular task in response to an explicit user interface input corresponding to the task. For example, a vacuum cleaning robot may be able to perform a general cleaning task in response to the utterance, "Robot, clean." However, in many cases, the user interface input that causes the robot to perform a task must be explicitly mapped to the task. Thus, the robot may not be able to perform a particular task in response to various free-form natural language inputs from a user attempting to control the robot. Summary of the Invention

[0003] Separately, efforts are being made to enable robust free-form (FF) natural language (NL) control of robots, such as enabling a robot to respond appropriately to any one of a variety of different types of verbal commands from a human instructing the robot. For example, in response to the FF NL command "Put the purple unicorn stuffed animal in the toy box," a robot task can be performed that includes (a) moving to the "purple unicorn stuffed animal," (b) picking up the "purple unicorn stuffed animal" (toy), (c) moving to the "toy box," and (d) placing the "purple unicorn stuffed animal" in the "toy box." However, for various FF NL commands, various techniques may fail in determining and / or executing the robot task steps, and / or may result in inefficiencies of the robot in performing the task step(s) and / or inefficiencies of the robot and / or other computations in determining the task step(s).

[0004] For example, some technologies are only capable of performing task(s) involving object(s) currently visible to the robot (e.g., within the line of sight(s) of the robot's vision component(s)). For example, for "put the purple unicorn stuffed animal in the toy box," these technologies cannot perform the task if the "purple unicorn stuffed animal" and / or the "toy box" are not currently visible to the robot. As another example, some technologies may store representations of previously detected objects that are not currently visible to the robot. However, such representations are often stored as one of multiple constrained, predefined categories, and, for example, a "purple unicorn stuffed animal" cannot be identified if a corresponding object representation is stored in association only with an entirely different representation. As yet another example, some technologies may enable exploration in locating the relevant object(s) in response to an FF NL instruction. However, such reactive exploration delays the execution of the corresponding task. Furthermore, in situations where the relevant object(s) are not present (or cannot be located) in the environment, such searches may be performed unnecessarily, resulting in a waste of robot resources.

[0005] In light of these and / or other considerations, some embodiments disclosed herein are directed to the generation and utilization of open vocabulary and queryable scene representations to facilitate language model-based robotic task planning and planned task execution. The scene representation is queryable using an open vocabulary, thereby eliminating the need to restrict the objects involved in the robotic task planning to a closed set. Some of these embodiments parse free-form (FF) natural language (NL) instructions provided by a human to determine object descriptor(s) associated with the FF NL instructions. Such object descriptor(s) may include descriptor(s) explicitly included in the FF NL instructions and / or descriptor(s) inferred from the FF NL instructions but not explicitly included in the FF NL instructions. The object descriptor(s) (e.g., their text embeddings) may then be used to query an environment map (e.g., represented as region embeddings of regions of interest and their associated estimated map locations) to determine which object descriptor(s) correspond to object(s) present in the environment and to determine the location(s) of these corresponding object(s). The object descriptor(s) determined to correspond to object(s) present in the environment and the location(s) of at least some of these corresponding object(s) may then be used to plan a robot task and perform the planned task. For example, a large-scale language model (LLM) may be utilized for planning, where the determined object descriptor(s) are processed using the LLM along with FF NL instructions to determine the robot skill(s) to perform to accomplish the task specified by the FF NL instructions. Utilizing the determined object descriptor(s) improves the robustness and / or accuracy of the robot skill(s) determination. Furthermore, querying and utilizing the environment map allows object(s) that are not visible to the robot at the time of the FF NL command and / or cannot be specified in the FF NL command to be considered and utilized during planning.Furthermore, querying and utilizing the environment map allows FF NL commands to perform robotic tasks robustly and accurately without having to adhere to the syntax of a specific object classification.

[0006] Some implementations disclosed herein relate to generating regions of interest, as well as estimated map locations and region embedding(s) for each region of interest, based on processing captured visual data instances throughout an environment. These implementations further associate and store at least the estimated location and region embedding(s) for each region of interest for later use. For example, each captured visual data instance may be processed using a class-independent object detection model to detect region(s) of interest, if any, within the visual data instance. Further, for each detected region, (1) an estimated map location of the detected region may be determined, and (2) region embedding(s) for each may be generated based on processing visual data from the visual data instances within the region of interest. For example, pixels within the region of interest may be processed using a visual language model (VLM) encoder to generate a region embedding. Optionally, multiple region embeddings are generated for each region of interest, each generated using a different VLM encoder. This enables the use of multiple region embeddings with other techniques disclosed herein. This is beneficial because some type(s) of embeddings are robust to certain type(s) of objects (e.g., out of distribution objects), while other type(s) of embeddings are more robust to certain other types of objects (e.g., common objects).

[0007] Some embodiments disclosed herein additionally or alternatively relate to determining object descriptors that describe objects relevant to the performance of the task and likely to be present in the environment based on (1) FF NL instructions for the robot to perform the task and (2) region embedding(s) generated for identified regions of interest in the environment. For example, a superset of candidate object descriptors that each describe an object relevant to the performance of the task may be determined based on the FF NL instructions. For example, object descriptor(s) explicitly present in the FF NL instructions may be determined, and / or object descriptor(s) that are not explicitly present in the FF NL instructions but are derivable from the FF NL instructions may be determined. Furthermore, the text embedding for each object descriptor may be compared to the region embedding for each region of interest. A comparison of the text embeddings and the region embeddings may generate an indication of the likelihood that the text (corresponding to the text embedding) describes the region of interest (corresponding to the region embedding). Any object descriptor whose text embedding is close (or at least a threshold amount close) to any of the region embeddings may be included in the subset of the superset of object descriptors.

[0008] Thus, the determined subset of object descriptors describes object(s) that are (a) relevant to the execution of the FF NL instruction's task and (b) likely to be included in the environment (as indicated by the closeness of the text embedding(s) of the corresponding object descriptor(s) to the region embedding(s). Through these and other methods, embodiments enable efficient and low-latency determination of object descriptors that describe objects that are relevant to the execution of the FF NL instruction's task and likely to be present in the environment. Furthermore, previously generated region embeddings for the region of interest may be utilized in such determination, thereby eliminating the need for on-demand searching to identify object(s) that are relevant to the execution of the FF NL instruction's task and likely to be present in the environment. Furthermore, comparison of text embeddings and region embeddings and / or other techniques described herein enable identification of object(s) in the environment that are relevant to the FF NL instruction without requiring the FF NL instruction to conform to any object classification grammar.

[0009] Some embodiments disclosed herein additionally or alternatively relate to utilizing a subset of object descriptors determined to describe object(s) relevant to performing the task of the FF NL instructions and likely to be included in the environment when determining robot skill(s) to be performed by the robot(s) to perform the task specified by the FF NL instructions. For example, the subset may be utilized, optionally without including other object descriptor(s), when determining robot skill(s) for the robot(s) to perform the task. This may prevent unnecessary consideration of robot skill(s) specific to object(s) that do not correspond to the object descriptor(s) of the subset and / or unnecessary and erroneous selection (and implementation) of such robot skill(s). In some embodiments, utilizing the determined subset of object descriptor(s) in determining robot skill(s) for the robot(s) to perform the task includes using an LLM to generate an instance of an LLM output based on processing the determined object descriptors and the FF NL instructions. In some of these embodiments, an instance(s) of the LLM output is used in determining the robot skills for a robot(s) in an environment to perform a task.

[0010] The above description is provided as a summary of only some of the embodiments disclosed herein. These and other embodiments are described in more detail herein, including in the detailed description and claims.

[0011] It should be understood that all combinations of the foregoing concepts, and of the foregoing concepts and additional concepts described in more detail herein, are contemplated as being part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. [Brief explanation of the drawings]

[0012] [Figure 1A] An example is shown in which a human gives free-form (FF) natural language (NL) commands to an exemplary robot. [Figure 1B1] 1B shows a simplified bird's-eye view of an exemplary environment in which the human and robot of FIG. 1A are located, illustrating exemplary visual data instances previously acquired within the environment. [Figure 1B2] FIG. 1B1 shows a bird's-eye view of the location of previously determined regions of interest that are determined to be associated with corresponding object descriptors that describe objects relevant to the task of the FF NL instruction in FIG. 1A. [Figure 2] 1 is a flowchart illustrating an exemplary method for controlling a robot based on FF NL input and based on descriptor(s) of an object in the same environment as the robot and associated with the FF NL input, according to embodiments disclosed herein. [Figure 3] 3 is a flow chart illustrating an implementation of block 300 of the method of FIG. 2. [Figure 4] 4 is a flow chart illustrating an implementation of block 400 of the method of FIG. 2. [Figure 5] 5 is a flow chart illustrating an implementation of block 500 of the method of FIG. 2. [Figure 6] 1 illustrates a schematic diagram of an exemplary architecture of a robot. [Figure 7] 1 illustrates a schematic diagram of an exemplary architecture of a computer system. DETAILED DESCRIPTION OF THE INVENTION

[0013] Some implementations disclosed herein generate open vocabulary and queryable scene representations to facilitate language model-based robotic task planning and planned task execution. The scene representation is queryable using an open vocabulary, eliminating the need to restrict the objects involved in robotic task planning to a closed set. Some of these implementations utilize a natural language-based object proposal module to parse free-form (FF) natural language (NL) instructions provided by a human and determine object descriptor(s) associated with the FF NL instructions. Such object descriptor(s) may include descriptor(s) explicitly included in the FF NL instructions and / or descriptor(s) inferred from the FF NL instructions but not explicitly included in the FF NL instructions. The object descriptor(s) (e.g., their text embeddings) may then be used to query an environment map (e.g., represented as region embeddings of regions of interest and their associated estimated map locations) to determine which object descriptor(s) correspond to object(s) present in the environment and to determine the location(s) of these corresponding object(s). The determined object descriptor(s) corresponding to object(s) present in the environment, and the location(s) of at least some of these corresponding object(s), may then be used in planning a robot task and performing the planned task. For example, a large-scale language model (LLM) may be utilized for planning, where the determined object descriptor(s) are processed using the LLM along with FF NL instructions to determine the robot skill(s) to perform to accomplish the task specified by the FF NL instructions. Utilizing the determined object descriptor(s) improves the robustness and / or accuracy of the robot skill(s) determination. Furthermore, querying and utilizing an environment map may allow object(s) not visible to the robot at the time of the FF NL command and / or not specified in the FF NL command to be considered and utilized in planning.Furthermore, querying and utilizing the environment map allows FF NL commands to perform robotic tasks robustly and accurately without having to adhere to the syntax of a specific object classification.

[0014] Referring now to the figures, FIG. 1A illustrates an example in which a human 101 provides a free-form (FF) natural language (NL) command 105 to an exemplary robot 110: "Get ready to wash the fruit."

[0015] 1A is a particular mobile robot. However, additional and / or alternative robots may be utilized with the technology disclosed herein, such as additional robots that differ in one or more respects from the robot 110 illustrated in FIG. 1A. For example, the technology described herein may utilize mobile forklift robots, unmanned aerial vehicles ("UAVs"), non-mobile robots, and / or humanoid robots instead of or in addition to the robot 110.

[0016] The robot 110 includes a base 113 with wheels on opposite sides thereof for movement of the robot 110. The base 113 may include, for example, one or more motors for driving the wheels of the robot 110 to achieve a desired direction, velocity, and / or acceleration of movement of the robot 110. The robot 110 also includes a robotic arm 114 having an end effector 115 in the form of a gripper having two opposing "fingers" or "digits."

[0017] The robot 110 also includes a vision component 111 that can generate visual data (e.g., images) related to the shape, color, depth, and / or other characteristics of an object(s) within the vision component's 111 line of sight. The vision component 111 can be, for example, a monocular camera, a stereographic camera (active or passive), and / or a 3D laser scanner. A 3D laser scanner can include one or more lasers that emit light and one or more sensors that collect data related to reflections of the emitted light. A 3D laser scanner can generate visual component data that is a 3D point cloud, with each point in the 3D point cloud defining the location of a surface point in 3D space. A monocular camera can include a single sensor (e.g., a charge-coupled device (CCD)) and can generate images that each include multiple data points that define color and / or grayscale values ​​based on physical properties sensed by the sensor. For example, a monocular camera can generate images that include red, blue, and / or green channels. Each channel may define a value for each of multiple pixels in the image, such as a value from 0 to 255 for each pixel in the image. A stereographic camera may include two or more sensors, each at a different viewpoint. In some of these embodiments, the stereographic camera generates images that each include multiple data points that define a depth value and a color and / or grayscale value based on characteristics sensed by the two sensors. For example, a stereographic camera may generate an image that includes a depth channel and red, blue, and / or green channels.

[0018] The robot 110 may also include one or more processors. For example, the processor processes the FF NL inputs and map data to determine object descriptor(s) associated with the robot task of the FF NL inputs, determines robot skill(s) for performing the robot task based on the FF NL inputs and object descriptor(s), controls the robot during performance of the robot task based on the determined robot skill(s), etc. For example, the one or more processors of the robot 110 can perform all or part of methods 200, 300, 400, and / or 500 described herein. Additional descriptions of some example structures and functionality of various robots are provided herein.

[0019] 1B1 , a simplified bird's-eye view of an exemplary environment in which the human 101 and robot 110 of FIG. 1A are located is shown. The human 101 and robot 110 are represented by circles in FIG. 1B1 . Additionally, environmental features 191, 192, 193, and 194 are shown in FIG. 1B1 . Environmental features 191, 192, 193, and 194 outline various landmarks in the environment. For example, the environment may be an office kitchen or a workplace kitchen, and features 191 and 192 may be countertops, feature 193 may be a kitchen island, and feature 194 may be a round table.

[0020] 1B1 also illustrates an exemplary visual data instance 180 that was previously captured within the environment. For example, robot 110 may have previously captured visual data instance 180 using vision component 111 while exploring the environment of FIG. 1B1. Visual data instance 180 captures a pear and a key that are present on a round table represented by feature 194. Note that the bird's-eye view does not show the pear, key, or other objects in the environment for simplicity.

[0021] Region of interest 184A of visual data instance 180 is also shown, encompassing the pear within the visual data instance. Region of interest 184B of visual data instance 180 is also shown, encompassing the key within visual data instance 180. As described herein, visual data instance 180 may be processed using a class-independent object detection model to identify regions of interest 184A and 184B. Additionally, visual data of the visual data instance corresponding to region of interest 184A may be processed using a visual language model (VLM) encoder to generate a region embedding of region of interest 184A. An estimated map location, represented by a circle at the end of a line connecting region of interest 184A to the circular table represented by features 194, may also be determined for region of interest 184A. The region embedding of region of interest 184A may be stored in association with the estimated map location of region of interest 184A and, optionally, in association with the size of the region of interest.

[0022] Additionally, the visual data of the visual data instance corresponding to region of interest 184B may be processed using a VLM encoder to generate a region embedding of region of interest 184B. An estimated map location, represented by a circle at the end of a line connecting region of interest 184B to the circular table represented by feature 194, may also be determined for region of interest 184B. The region embedding of region of interest 184B may be stored in association with the estimated map location of region of interest 184B and, optionally, in association with the size of the region of interest.

[0023] For simplicity, only a single visual data instance is shown in Figure 1B1. However, it should be noted that many additional visual data instances have been previously captured in the environment, and regions of interest and corresponding region embeddings, as well as similarly determined estimated map locations and associated data for these visual data instances, have been stored for use in the techniques disclosed herein. Method 300 (e.g., Figure 3) described below includes additional disclosure of embodiments such as identifying regions of interest and generating region embeddings for the regions of interest.

[0024] FIG. 1B2 shows the same bird's-eye view as FIG. 1B1. FIG. 1B2 also shows the locations (represented by stars) within the environment of previously determined regions of interest 184A and 184X that were determined to be relevant to the task of the FF NL instruction 105 of FIG. 1A. More specifically, FIG. 1B2 shows candidate object descriptors 107, each describing a corresponding object that may be relevant to the task of the FF NL instruction 105 of FIG. 1A. For example, the candidate object descriptors 107 may be generated at block 454 of the method 400 of FIG. 4, described below. For example, the object descriptors 107 "apple," "pear," and "banana" may be generated based on being determined to be members of a "fruit" class and "fruit" being included in the FF NL instruction 105. Also, for example, "sink" and "wash" may be generated based on providing prompts to a large language model (LLM) using the FF NL instruction 105. Notably, even though the FF NL instruction 105 does not include "sink," "wash," or any synonyms, an analysis of the prompt input to the LLM and the LLM output may determine that these object descriptors may be relevant to the task of the FF NL instruction 105.

[0025] 1B2 further illustrates that the candidate object descriptor "pear" has been determined to be associated with the region of interest 184A. For example, the text embedding of "pear" can be compared to the region embedding of the region of interest 184A, and the comparison can indicate at least a threshold degree of similarity between the two embeddings. As a result of determining that "pear" is associated with the region of interest 184A, "pear" can be utilized in planning the task and / or location of the region of interest 184A (and optionally, location(s) of similar regions of interest described herein), and the region(s) of interest can be utilized in task planning. For example, "pear" can be processed using the LLM, along with the FF NL instructions, to determine the robot skill(s) to perform to accomplish the task specified by the FF NL instructions 105. As another example, the location of the region of interest 184A can be utilized in performing the determined robot skill toward the "pear." For example, the skill "move to pear" can be performed by utilizing the location of the region of interest 184A.

[0026] 1B2 further illustrates that candidate object descriptor "sink" has been determined to be associated with region of interest 184X (which may correspond to a sink located in an island represented as feature 193). For example, the text embedding of "sink" can be compared to the region embedding of region of interest 184X (the region embedding may be generated based on visual data capturing "sink"), and the comparison may indicate that there is at least a threshold degree of similarity between the two embeddings. As a result of determining that "sink" is associated with region of interest 184X, "sink" can be utilized in task planning and / or task planning at the location of region of interest 184X (and optionally, the location(s) of similar region(s) of interest described herein). For example, "pear" can be processed using the LLM, along with the FF NL instructions, to determine the robot skill(s) to perform to accomplish the task specified by the FF NL instructions 105.

[0027] Further examples of determining candidate object descriptor(s) associated with region(s) of interest are provided herein, for example, in blocks 460 and 462 of method 400 of Figure 4. Further examples of utilizing such associated object descriptor(s) and / or utilizing location(s) of associated region(s) of interest are also provided herein, for example, in method 500 of Figure 5.

[0028] 2, a flowchart of an exemplary method 200 for controlling a robot based on FF NL inputs and based on descriptor(s) of objects (a) present in the robot's environment and (b) associated with the FF NL inputs is shown. For convenience, the operations of method 200 are described with reference to a system that performs the operations. This system may include one or more components of a robot, such as a robot processor and / or robot control system of robot 110, robot 620, and / or other robots, and / or may include one or more components of a computer system, such as computer system 710. Furthermore, although the operations of method 200 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0029] At block 300, the system generates regions of interest and estimated map locations and region embedding(s) for each region of interest based on visual data instances captured throughout the environment. At block 300, the system may further store at least the estimated locations and region embedding(s) for each region of interest in association with one another, for use in, for example, block 400 and / or block 500.

[0030] For example, in block 300, the system may process each captured visual data instance and detect region(s) of interest, if any, in the visual data instance using a class-independent object detection model. The class-independent object detection model may be a machine learning model trained to generate output indicating a bounding box (or other geometric shape) corresponding to any object(s) in the visual data instance. The corresponding bounding box (or other geometric shape) may indicate a region of interest within the visual data instance. The class-independent object detection model is class-independent in that it is trained to detect any object (e.g., to detect "object-ness"), not just object(s) of a particular defined class(es).

[0031] Further, for each detected region, the system may determine an estimated map location of the detected region. For example, the system may determine the estimated map location according to the location of the corresponding visual component that captured the visual data instance, the location of the region of interest within the visual data instance, and, optionally, depth data of the image (if available). Further, for each detected region, the system generates region embedding(s) for that region based on processing visual data from the visual data instance that is within the region of interest. For example, pixels within the region of interest may be processed using a visual language model (VLM) encoder to generate the region embedding. The VLM encoder may be trained to predict a natural language description of the image. For example, the VLM encoder may be trained to predict a probability distribution over a vocabulary of natural language descriptions of the image, such as a vocabulary of hundreds or thousands of natural language descriptions. In such an example, the probability distribution (or the output of a previous layer(s) of the model) may be used as the region embedding. Non-limiting examples of VLM encoders include a contrastive language-image pre-training (CLIP) encoder and a diversified imitation learning with diverse imitation learning (VILD) encoder. Optionally, multiple region embeddings are generated for each ROI, each generated using a different VLM encoder. This allows for the use of multiple region embeddings, for example, in block 400 (described below). This can be beneficial because some type(s) of embedding are more robust to certain type(s) of objects (e.g., out of distribution objects), while other type(s) are more robust to certain other type(s) of objects (e.g., common objects).

[0032] The environment into which the visual data instances utilized by the system in block 300 are captured is a constrained and optionally defined space in which the robot, referenced in block 500 (below), at least selectively operates. The environment may be, for example, a room within a building, multiple rooms within a building, an entire floor of a building, and / or an entire building. In some implementations, the environment in which the robot operates may be dictated by human input (e.g., defining an area to which the robot is confined and / or defining an area to which the robot is prohibited from entering) and / or may be dictated by the robot's constraint(s) (e.g., the robot may not be able to navigate stairs, open any type or certain type(s) of door, etc.).

[0033] The visual data instance utilized by the system in block 300 may be generated by one or more vision components that were (and possibly still are) in the environment. Such vision component(s) may include the robot's vision component(s) referenced in block 500 (described below), the vision component(s) of previous (and possibly current) additional robot(s) in the environment, and / or other fixed or non-fixed vision component(s) in the environment. In some implementations, the visual data instance includes (or is limited to) an image including multiple color channels (e.g., red, green, and blue (RGB) channels) and / or an image including a depth (D) channel. For example, the visual data instance may include an RGB image generated by a monocular camera vision component and / or an RGB-D image generated by a stereo camera vision component. One or more of the visual data instances utilized by the system in block 300 may optionally be generated by the robot(s) during exploration of the environment, such as exploration using predetermined waypoints in the environment and / or exploration using a frontier exploration algorithm.

[0034] In block 400, the system determines object descriptors that describe objects that are relevant to the performance of the task and that are likely to be present in the environment based on (1) the FF NL instructions of the robot performing the task and (2) the region embedding(s) of the region of interest generated in block 300.

[0035] For example, in block 400, based on the FF NL instructions, the system may determine a superset of candidate object descriptors that each describe an object relevant to the execution of the task. For example, the system may determine object descriptor(s) that are explicitly present in the FF NL instructions and / or object descriptor(s) that are not explicitly present in the FF NL instructions but are derivable from the FF NL instructions for inclusion in the superset.

[0036] Further, in block 400, the system may compare the text embeddings for each object descriptor with the region embeddings for each region of interest. For example, comparing the text embeddings to the region embeddings may include generating a dot product result between the embeddings, determining a Euclidean distance measure between the embeddings, and / or other comparisons between the embeddings. Comparing the text embeddings to the region embeddings may yield a measure (e.g., a dot product result, a distance measure, etc.) that indicates the likelihood that the text (corresponding to the region embedding) describes the region of interest (corresponding to the region embedding). Any object descriptor whose text embedding is not close to any (or at least a threshold amount) of the region embeddings, as indicated by the comparison (e.g., the corresponding measure(s) do not meet the threshold(s)), may be excluded from the superset subset of object descriptors. Any object descriptor whose text embedding is close to any (or at least a threshold amount) of the region embeddings may be included in the superset subset of object descriptors.

[0037] Thus, the determined subset of object descriptors describes object(s) that are (a) relevant to the execution of the task of the FF NL instruction and (b) likely to be contained in the environment (as indicated by the closeness of the text embedding(s) of the corresponding object descriptor(s) to the region embedding(s)).

[0038] In these and other ways, embodiments enable a system to efficiently and with low latency determine object descriptors that describe objects that are relevant to the execution of the FF NL instruction's task and that are likely to be present in the environment. Moreover, previously generated region embeddings for each region of interest may be utilized in such determinations, thereby eliminating the need for on-demand searching to identify the object(s) that are relevant to the execution of the FF NL instruction's task and that are likely to be present in the environment. Furthermore, comparison of text embeddings with region embeddings and / or other techniques described herein enables identification of object(s) in the environment that are relevant to the FF NL instruction without requiring the FF NL instruction to conform to any object classification grammar.

[0039] In block 500, the system utilizes the determined object descriptor(s) determined in block 400 when determining robot skill(s) for the robot(s) to perform when performing the task specified in the FF NL instruction in block 400. As described above, the determined object descriptors may be a subset that describes objects that are relevant to performing the task of the FF NL instruction and that are likely to be present in the environment. In block 500, the system may utilize such subset, without other object descriptor(s), when determining robot skill(s) for the robot(s) to perform when performing the task. This may prevent unnecessary consideration of robot skill(s) specific to object(s) that do not correspond to the object descriptor(s) of the subset and / or unnecessary and erroneous selection (and implementation) of such robot skill(s).

[0040] In some embodiments, in block 500, the system utilizes the determined object descriptor(s) determined in block 400 to generate an instance of a large scale language model (LLM) based on processing the determined object descriptor(s) and the FF NL instructions, using the LLM in determining robot skill(s) to perform in performing a task for the robot(s). In some of these embodiments, the system further uses the instance(s) of the LLM output in determining robot skill(s) to perform in performing a task for the robot(s) in the environment.

[0041] 3, 4, and 5, non-limiting examples of blocks 300, 400, and 500 of method 200 of FIG. 2 are described. For convenience, as with FIG. 2, the operations of methods 300, 400, and 500 are described with reference to a system that performs the operations. This system may include one or more components of a robot, such as a robot processor and / or robot control system of robot 110, robot 620, and / or other robots, and / or may include one or more components of a computer system, such as computer system 710. Furthermore, although the operations of methods 300, 400, and 500 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0042] Referring first to FIG. 3, a flow chart illustrating an implementation of block 300 of the method of FIG. 2 is provided.

[0043] In block 352, the system captures visual data instances during exploration of the environment. For example, the visual data instances may include RGB or RGB-D images of the visual component(s) of one or more robot(s), and exploration may be performed by the robot(s) using waypoint exploration and / or frontier exploration techniques.

[0044] In block 354, the system determines whether any of the visual data instances captured in block 352 remain unprocessed. If not, the system proceeds to block 366 and method 300 ends, although it may optionally be re-executed, for example, in response to further exploration of the environment and / or additional instances of visual data of the environment being captured during non-exploration tasks. If, in block 354, the system determines that there are unprocessed visual data instances, the system proceeds to block 356.

[0045] At block 356, the system processes the visual data instance using the class-independent object detection model to identify region(s) of interest in the visual data instance. For example, processing the visual data instance using the class-independent object detection model may generate output indicative of region(s) of interest, such as bounding box(es) or other geometric region(s) of the visual data instance.

[0046] In block 354, the system determines whether there are any unprocessed regions of interest (ROIs) from the visual data instance. If not, the system continues processing by returning to block 354. If there are, the system selects an unprocessed region of interest (ROI) and proceeds to block 360.

[0047] At block 360, the system generates a region embedding of a selected region of interest (ROI) based on processing the visual data of the ROI using a VLM encoder. For example, the visual data instance can be an image, the region of interest can surround pixels of the image, and the system can process the surrounded pixels (exclusively, and optionally after scaling) using a VLM encoder to generate a region embedding. This region embedding can be the final output of the VLM encoder or, optionally, an intermediate output of the VLM encoder.

[0048] Block 360 optionally includes block 360A, in which the system generates additional region embedding(s) for the selected ROI. Each of the additional region embedding(s) may be generated based on processing the visual data for the selected region of interest (ROI) using a corresponding additional VLM encoder. For example, block 360 may include processing surrounded pixels of a region of interest using a first VLM encoder to generate an embedding for the first region, a second VLM encoder to generate an embedding for the second region, and a third VLM encoder to generate an embedding for the third region.

[0049] In block 362, the system determines an estimated map location for the selected ROI. For example, the system may determine the estimated map location according to the location of the corresponding vision component that captured the visual data instance, the location of the region of interest within the visual data instance, and, optionally, image depth data (if available). For example, if the vision component is a robot vision component, the robot's position on the world map may be used in estimating the map location, along with the pose of the vision component (e.g., relative to a robot reference point) and the location of the region of interest within the visual data instance. The estimated map location may be, for example, a three-dimensional position (X, Y, Z) or a four-dimensional position (including height) in Cartesian space, referenced to a reference point in a map of the environment. Note that in various implementations, the estimated map location does not fully specify the six-dimensional pose of the associated object of the region of interest. Rather, the estimated map location may specify only the positional dimensions of the associated object, not its orientation.

[0050] At block 364, the system generates an entry that associates the region embedding(s) of the selected ROI (generated at block 360) with the estimated map location of the selected ROI (generated at block 362), and optionally with the size of the selected ROI (e.g., height / width if the ROI is a bounding box, or just one value if the height / width are the same). For example, this entry may include a unique identifier for the ROI, which can be associated (e.g., using a pointer or other database mapping) with the region embedding of the ROI, the estimated map location of the ROI, and optionally the size of the ROI.

[0051] The system then returns to block 358.

[0052] Referring now to FIG. 4, a flow chart illustrating an implementation of block 400 of the method of FIG. 2 is provided.

[0053] In block 452, the system receives FFNL instructions for the robot to perform a task. For example, the instructions may be voice utterances provided by a human in the environment with the robot, and audio data capturing the voice utterances may be processed using an automatic speech recognition (ASR) model to generate a transcription that includes the FFNL instructions. The audio data may be captured via microphone(s) on the robot or elsewhere in the environment, and the ASR model (and associated processing) may reside on the robot or other computing device(s) in the environment.

[0054] In block 454, the system determines, based on the FF NL instructions, object descriptor(s), each describing a corresponding object relevant to the performance of the robot task. In determining the object descriptor(s), the system optionally executes sub-block(s) 454A, 454B, and / or 454C.

[0055] In sub-block 454A, the system extracts object descriptor(s) directly from the FF NL instruction. For example, the system can extract noun(s) and / or adjective(s) directly from the FF NL instruction. For example, if the NL instruction is "Give me some first aid supplies," then "first aid supplies" may be extracted.

[0056] In sub-block 454B, the system prompts the LLM to generate object descriptor(s) based on the FF NL instructions. The LLM may be different from or the same as the LLM described in FIG. 5, although different priming and / or prompting may optionally be applied. For example, the LLM may be separate and trained to generate object descriptor(s) for object(s) associated with the NL input processed using the LLM. For example, processing the NL input with an LLM model may generate an LLM output including a probability distribution over candidate word structures, where the probability distribution may be used to select candidate word structure(s), and training the LLM to select the selected candidate word structure(s) associated with the NL input. The system may use the LLM to process all or a portion of the FF NL input when generating the object descriptor(s) in sub-block 454B. As an example of sub-block 454B, if the FF NL instruction is "light up the room," the system may prompt the LLM, based on the FF NL instruction, to generate an LLM output that indicates an object descriptor(s) that includes "switch." Note that the LLM output can indicate "switch" even though the FF NL instruction does not include that term or a synonym of that term.

[0057] In sub-block 454C, the system generates specific object descriptor(s) for the category object descriptor determined in block 454A or determined in block 454B. For example, the system may determine that the object descriptor determined in block 454A or determined in block 454B is a category object descriptor that describes a category, and in response, may generate, for the category, specific object descriptor(s) that describe the corresponding members of that category. The system may utilize a knowledge graph or other ontology structure in determining the category object descriptor and the corresponding specific object descriptor(s). As an example of sub-block 454C, for a category object descriptor of "fruit," the system may determine specific object descriptors such as "banana," "apple," and "orange."

[0058] At block 456, the system generates corresponding descriptor embedding(s) for each object descriptor determined at block 454. For example, the system may generate descriptor embeddings for the object descriptors based on processing the descriptor embeddings using a text encoder, which may be a trained machine learning model. The text encoder may optionally be used in training the VLM encoder used in method 300 of FIG. 3.

[0059] In block 458, the system selects a descriptor embedding from those generated in block 456.

[0060] At block 460, the system compares the selected descriptor embedding to a region embedding for the region of interest in the environment. The region embedding may be one generated and stored in the most recent iteration of method 300 of FIG. 3. For example, comparing the selected descriptor embedding to the region embedding may include generating a dot product result between the embeddings, determining a Euclidean distance measure between the embeddings, and / or other comparisons between the embeddings. The comparison of the two embeddings may generate a measure (e.g., a dot product result, a distance measure, etc.) that indicates the likelihood that the selected descriptor (corresponding to the selected descriptor embedding) describes the region of interest (corresponding to the region embedding).

[0061] In block 462, the system determines whether the comparison of block 460 indicates that the descriptor embedding matches any of the region embeddings (or at least a threshold amount thereof). For example, if the comparison of block 460 includes generating a similarity measure for the comparison of the selected descriptor embedding with each of the region embeddings, a match may be determined when the similarity measure meets a threshold value.

[0062] If, in block 462, the system determines that the descriptor embedding does not match any of the region embeddings (or at least a threshold amount of the region embeddings), the system proceeds to block 468. If, in block 462, the system determines whether the descriptor embedding matches at least one of the region embeddings (or at least a threshold amount of the region embeddings), the system proceeds to block 464.

[0063] In block 464, the system adds the selected object descriptor corresponding to the selected descriptor embedding to the current context list. Thus, a selected object descriptor is added to the current context list when the comparison in block 460 indicates that it sufficiently describes the object(s) in the environment.

[0064] In block 466, the system stores the estimated map location(s) for the matching region embedding(s) in association with the object descriptor. For example, if the object descriptor is "fruit," it may be stored in association with the estimated map location(s) of the region(s) of interest that captured a particular banana, and may also be stored in association with other estimated map location(s) of the region(s) of interest that captured a "pear."

[0065] In block 466, the system optionally merges estimated map location(s) for similar matching region embedding(s) in sub-block 466A. For example, there may be five different regions of interest, each generated based on a different visual data instance, but each capturing the same object. Furthermore, one or more of these five different regions may have different estimated map locations, e.g., due to imprecision in the generation of the corresponding estimated map locations. To that end, the system may identify similar matching region embeddings and merge their respective estimated map location(s) into a single estimated map location. For example, the single estimated map location may be the average value of the estimated map locations. In these and other ways, the system may determine that multiple regions of interest relate to the same object and effectively treat them as a single merged region of interest. In determining that matching region embeddings are similar to one another, the system may compare the region embeddings themselves, their estimated map locations, and / or their sizes. For example, if the embeddings of a first region and a second region are close to each other (e.g., within a threshold distance in embedding space), their estimated map locations are close to each other (e.g., within a threshold distance in Cartesian space), and / or their sizes are close to each other (e.g., within a threshold percentage of each other), the region embeddings may be determined to be similar to each other. In other words, the region embeddings may be determined to relate to the same object.

[0066] At block 468, the system determines whether there are more unprocessed descriptor(s). If so, the system proceeds to block 458 and selects another descriptor embedding for another object descriptor. If not, the system proceeds to block 470 and method 400 ends. Upon receiving a new FF NL instruction, method 400 may be executed again.

[0067] Referring now to FIG. 5, a flow chart illustrating an embodiment of block 500 of the method of FIG. 2 is provided.

[0068] In block 552, the system processes the LLM, the object descriptor(s) in the current context list, and the FF NL instruction to generate an LLM output that models the probability distribution. The object descriptor(s) in the current context list processed by the system in block 552 may be the object descriptor(s) from the current context list generated through the iteration of block 464 of method 400 of Figure 4. The FF NL instruction processed by the system in block 552 may be the one received in block 452 of method 400 of Figure 4.

[0069] In block 554, the system determines whether to perform one of the robot skills based on the LLM output and the skill description(s) of the robot skill(s). In the first iteration of block 554, the LLM output is that generated in block 552. In further iterations of block 554, the LLM output is that generated in the most recent iteration of block 562.

[0070] In some implementations, at block 554, the LLM output can model a probability distribution over word compositions, and the system can generate a corresponding skill grounding index for each skill description(s) that reflects the probability of the skill description and compare it to its corresponding robot skill in the LLM output. In other words, the skill grounding index for a skill description can reflect the probability of that skill description (and corresponding robot skill) appearing in the probability distribution of the LLM output. In some of these implementations, the system can determine to implement a robot skill if the skill grounding index for that skill description (a) is the most probable skill grounding index among all skill grounding indexes generated in the iterations of block 554, and optionally (b) satisfies a threshold. Furthermore, in some of these implementations, the system can determine not to implement any robot skill if all skill grounding indexes generated in the iterations of block 554 do not satisfy a threshold. In some additional or alternative embodiments, in block 554, the system also generates a grounding indicator for the “completed” description (e.g., a description indicating that the task is completed / finished and / or a description indicating that the task is unachievable). In some of these embodiments, the system may decide not to perform any robotic skills when the grounding indicator for the “completed” description is the most likely and / or meets a threshold. Regardless of the technique(s) utilized in determining not to perform a robotic skill, if such a determination is made in the first iteration of block 554, it may indicate that the robot is unable to perform the task (e.g., required environmental object(s) are unavailable and / or required robotic skill(s) are not performable by the robot). If such a determination is made in a subsequent iteration of block 554, it may indicate that task execution is complete (e.g., previously performed robotic skill(s) have completed the task).

[0071] Block 554 may include sub-block 554A. In block 554A, the system compares the LLM output to a skill description for each of N candidate robot skills, where N is a subset of the superset of M robot skills that the robot is capable of performing, in making a decision based on the LLM output and the skill description(s). Sub-block 554A may include an optional further sub-block 554A1, in which the system selects N candidate robot skills from the superset of M robot skills. The selection is made based on comparing the skill descriptions for the superset of M robot skills to the object descriptor(s) in the current context list (e.g., determined in iteration(s) of block 464 of method 400 of FIG. 4 ) and / or the region embedding of the region of interest (e.g., determined in iteration(s) of block 360 of method 300 of FIG. 3 ). For example, the system may compare text embeddings of at least a portion of the skill descriptions of the superset of M robot skills (e.g., at least a portion describing an object(s) that can be interacted with based on the robot skill) with the text embeddings and / or region embeddings of the object descriptor(s). Furthermore, the system may select N candidate robot skills for which the comparison indicates at least a threshold degree of similarity. In these and other methods, the subset of N candidate robot skills may include only skills related to the object descriptor(s) in the current context list and / or objects in the environment (as indicated by the region embeddings). Such selection of the subset of N candidate robot skills may prevent an erroneous decision at block 554 to implement robot skills that are irrelevant to the task and / or infeasible given the current environmental objects. Such selection of the subset of N candidate robot skills may additionally or alternatively enable efficient comparison of the LLM output with the corresponding candidate skill descriptions.

[0072] As a specific example of block 554A, assume that the object descriptor(s) in the current context list include "banana," but exclude "bottle," "drink container," or any similar descriptor(s). Further, assume that "pick up fruit" is a skill descriptor for a candidate robot skill, and "pick up bottle" is a skill descriptor for another candidate robot skill. In such an example, the candidate robot skill for "pick up fruit" may be selected in block 554A if a comparison of the text embeddings of "banana" to the text embeddings of "fruit" indicates a threshold degree of similarity. However, the candidate robot skill for "pick up bottle" may be excluded in block 554A based on a comparison of the text embeddings of "bottle" to the text embeddings of the object descriptor(s) in the current list not indicating a threshold degree of similarity.

[0073] At block 556, the system determines that if the decision at block 554 was to not perform any of the robotic skills, then proceed to block 558. If the system proceeds to block 558 on the first iteration of block 554, then the system determines an "error" indicating that the task cannot be performed. Optionally, the system provides a user interface output (e.g., auditory, visual, and / or tactile) to indicate to the human user(s) that the task cannot be performed. If the system proceeds to block 558 on a subsequent iteration of block 554, then the system determines that the task is complete. Optionally, the system provides a user interface output (e.g., auditory, visual, and / or tactile) to indicate to the human user(s) that the task cannot be completed.

[0074] In block 556, the system determines that if the decision in block 554 was to not perform one of the robot skills, then proceed to blocks 560 and 562.

[0075] In block 560, the system implements the determined robot skill (determined in the most recent iteration of block 554). For example, if the determined robot skill has a skill descriptor of "navigate to bananas," the system may implement the determined robot skill to navigate the robot to bananas in the environment.

[0076] In some implementations, block 560 includes sub-block 560A, in which the system utilizes the map locations determined for the corresponding regions of interest (e.g., regions of interest corresponding to matching region embedding(s) determined in the iteration(s) of block 466 of method 400) when performing the robot skill. For example, if the determined robot skill has a skill descriptor of “move to banana,” the system may determine region embedding(s) for the region(s) of interest that have a threshold degree of similarity to the text embedding of “banana.” Furthermore, the system may utilize a map location for one of these region embedding(s) when performing the “move to robot” robot skill (e.g., the robot skill can navigate to a map location). The map location may be, for example, the merged estimated map location determined in the iteration of sub-block 466A of method 400 of FIG. 4 .

[0077] In block 562, the system processes the skill description of the implemented robot skill using the LLM to generate additional LLM outputs, and then returns to block 554 to perform a new iteration of block 554 taking into account the additional LLM outputs. In some implementations, block 554 includes sub-block 554A, where the system processes the skill description of the implemented robot skill using the LLM, and the object descriptor(s) of the current context list and FF NL instruction.

[0078] 6 schematically illustrates an exemplary structure of a robot 620. The robot 620 includes a robot control system 660, one or more motion components 640a-640n, and one or more sensors 642a-642m. The sensors 642a-642m may include, for example, vision sensors, light sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors, accelerometers, gyroscopes, thermometers, barometers, etc. Although the sensors 642a-642m are shown as being integrated with the robot 620, this is not meant to be limiting. In some implementations, the sensors 642a-642m may be located external to the robot 620, for example, as stand-alone units.

[0079] The motion components 640a-640n may include, for example, one or more end effectors and / or one or more servo motors or other actuators to achieve movement of one or more components of the robot. For example, the robot 620 may have multiple degrees of freedom, and each of the actuators may control the actuation of the robot 620 within one or more degrees of freedom in response to a control command. As used herein, the term actuator encompasses a mechanical or electrical device that produces movement (e.g., a motor) in addition to any driver(s) that may be associated with the actuator and that converts received control commands into one or more signals to drive the actuator. Thus, providing a control command to an actuator may include providing the control command to a driver that converts the control command into an appropriate signal to drive an electrical or mechanical device to produce the desired movement.

[0080] The robot control system 660 may be implemented in one or more processors, such as the CPU, GPU, and / or other controller(s) of the robot 620. In some embodiments, the robot 620 may include a "brain box," which may include all or a portion of the control system 660. For example, the brain box may provide real-time bursts of data to the motion components 640a-n, each of which includes, among other things, a set of one or more control commands that dictate parameters of movement (if any) for each of the one or more motion components 640a-n. In some embodiments, the robot control system 660 may perform one or more aspects of the method(s) described herein, such as method 200 of FIG. 2 and / or method 300 of FIG. 3, method 400 of FIG. 4, or method 500 of FIG. 5.

[0081] As described herein, in some implementations, all or a portion of the control commands generated by control system 660 in controlling the robot during performance of a robotic task may be generated based on the determined robot skill(s) associated with the robotic task and, optionally, based on the determined map location(s) of the environmental object(s). In some implementations, although control system 660 is shown in FIG. 6 as an integral part of robot 620, all or a portion of control system 660 may be implemented in a component separate from but in communication with robot 620. For example, all or a portion of control system 660 may be implemented on one or more computing devices in wired and / or wireless communication with robot 620, such as computing device 710.

[0082] 7 is a block diagram of an exemplary computing device 710 that may optionally be utilized to perform one or more aspects of the techniques described herein. The computing device 710 typically includes at least one processor 714 that communicates with several peripheral devices via a bus subsystem 712. These peripheral devices may include, for example, a storage subsystem 724 including a memory subsystem 725 and a file storage subsystem 726, a user interface output device 720, a user interface input device 722, and a network interface subsystem 716. The input and output devices enable user interaction with the computing device 710. The network interface subsystem 716 provides an interface to external networks and is connected to corresponding interface devices in other computing devices.

[0083] The user interface input devices 722 may include pointing devices such as keyboards, mice, trackballs, touchpads, graphics tablets, scanners, touchscreens integrated into displays, audio input devices such as voice recognition systems, microphones, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 710 or a communications network.

[0084] The user interface output devices 720 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or other mechanism for producing a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include any type of device or method for outputting information from the computing device 710 to a user or to another machine or computing device.

[0085] Storage subsystem 724 stores programs and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 724 may include logic for performing selected aspects of method 200 of FIG. 2, method 300 of FIG. 3, method 400 of FIG. 4, and / or method 500 of FIG. 5.

[0086] These software modules typically execute on the processor 714 alone or in combination with other processors. The memory 725 used by the storage subsystem 724 may include multiple memories, such as a main random access memory (RAM) 730 for storing instructions and data during program execution and a read-only memory (ROM) 732 in which fixed instructions are stored. The file storage subsystem 726 provides persistent storage of program and data files and may include a hard disk drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of an embodiment may be stored by the file storage subsystem 726 in the storage subsystem 724 or on another machine accessible by the processor(s) 714.

[0087] Bus subsystem 712 provides a mechanism that allows the various components and subsystems of computing device 710 to communicate with each other in an intended manner. Although bus subsystem 712 is shown illustratively as a single bus, other implementations of a bus subsystem may use multiple buses.

[0088] Computing device 710 may be a variety of types of device, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device 710 shown in Figure 7 is intended only as a specific example to illustrate some implementations. Many other different configurations of computing device 710 are possible, having more or fewer components than the computing device shown in Figure 7.

[0089] Other embodiments may include a non-transitory computer-readable storage medium storing instructions executable by one or more processor(s) (e.g., central processing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s)), and / or tensor processing unit(s) (TPU(s))) to perform a method, such as one or more of the methods described herein. Still other embodiments may include one or more computer and / or one or more robotic systems including one or more processors operable to execute the stored instructions to perform a method, such as one or more of the methods described herein.

[0090] In some embodiments, a method is provided that includes identifying captured visual data instances using one or more vision components throughout an environment of at least one robot. The method further includes processing the visual data instances to identify regions of interest within the environment and, for each region of interest, determining an estimated location of the region of interest and a region embedding for the region of interest, the region being in a natural language embedding space and semantically corresponding to visual features of the region of interest. The method further includes storing, for each region of interest, an association between the estimated location of the region of interest and the region embedding of the region of interest. The method further includes identifying instructions for the robot to perform a task. The instructions may be free-form natural language instructions generated based on user interface input provided by a user via one or more user interface input devices. The method further includes determining, based on the instructions, object descriptors that each describe corresponding candidate environmental objects relevant to performing the task. The method further includes comparing the object descriptor embeddings for the object descriptors with the region embeddings for the regions of interest to identify a subset of object descriptors that each describe corresponding objects that are likely to be present in the environment. The method further includes, in response to identifying the subset of object descriptors, processing the subset of object descriptors and the instructions using a large-scale language model (LLM) to generate an LLM output. The generated LLM output may model a probability distribution of candidate word configurations that depends on the object descriptors and the instructions. The method further includes determining, based on the LLM output, a skill description, the skill description being a natural language description of a robot skill executable by the robot to perform the robot skill. In response to determining to perform the robot skill, the method further includes causing the robot to perform the robot skill in the current environment.

[0091] These and other implementations of the technology disclosed herein may include one or more of the following features.

[0092] In some implementations, the natural language description of the robot skill includes a skill action descriptor and a skill object descriptor (e.g., one that matches one of the object descriptors of the subset). In some versions of these implementations, the method further includes (a) identifying a given region of interest among the regions of interest based on comparing a skill object descriptor embedding for the skill object descriptor with a region embedding of the given region of interest, and (b) in response to identifying the given region of interest, using an estimated location of the region of interest in causing the robot to perform the robot skill in the environment. In some variations of these versions, the robot skill is a navigation skill, and causing the robot to perform the robot skill in the environment using the estimated location of the region of interest includes moving the robot to a specific location determined based on the robot. In some additional or alternative variations of these versions, the method further includes identifying an additional, given region of interest among the regions of interest based on comparing an embedding of the skill object descriptor for each skill object descriptor with an embedding of the region for each additional, given region of interest; determining that the region of interest and the additional region of interest correspond to the same object based on the estimated position of the region of interest and the estimated position of the additional region of interest; and, in response to determining that the region of interest and the additional region of interest correspond to the same object, using the estimated position of the region of interest and the estimated position of the additional region of interest when causing the robot to perform a robotic skill in the environment. In some of these additional or alternative variations, the robot skill is a navigation skill, and using the estimated position of the region of interest and the estimated position of the additional region of interest to cause the robot to perform the robotic skill in the environment includes (a) determining a specific location according to the estimated position of the region of interest and the estimated position of the additional region of interest; and (b) moving the robot to the specific location. Furthermore, in some of these additional or alternative variations, determining that the region of interest and the additional region of interest correspond to the same object is further based on comparing a first size of the first region of interest with a second size of the second region of interest.

[0093] In some implementations, processing the visual data instances to identify regions of interest in the environment and determining an estimated location and region embedding for each of the regions of interest includes: processing the given visual data instance using a class-independent object detection model for the given visual data instance to identify a given region of interest in the visual data instance; determining an estimated location of the given region of interest based on the given region of interest and a pose of the vision components when the given visual data instance was captured; and generating a region embedding for the given region of interest based on processing a portion of the given visual data instance that corresponds to the given region of interest. Processing the portion may use a visual language model (VLM) encoder trained to predict a natural language description of an image.

[0094] In some embodiments, the method further includes, in response to determining the robot skill to perform, using the LLM to process the subset of object descriptors, the instruction, and the skill description of the robot skill to generate an additional LLM output that models an additional probability distribution over candidate word configurations that depends on the object descriptor, the instruction, and the skill description, determining to perform the additional robot skill based on the additional LLM output and an additional skill description, the additional skill description being an additional natural language description of the additional robot skill executable by the robot, and, in response to determining to perform the additional robot skill, causing the robot to perform the additional robot skill in the environment after performance of the robot skill in the environment. In some of these embodiments, the method further includes, in response to determining to perform the additional robot skill, using the LLM to process the subset of object descriptors, the instruction, the skill description of the robot skill, and an additional skill description of the additional robot skill to generate a further LLM output that models an additional probability distribution over candidate word configurations that depends on the object descriptor, the instruction, the skill description, and the additional skill description, and determining that performance of the task by the robot is complete based on the further LLM output.

[0095] In some embodiments, the method further includes, in response to determining to perform the robot skill, using the LLM to process a subset of the object descriptors, instructions, and skill descriptions of the robot skill to generate additional LLM outputs that model additional probability distributions over candidate word configurations that depend on the object descriptors, instructions, and skill descriptions, and determining that performance of the task by the robot is complete based on the additional LLM outputs.

[0096] In some implementations, the method further includes generating object descriptor embeddings. In some of these implementations, generating each of the object descriptor embeddings includes processing the corresponding object descriptor using a text encoding model to generate the corresponding object descriptor embedding.

[0097] In some embodiments, the object descriptors include one or more object descriptors not explicitly specified in the instruction. In some versions of these embodiments, determining, based on the instruction, object descriptors that describe each corresponding candidate environmental object relevant to the execution of the task includes processing the instruction to generate an alternative LLM output using the LLM or an additional LLM and determining one or more of the object descriptors based on the alternative LLM output. In some variations of these versions, the method further includes determining a category descriptor for a category based on the alternative LLM output, and determining a given one of the object descriptors based on the given descriptor being a descriptor of a particular object that is a member of the category and based on the category descriptor being determined based on the alternative LLM output. In some additional or alternative variations of these versions, the method further includes identifying a category descriptor for a category present in the instruction and determining a given one of the object descriptors based on the given descriptor being a descriptor of a particular object that is a member of the category and based on the category descriptor being present in the instruction.

[0098] In some embodiments, determining to perform the robot skill based on the LLM output and the skill description, which is a natural language description of the robot skill, includes determining that a probability distribution of the LLM output indicates the skill description has a probability that meets a probability threshold, the probability being greater than other probabilities determined for other candidate skill descriptions for other candidate robot skills that the robot is capable of performing. In some versions of these embodiments, the method further includes selecting only the robot skill and other candidate robot skills from the superset of skills that are capable of being performed by the robot, and determining the probability and other probabilities for only the robot skill and other candidate robot skills in response to the selecting. In some of these versions, selecting only the robot skill and other candidate robot skills is based on comparing the skill descriptor and other skill descriptor to a subset of object descriptors and / or a region embedding of the region of interest.

[0099] In some implementations, a method is provided that includes generating regions of interest and, for each region of interest, an estimated map location and corresponding region embedding based on processing captured visual data instances throughout an environment of one or more robots. The method further includes receiving free-form (FF) natural language (NL) instructions provided via one or more user interface input devices that instruct the robots to perform a task. The method further includes determining object descriptors based on the FF NL instructions and the region embedding for the regions of interest, each object descriptor describing objects relevant to performing the task and likely to be present in the environment. The method further includes utilizing the determined object descriptors in determining robot skills to be performed by at least one robot(s) in performing the task.

[0100] These and other implementations of the technology disclosed herein may include one or more of the following features.

[0101] In some embodiments, the method further includes causing at least one of the robots to perform a robotic skill in the environment.

[0102] In some embodiments, utilizing the determined object descriptors in determining robotic skills to be performed by at least one robot(s) in performing the task includes utilizing the determined object descriptors in large language model (LLM)-based robot planning. In some versions of these embodiments, utilizing the determined object descriptor(s) in determining robotic skill(s) to be performed by at least one robot(s) in performing the task includes generating an instance of an LLM output based on processing the determined object descriptors and an FF LLM instruction using an NL instruction. In some of these versions, utilizing the determined object descriptors to determine robotic skills to be performed by at least one robot(s) in performing the task includes using the instance(s) of the LLM output to determine robotic skills to be performed by robot(s) in the environment in performing the task.

[0103] In some embodiments, the method further includes utilizing at least one of the determined map locations in performing one or more of the determined robot skills.

Claims

1. 1. A method implemented by one or more processors, comprising: capturing visual data instances across an environment of at least one robot using one or more vision components; processing the visual data instances to identify regions of interest within the environment; and for each of the regions of interest: an estimated location of the region of interest; determining a region embedding of the region of interest that is in a natural language embedding space and that semantically corresponds to visual features of the region of interest; storing, for each of the regions of interest, an association between the estimated location of the region of interest and the region embedding of the region of interest; identifying instructions for a robot to perform a task, the instructions being free-form natural language instructions generated based on user interface inputs provided by a user via one or more user interface input devices; determining, based on the instructions, object descriptors describing corresponding candidate environment objects relevant to the performance of the task; comparing an object descriptor embedding of the object descriptor with the region embedding of the region of interest; identifying a subset of the object descriptors, each of which describes a corresponding object that is likely to be present in the environment; responsive to identifying the subset of object descriptors, using a large language model (LLM) to process the subset of object descriptors and the instruction and generate an LLM output that models a probability distribution over candidate word configurations that is dependent on the object descriptors and the instruction; determining to execute the robot skill based on the LLM output and a skill description, the skill description being a natural language description of a robot skill executable by the robot; In response to determining to perform the robotic skill, causing the robot to perform the robot skill within the environment; and A method comprising:

2. The method of claim 1 , wherein the natural language description of the robot skill includes a skill action descriptor and a skill object descriptor.

3. identifying the given region of interest from the regions of interest based on comparing a skill object descriptor embedding of the skill object descriptor with the region embedding of the given region of interest; responsive to identifying the given region of interest, using the estimated location of the region of interest when causing the robot to perform the robot skill within the environment; The method of claim 2 further comprising:

4. the robot skill is a navigation skill, and using the estimated location of the region of interest in causing the robot to perform the robot skill in the environment comprises: The method of claim 3 , comprising moving the robot to a specific location determined based on the estimated location.

5. identifying the additional given region of interest among the regions of interest based on comparing an embedding of the skill object descriptor in the skill object descriptor with the region embedding of the additional given region of interest; determining, based on the estimated position of the region of interest and the estimated position of the additional region of interest, that the region of interest and the additional region of interest correspond to the same object; responsive to determining that the region of interest and the additional region of interest correspond to the same object, using the estimated location of the region of interest and the estimated location of the additional region of interest when causing the robot to perform the robot skill in the environment; The method of claim 3 further comprising:

6. the robot skill is a navigation skill, and using the estimated position of the region of interest and the estimated position of the additional region of interest when causing the robot to perform the robot skill in the environment comprises: determining a specific location in response to the estimated location of the region of interest and the estimated location of the additional region of interest; moving the robot to the specific position; The method of claim 5 , comprising:

7. 6. The method of claim 5, wherein determining that the region of interest and the additional region of interest correspond to the same object is further based on comparing a first size of a first region of interest with a second size of a second region of interest.

8. The method of claim 3 , wherein the skill object descriptor matches one of the object descriptors of the subset.

9. Processing the visual data instances to identify the regions of interest within the environment and to determine, for each of the regions of interest, the estimated location and embedding of the region, comprises: For a given one of said visual data instances, processing the given visual data instance using a class-independent object detection model to identify a given region of interest within the visual data instance; determining the estimated location of the given region of interest based on the given region of interest and a pose of a vision component when the given instance of visual data was captured; generating the region embedding for the given region of interest based on processing a portion of the given visual data instance that corresponds to the given region of interest, wherein processing the portion is using a visual language model (VLM) encoder that has been trained to predict a natural language description of an image; The method of claim 1 , comprising:

10. In response to determining to perform the robotic skill, using the LLM to process the subset of object descriptors, the instructions, and the skill descriptions of the robot skills to generate additional LLM outputs that model additional probability distributions over the candidate word configurations that depend on the object descriptors, the instructions, and the skill descriptions; determining to perform the additional robot skill based on the additional LLM output and an additional skill description, the additional skill description being an additional natural language description of an additional robot skill executable by the robot; In response to determining to perform the additional robotic skill, and after performing the robot skill in the environment, causing the robot to perform the additional robot skill in the environment. The method of claim 1 further comprising:

11. In response to determining to perform the additional robotic skill, using the LLM to process the subset of object descriptors, the instruction, the skill description of the robot skill, and the additional skill description of the additional robot skill, and generate a further LLM output that models an additional probability distribution over the candidate word configurations that depends on the object descriptor, the instruction, the skill description, and the additional skill description; determining, based on the further LLM output, that performance of the task by the robot has been completed; and The method of claim 10 further comprising:

12. In response to determining to perform the robotic skill, using the LLM to process the subset of object descriptors, the instructions, and the skill descriptions of the robot skills to generate additional LLM outputs that model additional probability distributions over the candidate word configurations that depend on the object descriptors, the instructions, and the skill descriptions; determining, based on the further LLM output, that performance of the task by the robot has been completed; and The method of claim 1 further comprising:

13. The method of claim 1 , further comprising generating the object descriptor embedding.

14. generating each of said object descriptor embeddings, The method of claim 13 , comprising processing the corresponding object descriptor using a text encoding model to generate the corresponding object descriptor embedding.

15. The method of claim 1 , wherein the object descriptors include one or more object descriptors that are not explicitly specified in the instruction.

16. determining, based on the instructions, object descriptors describing corresponding candidate environment objects associated with performing the task, each object descriptor describing a corresponding candidate environment object; processing the instruction using the LLM or an additional LLM to generate an alternative LLM output; determining one or more of the object descriptors based on the alternative LLM output; 16. The method of claim 15, comprising:

17. determining a category descriptor for the category based on the alternative LLM output; determining a given descriptor of the object descriptor based on the given descriptor being a descriptor of a particular object that is a member of the category and based on the category descriptor being determined based on the alternative LLM output; 17. The method of claim 16, further comprising:

18. identifying category descriptors for categories present in said instructions; determining a given one of the object descriptors based on the given descriptor being a descriptor of a particular object that is a member of the category and based on the category descriptor being present in the instruction; 17. The method of claim 16, further comprising:

19. determining to perform the robot skill based on the LLM output and the skill description, the skill description being the natural language description of the robot skill, 2. The method of claim 1, comprising determining that the probability distribution of the LLM output indicates the skill description with a probability that meets a probability threshold, the probability being greater than other probabilities determined for other candidate skill descriptions for other candidate robot skills that the robot is capable of performing.

20. selecting only the robot skill and the other candidate robot skills from a superset of skills executable by the robot; determining the probability and the other probability for only the robot skill and the other candidate robot skills in response to the selecting; 20. The method of claim 19, further comprising:

21. The method of claim 20, wherein selecting only the robot skill and the other candidate robot skills is based on comparing the skill descriptor and the other skill descriptors to the subset of object descriptors and / or the region embedding of the region of interest.

22. generating regions of interest and, for each of said regions of interest, an estimated map location and a corresponding region embedding based on processing the captured visual data instances throughout the environment of one or more robots; receiving free-form (FF) natural language (NL) instructions provided via one or more user interface input devices that instruct the robot to perform a task; determining object descriptors that describe objects that are relevant to the execution of the task and likely to be present in the environment based on the FF NL instructions and the region embedding of the region of interest; utilizing the determined object descriptors in determining robot skills to be performed by at least one of the robot(s) in performing the task; The method comprising:

23. 23. The method of claim 22, further comprising causing the at least one of the robots to perform the robotic skill in the environment.

24. Utilizing the determined object descriptors when determining the robot skills to be performed by at least one of the robot(s) when performing the task includes: The method of claim 22, comprising utilizing the determined object descriptors for large language model (LLM) based robot planning.

25. Utilizing the determined object descriptors in determining the robot skills to be performed by at least one of the robot(s) in performing the task, 25. The method of claim 24, further comprising using an LLM to instantiate an LLM output based on the determined object descriptor and processing the FF NL instruction.

26. Utilizing the determined object descriptors in determining the robot skills to be performed by at least one of the robot(s) in performing the task, 25. The method of claim 24, further comprising using the instance(s) of the LLM output in determining robot skills to be performed by robot(s) in the environment in performing the task.

27. 23. The method of claim 22, further comprising utilizing at least one of the determined map locations in performing one or more of the determined robot skills.

28. 1. A method implemented by one or more processors, comprising: capturing visual data instances across an environment of at least one robot using one or more vision components; For each of said visual data instances: processing the visual data instance using a class-independent object detection model to identify regions of interest within the visual data instance; For each of a plurality of regions of interest identified from the visual data instance, determining an estimated location of the region of interest; generating a region embedding for the region of interest based on processing visual data corresponding to the region of interest from among the corresponding visual data instances, wherein generating the region embedding comprises processing the visual data using a visual language model encoder trained to predict a natural language description of an image; generating an entry associating the estimated location with the region embedding; Following creating the entry, identifying instructions for a robot to perform a task, the instructions being free-form natural language instructions generated based on user interface inputs provided by a user via one or more user interface input devices; determining, based on the instructions, one or more object descriptors each describing a corresponding object involved in performing the task; processing each of the one or more object descriptors using a text encoding model to generate a corresponding object descriptor embedding; comparing the object descriptor embeddings to the region embeddings of the entries to identify a subset of the object descriptor embeddings, each subset corresponding to at least one of the entries; responsive to identifying a subset of object descriptors, processing the subset of object descriptors and the instructions using a large scale language model (LLM) to generate an LLM output that models a probability distribution over candidate word configurations that is dependent on the object descriptors and the instructions; determining to execute the robot skill based on the LLM output and a skill description, the skill description being a natural language description of a robot skill executable by the robot; In response to determining to perform the robotic skill, causing the robot to perform the robot skill within the environment; and A method comprising:

29. A system comprising a memory storing instructions and one or more processors operable to execute said instructions to implement a method according to any of the preceding claims.

30. 30. The system of claim 29, wherein the system includes one or more robots.

31. One or more computer readable media storing instructions that, when executed by one or more processors, perform the method of any of claims 1 to 28.

Images