Robot control based on natural language instructions and based on descriptors of objects present in environment of robot

Abstract

Some implementations relate to generating regions of interest, and an estimated map location and region embedding for each region of interest based on processing instances of visual data captured throughout the environment. Some implementations additionally or alternatively relate to determining an object descriptor describing an object related to performing a task and possibly present in an environment based on (1) a free form (FF) natural language (NL) instruction for the robot to perform the task and (2) a generated region embedding for an identified region of interest in the environment. Some implementations additionally or alternatively involve utilizing a subset of object descriptors that are determined to describe objects that are related to a task that executes an FF NL instruction and that may be included in an environment in determining robot skills that the robot implements in executing the task specified in the FF NL instruction.

Description

Background Art

[0001] Many robots are programmed to perform certain tasks. For example, a robot on an assembly line may be programmed to recognize certain objects and perform specific actions on those objects.

[0002] Further, some robots may perform certain tasks in response to explicit user interface input corresponding to a certain task. For example, a vacuuming robot may perform a general vacuuming task in response to the spoken utterance "robot, clean". However, typically, user interface input that causes a robot to perform a certain task must be explicitly mapped to a task. As a result, the robot may not be able to perform certain tasks in response to various free-form natural language inputs from a user attempting to control the robot. Summary of the invention

[0003] Efforts have been made to attempt to implement robust free-form (FF) natural language (NL) control of robots. For example, enabling a robot to respond appropriately in response to any of a variety of different typed or spoken instructions from a person to the robot. For example, in response to the FF NL instruction "put the purple unicorn plush toy in the toy bin", a robot task including (a) navigating to the "purple unicorn plush toy", (b) picking up the "purple unicorn plush toy" (toy), (c) navigating to the "toy bin", and (d) putting the "purple unicorn plush toy" into the "toy bin" can be performed. However, for various FF NL instructions, various techniques may fail to determine and / or implement the steps of the robot task - and / or may suffer from robotic inefficiencies in implementing the steps of the task and / or robotic and / or other computational inefficiencies in determining the steps of the task.

[0004] For example, some techniques can only perform tasks involving objects that are currently observable by the robot (e.g., within the line of sight of the robot's visual components). For example, if the robot cannot currently observe "purple unicorn plush" and / or "toy box", then those techniques will not be able to "put the purple unicorn plush thing into the toy box". As another example, some techniques can store representations of objects that were previously detected and are currently unobservable by the robot. However, such representations are typically stored as one of a plurality of constrained predefined categories, such that, for example, "purple unicorn plush" cannot be identified because the representations of the corresponding objects are only stored in association with different representations. As yet another example, some techniques can implement exploration in response to FF NL instructions to attempt to locate related objects. However, such reactive exploration delays the execution of the corresponding task. Further, in a case where the relevant objects do not exist (or cannot be located) in the environment, such exploration is performed unnecessarily, resulting in a waste of robot resources.

[0005] In view of these and / or other considerations, some implementations disclosed herein are directed to generating and utilizing an open vocabulary and a queryable scene representation to facilitate robot task planning based on a language model and the implementation of the planned tasks. The scene representation can be queried using an open vocabulary, which prevents the objects involved in the robot task planning from having to be restricted to a closed set. Some of those implementations parse free form (FF) natural language (NL) instructions provided by a person and determine object descriptors associated with the FF NL instructions. Such object descriptors may include descriptors explicitly included in the FF NL instructions and / or descriptors inferred from the FF NL instructions but not explicitly included in the instructions. Object descriptors (e.g., their text embeddings) can then be used to query an environment map (e.g., the environment map is represented as a region embedding of an area of ​​interest and its associated estimated map position) to determine which object descriptors correspond to objects present in the environment and determine the locations of those corresponding objects. Then, the object descriptors determined to correspond to objects present in the environment and the locations of at least some of those corresponding objects can be used for robot task planning and the implementation of the planned tasks. For example, a large language model (LLM) may be utilized for planning, and the LLM may be used in conjunction with FF NL instructions to process determined object descriptors to determine a robot skill that is implemented to complete a task specified by the FF NL instructions. The determined object descriptors are utilized to improve the robustness and / or accuracy of the determined robot skill. Further, querying and utilizing the environment map enables consideration and use of objects in planning that may not be observable by the robot at the time of the FF NL command and / or that may not be specified in the FF NL command. Still further, querying and utilizing the environment map enables robust and accurate robot task execution without requiring the FF NL commands to conform to any particular object classification grammar.

[0006] Some implementations disclosed herein involve generating: regions of interest, and an estimated map location and region embedding for each region of interest based on processing captured visual data instances throughout an environment. Those implementations further store at least the estimated location and region embedding for each region of interest and associated with each other for later use. For example, a class-agnostic object detection model may be used to process each captured visual data instance in detecting regions of interest (if any) in the visual data instance. Further, for each detected region: (1) an estimated map location of the detected region may be determined, and (2) a region embedding for this region may be generated based on processing visual data within the region of interest from the visual data instance. For example, pixels within the region of interest may be processed using a visual language model (VLM) encoder to generate a region embedding. Optionally, multiple region embeddings are generated for each region of interest, and each region embedding is generated using a different VLM encoder. This enables the use of multiple region embeddings in other techniques disclosed herein, which is beneficial because some types of embeddings are more robust to certain types of objects (e.g., out-of-distribution objects), and other types of embeddings are more robust to certain other types of objects (e.g., common objects).

[0007] Some implementations disclosed herein additionally or alternatively involve determining, based on (1) FFNL instructions for a robot to perform a task and (2) generated region embeddings for an identified region of interest in an environment: object descriptors describing objects that are relevant to performing the task and that may be present in the environment. For example, a superset of candidate object descriptors that each describe an object relevant to performing the task may be determined based on the FF NL instructions. For example, object descriptors that are explicitly present in the FF NL instructions may be determined and / or object descriptors that are not explicitly present in the FF NL instructions but may be derived from the instructions may be determined. Further, text embeddings for object descriptors may be compared to region embeddings for regions of interest. Comparison of the text embeddings to the region embeddings may produce a measure indicating the likelihood that the text (corresponding to the text embedding) describes the region of interest (corresponding to the region embedding). Any object descriptor whose text embedding is close to any (or at least a threshold amount of) region embeddings may be included in a subset of the superset of object descriptors.

[0008] Thus, a subset of the determined object descriptors describes objects that: (a) are relevant to the task of executing the FF NL instruction, and (b) are likely to be included in the environment (as indicated by the text embedding of its corresponding object descriptor that is proximate to the region embedding). In these and other ways, an implementation enables efficient and low-latency determination of object descriptors that describe objects that are relevant to the task of executing the FF NL instruction and that are likely to be present in the environment. Further, such determination can be made using previously generated region embeddings for the region of interest—without the need for on-demand exploration to identify objects that are relevant to the task of executing the FF NL instruction and that are likely to be present in the environment. Still further, comparing text embeddings and region embeddings and / or other techniques described herein enables identification of objects in the environment that are relevant to the FF NL instruction without the FF NL instruction needing to conform to any object classification grammar.

[0009] Some implementations disclosed herein additionally or alternatively involve utilizing a subset of object descriptors determined to describe objects related to the task of performing the FF NL instruction and that may be included in the environment in determining the robot skills that the robot implements in performing the task specified in the FF NL instruction. For example, the subset may be utilized in determining the robot skills that the robot implements in performing the task, optionally without other object descriptors. This prevents wasteful consideration of robot skills specific to objects that do not correspond to the object descriptors of the subset and / or wasteful misselection (and implementation) of such robot skills. In some implementations, utilizing the determined subset of object descriptors in determining the robot skills that the robot implements in performing the task includes generating an instance of LLM output based on the object descriptors and FF NL instructions determined using LLM processing. In some of those implementations, an instance of LLM output is used in determining the robot skills that the robot implements in performing the task in the environment.

[0010] The above description is provided as an overview of only some implementations disclosed herein. These and other implementations are described in more detail herein, including in the detailed description and claims.

[0011] It should be understood that all combinations of the aforementioned concepts and additional concepts described in more detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1A An example of a person providing free-form (FF) natural language (NL) instructions to an example robot is shown.

[0013] Figure 1B1 Shows Figure 1AA simplified bird's-eye view of an example environment in which a person and a robot are located, and an example visual data instance previously captured in the environment is shown.

[0014] Figure 1B2 Shows Figure 1B1 A bird's-eye view of the area of ​​interest previously identified and depicted Figure 1A The FF NL instruction is associated with the corresponding object descriptor of the task-related object.

[0015] Figure 2 is a flow chart illustrating an example method of controlling a robot based on FF NL input and based on descriptors of objects present in an environment with the robot and related to the FF NL input according to implementations disclosed herein.

[0016] Figure 3 It is shown Figure 2 Flow diagram of an implementation of block 300 of the method.

[0017] Figure 4 It is shown Figure 2 4. A flow chart of an implementation of block 400 of the method of FIG.

[0018] Figure 5 It is shown Figure 2 Flow diagram of an implementation of block 500 of the method.

[0019] Figure 6 An example architecture of a robot is schematically depicted.

[0020] Figure 7 An example architecture of a computer system is schematically depicted. DETAILED DESCRIPTION

[0021] Some implementations disclosed herein generate open vocabulary and queryable scene representations to facilitate robot task planning based on language models and the implementation of the planned tasks. The scene representation can be queried using the open vocabulary, which prevents the objects involved in the robot task planning from having to be restricted to a closed set. Some of those implementations utilize a natural language-based object proposal module to parse free form (FF) natural language (NL) instructions provided by a person and determine object descriptors associated with the FF NL instructions. Such object descriptors may include descriptors explicitly included in the FF NL instructions and / or descriptors inferred from the FF NL instructions but not explicitly included in the instructions. Object descriptors (e.g., their text embeddings) can then be used to query an environment map (e.g., the environment map is represented as a region embedding of an area of ​​interest and its associated estimated map position) to determine which object descriptors correspond to objects present in the environment and determine the locations of those corresponding objects. Then, the object descriptors corresponding to objects present in the environment and the locations of at least some of those corresponding objects can be used for robot task planning and the implementation of the planned tasks. For example, a large language model (LLM) may be utilized for planning, and the LLM may be used in conjunction with FF NL instructions to process determined object descriptors to determine a robot skill that is implemented to complete a task specified by the FF NL instructions. The determined object descriptors are utilized to improve the robustness and / or accuracy of the determined robot skill. Further, querying and utilizing the environment map enables consideration and use of objects in planning that may not be observable by the robot at the time of the FF NL command and / or that may not be specified in the FF NL command. Still further, querying and utilizing the environment map enables robust and accurate robot task execution without requiring the FFNL commands to conform to any particular object classification grammar.

[0022] Now turning to the attached figure, Figure 1A An example is shown of a person 101 providing a free-form (FF) natural language (NL) instruction 105 to an example robot 110, "get the fruit ready to wash."

[0023] Figure 1A The robot 110 shown in FIG. 1 is a specific mobile robot. However, additional and / or alternative robots may be utilized with the techniques disclosed herein, such as those that are similar in one or more aspects to those of FIG. Figure 1A The robot 110 shown is different from additional robots. For example, in the technology described herein, instead of or in addition to the robot 110, a mobile forklift robot, an unmanned aerial vehicle ("UAV"), a non-mobile robot, and / or a humanoid robot may also be utilized.

[0024] The robot 110 includes a base 113 having wheels on opposite sides thereof for movement of the robot 110. The base 113 may include, for example, one or more motors for driving the wheels of the robot 110 to achieve a desired direction, speed, and / or acceleration of movement of the robot 110. The robot 110 also includes a robot arm 114 having an end effector 115 in the form of a gripper having two opposing "fingers" or "digits".

[0025] The robot 110 also includes a vision component 111 that can generate visual data (e.g., images) related to the shape, color, depth, and / or other features of an object within the line of sight of the vision component 111. The vision component 111 can be, for example, a monocular camera, a stereo camera (active or passive), and / or a 3D laser scanner. The 3D laser scanner may include one or more lasers that emit light and one or more sensors that collect data related to the reflection of the emitted light. The 3D laser scanner can generate vision component data, which is a 3D point cloud, wherein each point in the 3D point cloud defines the location of a point of a surface in 3D space. The monocular camera may include a single sensor (e.g., a charge coupled device (CCD)), and based on the physical properties sensed by the sensor, generates images, each of which includes a plurality of data points defining color values ​​and / or grayscale values. For example, the monocular camera can generate an image including red, blue, and / or green channels. Each channel can define a value for each of a plurality of pixels of the image, such as defining a value from 0 to 255 for each of the pixels of the image. A stereo camera may include two or more sensors, each located at a different vantage point. In some of those implementations, the stereo camera generates an image based on characteristics sensed by the two sensors, each of the images including a plurality of data points defining a depth value and a color value and / or a grayscale value. For example, a stereo camera may generate an image including a depth channel and red, blue, and / or green channels.

[0026] The robot 110 also includes one or more processors that, for example: process the FF NL input and the map data to determine an object descriptor associated with the robot task of the FF NL input; determine a robot skill for performing the robot task based on the FF NL input and the object descriptor; control the robot based on the determined robot skill during the performance of the robot task; etc. For example, the one or more processors of the robot 110 may implement all or aspects of the methods 200, 300, 400, and / or 500 described herein. Additional descriptions of some examples of the structure and functionality of various robots are provided herein.

[0027] Now go to Figure 1B1 , showing Figure 1AA simplified bird's-eye view of an example environment in which a person 101 and a robot 110 are located. Figure 1B1 , a person 101 and a robot 110 are represented as circles. Further, Figure 1B1 1 shows environmental features 191, 192, 193, and 194. Environmental features 191, 192, 193, and 194 show the outlines of landmarks in the environment. For example, the environment may be an office kitchen or a work kitchen, and features 191 and 192 may be countertops, feature 193 may be a kitchen island, and feature 194 may be a round table.

[0028] Figure 1B1 Also shown in FIG. 1 is an example visual data instance 180 previously captured in the environment. For example, the robot 110 may have previously captured Figure 1B1 Visual data instance 180 was captured using visual component 111 during a previous exploration of the environment of . Visual data instance 180 captures a pear and a key, both present on a round table represented by feature 194. It should be noted that in the bird's-eye view, the pear, the key, and other objects of the environment are not shown for simplicity.

[0029] Region of interest 184A of visual data instance 180 is also shown, and encompasses a pear in the visual data instance. Region of interest 184B of visual data instance 180 is also shown, and encompasses a key in visual data instance 180. As described herein, visual data instance 180 may be processed using a class-agnostic object detection model to identify regions of interest 184A and 184B. Further, visual data corresponding to region of interest 184A in the visual data instance may be processed using a visual language model (VLM) encoder to generate a region embedding for region of interest 184A. An estimated map location represented by a circle at the end of a line connecting region of interest 184A to a round table represented by feature 194 may also be determined for region of interest 184A. The region embedding for region of interest 184A may be stored in association with the estimated map location for region of interest 184A, and optionally in association with the size of the region of interest.

[0030] Still further, visual data corresponding to region of interest 184B in the visual data instance may be processed using a VLM encoder to generate a region embedding for region of interest 184B. An estimated map location represented by a circle at the end of a line connecting region of interest 184B to the round table represented by feature 194 may also be determined for region of interest 184B. The region embedding for region of interest 184B may be stored in association with the estimated map location for region of interest 184B, and optionally with a size of the region of interest.

[0031] To keep it simple, Figure 1B1 Only a single visual data instance is shown in FIG. However, it is noted that many additional visual data instances have been previously captured in the environment, and regions of interest and corresponding region embeddings and estimated map locations have been similarly determined for those visual data instances, and the associated data has been stored for use in the techniques disclosed herein. The method 300 described below (e.g., Figure 3 ) includes additional disclosure of implementations of identifying regions of interest, generating region embeddings for regions of interest, etc.

[0032] Figure 1B2 Shows Figure 1B1 Same aerial view of . Figure 1B2 It also shows that it has been determined to be Figure 1A The FF NL instruction 105 is tasked with determining the location of the previously determined regions of interest 184A and 184X in the environment (indicated by the asterisks). More specifically, Figure 1B2 Shows the respective description and Figure 1A The task of the FF NL instruction 105 is potentially related to the candidate object descriptor 107 of the corresponding object. For example, the candidate object descriptor 107 may be described below. Figure 4 454 of method 400. For example, object descriptors 107 of "apple", "pear", and "banana" may be generated based on being determined to be members of the "fruit" class and "fruit" being included in the FF NL instruction 105. Also, for example, "sink" and "brush" may be generated based on prompting a large language model (LLM) using the FF NL instruction 105. Notably, even if the FF NL instruction 105 does not mention "sink", "brush", or any synonyms, prompting the LLM and analyzing the resulting LLM output may still result in those object descriptors being determined to be potentially relevant to the task of the FF NL instruction 105.

[0033] further, Figure 1B2It is shown that the candidate object descriptor "pear" has been determined to be associated with the region of interest 184A. For example, the text embedding of "pear" can be compared to the region embedding of the region of interest 184A, and the comparison can indicate at least a threshold degree of similarity between the two embeddings. As a result of "pear" being determined to be associated with the region of interest 184A, "pear" can be utilized in mission planning and / or the location of the region of interest 184A (and optionally, the location of similar regions of interest as described herein) can be utilized in mission planning. For example, "pear" can be processed using the LLM and in conjunction with the FF NL instructions to determine a robot skill to implement to complete the task specified by the FF NL instructions 105. As another example, the location of the region of interest 184A can be used to implement a determined robot skill for "pear". For example, a "navigate to pear" skill can be implemented by utilizing the location of the region of interest 184A.

[0034] Going further, Figure 1B2 It is shown that the candidate object descriptor "sink" has been determined to be associated with the region of interest 184X (which may correspond to a sink located in the island represented by feature 193). For example, the text embedding of "sink" may be compared to the region embedding of the region of interest 184X (which may be generated based on visual data capturing "sink"), and the comparison may indicate at least a threshold degree of similarity between the two embeddings. As a result of "sink" being determined to be associated with the region of interest 184X, "sink" may be utilized in task planning and / or the location of the region of interest 184X (and optionally, the location of similar regions of interest as described herein) may be utilized in task planning. For example, "sink" may be processed using LLM and in conjunction with FF NL instructions to determine a robot skill that is implemented to complete the task specified by FF NL instructions 105.

[0035] In this article, for example Figure 4 Further examples of determining candidate object descriptors associated with a region of interest are provided in blocks 460 and 462 of method 400. Figure 5 A further example of utilizing the locations of such related object descriptors and / or related regions of interest is provided in method 500 of .

[0036] Now go to Figure 2, a flow chart of an example method 200 for controlling a robot based on FF NL input and based on descriptors of objects that: (a) exist in an environment with the robot and (b) are related to the FF NL input. For convenience, the operations of method 200 are described with reference to a system that performs the operations. This system may include one or more components of a robot, such as a robot processor and / or a robot control system of robot 110, robot 620, and / or other robots, and / or may include one or more components of a computer system (such as computer system 710). In addition, although the operations of method 200 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0037] At block 300, the system generates, based on the visual data instances captured throughout the environment: regions of interest, and an estimated map location and region embedding for each region of interest. At block 300, the system may further store at least the estimated location and region embedding for each region of interest and associated with each other for use in, for example, blocks 400 and / or 500.

[0038] For example, at block 300, the system may process each captured visual data instance using a class-agnostic object detection model to detect regions of interest, if any, in the visual data instance. The class-agnostic object detection model may be a machine learning model trained to generate outputs indicating corresponding bounding boxes (or other geometric shapes) for any object of the visual data instance. The corresponding bounding boxes (or other geometric shapes) may indicate regions of interest in the visual data instance. The class-agnostic object detection model is class-agnostic because it is trained to detect any object (e.g., detect "objectness"), not just objects of certain defined classes.

[0039] Further, for each detected region, the system may determine an estimated map location of the detected region. For example, the system may determine the estimated map location based on the location of the corresponding visual component of the captured visual data instance, the location of the region of interest within the visual data instance, and (optionally) the depth data of the image (if available). Further, for each detected region, the system generates a region embedding for this region based on processing the visual data within the region of interest from the visual data instance. For example, a visual language model (VLM) encoder may be used to process pixels within the region of interest to generate a region embedding. The VLM encoder may be trained to predict a natural language description of an image. For example, the VLM encoder may be trained to predict a probability distribution over a vocabulary of natural language descriptions of an image (such as a vocabulary of hundreds or thousands of natural language descriptions). In such an example, the probability distribution (or the output of an earlier layer of the model) may be used as a region embedding. Non-limiting examples of VLM encoders include contrastive language image pre-training (CLIP) encoders and variational imitation learning with diverse quality demonstrations (VILD) encoders. Optionally, multiple region embeddings are generated for each ROI, and each region embedding is generated using a different VLM encoder. This enables the use of multiple region embeddings, for example in block 400 (described below). This can be beneficial because some types of embeddings are more robust to certain types of objects (e.g., out-of-distribution objects), and other types of embeddings are more robust to certain other types of objects (e.g., common objects).

[0040] The environment in which the visual data instances utilized by the system at block 300 are captured is a constrained and optionally defined space in which a robot (referenced in block 500 (below)) at least selectively operates. The environment may be, for example, a room in a building, multiple rooms in a building, an entire floor of a building, and / or an entire building. In some implementations, the environment in which the robot operates may be determined by human input (e.g., defining areas to which the robot is restricted and / or defining areas that the robot is prohibited from entering) and / or may be determined by the constraints of the robot (e.g., the robot may not be able to navigate stairs, open any or certain types of doors, etc.).

[0041] The visual data instances utilized by the system at block 300 may be generated by one or more visual components in the environment (and perhaps still in the environment). Such visual components may include the visual components of the robot referenced in block 500 (below), the visual components of additional robots previously (and possibly currently) in the environment, and / or other fixed or non-fixed visual components in the environment. In some implementations, the visual data instances include (or are limited to) images including multiple color channels (e.g., red, green, and blue (RGB) channels) and / or including depth (D) channels. For example, the visual data instances may include RGB images generated by a monocular camera vision component and / or RGB-D images generated by a stereo camera vision component. One or more of the visual data instances utilized by the system at block 300 may optionally be generated by the robot during exploration of the environment (such as exploration using predetermined waypoints in the environment and / or exploration using a frontier exploration algorithm).

[0042] At block 400 , the system determines, based on (1) the FF NL instructions for the robot to perform the task and (2) the region embedding for the region of interest generated at block 300 , object descriptors describing objects that are relevant to performing the task and that may be present in the environment.

[0043] For example, the system may determine a superset of candidate object descriptors that each describe an object associated with performing a task based on the FF NL instructions at block 400. For example, the system may determine object descriptors that are explicitly present in the FF NL instructions and / or object descriptors that are not explicitly present in the FFNL instructions but that can be inferred from the instructions for inclusion in the superset.

[0044] Further, at block 400, the system may compare the text embedding for the object descriptor with the region embedding for the region of interest. For example, the comparison of the text embedding with the region embedding may include generating the result of an inner product between the embeddings, determining a Euclidean distance metric between the embeddings, and / or other comparisons between the embeddings. The comparison of the text embedding with the region embedding may produce a metric (e.g., the result of the inner product, a distance metric, etc.) indicating the likelihood that the text (corresponding to the text embedding) describes the region of interest (corresponding to the region embedding). Any object descriptor (whose text embedding is not close to any (or at least a threshold amount) of the region embedding as indicated by the comparison (e.g., the corresponding metric does not meet a threshold)) may be excluded from the subset of the superset of object descriptors. Any object descriptor (whose text embedding is close to any (or at least a threshold amount) of the region embedding) may be included in the subset of the superset of object descriptors.

[0045] Thus, the determined subset of object descriptors describes objects that: (a) are relevant to the task of executing FF NL instructions and (b) are likely to be included in the environment (as indicated by text embeddings of their corresponding object descriptors proximate to the region embedding).

[0046] In these and other ways, implementations enable the system to efficiently and with low latency determine object descriptors describing objects that are relevant to the task of executing FF NL instructions and that may be present in the environment. Further, such determination can be made using previously generated region embeddings for regions of interest - without the need for on-demand exploration to identify objects that are relevant to the task of executing FF NL instructions and that may be present in the environment. Still further, comparing text embeddings and region embeddings and / or other techniques described herein enables identification of objects in the environment that are relevant to FF NL instructions without the FF NL instructions having to conform to any object classification grammar.

[0047] At block 500, the system utilizes the determined object descriptors determined in block 400 in determining the robot skill implemented by the robot in performing the task specified in the FF NL instruction of block 400. As described above, the determined object descriptors may be a subset of objects that are relevant to the task of performing the FF NL instruction and that may be present in the environment. At block 500, the system may utilize such a subset, without other object descriptors, in determining the robot skill implemented by the robot in performing the task. This may prevent wasteful consideration of robot skills specific to objects that do not correspond to the object descriptors of the subset and / or wasteful erroneous selection (and implementation) of such robot skills.

[0048] In some implementations, at block 500, the system utilizes the determined object descriptors determined in block 400 in determining the robotic skill that the robot implements in performing the task based on processing the determined object descriptors and FF NL instructions using a large language model (LLM) to generate an instance of an LLM output. In some of those implementations, the system further uses the instance of the LLM output in determining the robotic skill that the robot implements in performing the task in the environment.

[0049] Now turn to Figure 3 , Figure 4 and Figure 5 , describes Figure 2 Non-limiting examples of blocks 300, 400, and 500 of method 200. For convenience, and similar to Figure 2, the operations of methods 300, 400, and 500 are described with reference to a system that performs the operations. This system may include one or more components of a robot, such as a robot processor and / or a robot control system of robot 110, robot 620, and / or other robots, and / or may include one or more components of a computer system (such as computer system 710). Moreover, although the operations of methods 300, 400, and 500 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0050] First go to Figure 3 , providing an illustration of Figure 2 Flow diagram of an implementation of block 300 of the method.

[0051] At block 352, the system captures visual data instances during exploration of the environment. For example, the visual data instances may include RGB images or RGB-D images of visual components of one or more robots, and the exploration may be performed by the robot using waypoint exploration and / or frontier exploration techniques.

[0052] At block 354, the system determines whether there are unprocessed visual data instances from those visual data instances captured at block 352. If not, the system proceeds to block 366 and the method 300 ends, but the method may optionally be performed again in response to, for example, further exploration of the environment and / or additional instances of visual data of the environment captured during performance of a non-exploratory task. If at block 354 the system determines that there are unprocessed visual data instances, the system proceeds to block 356.

[0053] At block 356, the system processes the visual data instance using the class-agnostic object detection model to identify a region of interest in the visual data instance. For example, processing the visual data instance using the class-agnostic object detection model may generate an output indicating the region of interest, such as a bounding box or other geometric region of the visual data instance.

[0054] At block 354, the system determines whether there is an unprocessed region of interest from the visual data instance. If not, the system proceeds back to block 354. If so, the system selects an unprocessed region of interest (ROI) and proceeds to block 360.

[0055] At block 360, the system generates a region embedding for a selected region of interest (ROI) based on processing the visual data using a VLM encoder for the ROI. For example, the visual data instance may be an image, the region of interest may define pixels of the image, and the system may (specifically, and optionally after scaling) process the boundary pixels using a VLM encoder to generate a region embedding. The region embedding may be the final output of the VLM encoder, or alternatively, an intermediate output of the VLM encoder.

[0056] Block 360 optionally includes block 360A in which the system generates additional region embeddings for the selected ROI. Each of the additional region embeddings may be generated based on processing visual data for the selected region of interest (ROI) using a corresponding additional VLM encoder. For example, block 360 may include processing bounded pixels of the region of interest using a first VLM encoder to generate a first region embedding, processing bounded pixels of the region of interest using a second VLM encoder to generate a second region embedding, and processing bounded pixels of the region of interest using a third VLM encoder to generate a third region embedding.

[0057] In box 362, the system determines the estimated map position of the selected ROI. For example, the system may determine the estimated map position based on the positioning of the corresponding visual component that captured the visual data instance, the position of the region of interest within the visual data instance, and (optionally) the depth data of the image (if available). For example, in the case where the visual component is a robot vision component, the positioning of the robot in the world map, as well as the pose of the visual component (e.g., relative to the robot reference point) and the position of the region of interest within the visual data instance can be used in the estimated map position. The estimated map position can be, for example, a 3-dimensional position (X, Y, Z) or a 4-dimensional position (including height) of a reference point in Cartesian space and referenced to a map of the environment. It should be noted that in various implementations, the estimated map position does not fully specify the 6-dimensional pose of the associated object of the region of interest. Instead, it can specify only the position dimension of the associated object without any specification of the orientation of the associated object.

[0058] At block 364, the system generates an entry associating the region embedding for the selected ROI (generated at block 360) with the estimated map location for the selected ROI (generated at block 362) and optionally with the size of the selected ROI (e.g., height / width when the ROI is a bounding box - or just one value when height / width are the same). For example, the entry may include a unique identifier for the ROI, and this unique identifier may be associated (e.g., using a pointer or other database mapping) with the region embedding for the ROI, the estimated map location of the ROI, and optionally the size of the ROI.

[0059] The system then proceeds back to box 358.

[0060] Next go to Figure 4 , providing an illustration of Figure 2 4. A flow chart of an implementation of block 400 of the method of FIG.

[0061] At block 452, the system receives FF NL instructions for the robot to perform a task. For example, the instructions may be spoken utterances provided by a person in an environment with the robot, and audio data capturing the spoken utterances may be processed using an automatic speech recognition (ASR) model to generate a transcription including the FF NL instructions. The audio data may be captured via a microphone of the robot or elsewhere in the environment, and the ASR model (and associated processing) may be on the robot or other computing device in the environment.

[0062] At block 454, the system determines object descriptors that each describe a corresponding object associated with the performance of the robot task based on the FF NL instructions. In determining the object descriptors, the system optionally executes sub-blocks 454A, 454B, and / or 454C.

[0063] At subblock 454A, the system extracts object descriptors directly from the FF NL instruction. For example, the system can extract nouns and / or adjectives directly from the FF NL instruction. For example, if the NL instruction is "give me some first-aid items", then "first-aid items" can be extracted.

[0064] At subblock 454B, the system prompts the LLM to generate an object descriptor based on the FF NL instruction. Figure 5 The LLM described in , or can be used with Figure 5, but may optionally be primed and / or prompted differently. For example, the LLM may be different and may be trained to generate an object descriptor of an object related to the NL input processed using the LLM. For example, processing the NL input using the LLM model may generate an LLM output that includes a probability distribution over candidate word components, wherein the probability distribution may be utilized to select a word component, and due to the training of the LLM, the selected word component will be related to the NL input. In subblock 454B, the system may process all or part of the FF NL input using the LLM. As an example of subblock 454B, if the FF NL instruction is "light up the room", the system may prompt the LLM based on the FF NL instruction to generate an LLM output indicating an object descriptor that includes "switch". Note that the LLM output may indicate "switch" even though the FF NL instruction does not include this term or any synonyms of this term.

[0065] At sub-block 454C, the system generates a specific object descriptor for the category object descriptor determined at block 454A or determined at block 454B. For example, the system may determine that the object descriptor determined at block 454A or determined at block 454B is a category object descriptor that describes a category, and in response, generate specific object descriptors for the category that each describe a corresponding member of the category. The system may utilize a knowledge graph or other ontology structure in determining the category object descriptor and the corresponding specific object descriptor. As an example of sub-block 454C, for the category object descriptor of "fruit", the system may determine specific object descriptors such as "banana", "apple", "orange", etc.

[0066] At block 456, the system generates a corresponding descriptor embedding for each object descriptor determined at block 454. For example, the system can generate descriptor embeddings for the object descriptors based on processing the descriptor embeddings using a text encoder, which can be a trained machine learning model. The text encoder can optionally be trained on the Figure 3 The encoder utilized in the VLM encoder utilized in method 300.

[0067] At block 458 , the system selects a descriptor embedding from those generated at block 456 .

[0068] At block 460, the system compares the selected descriptor embedding to the region embedding for the region of interest in the environment. Figure 3The comparison of the selected descriptor embedding to the region embeddings may include, for example, the result of generating an inner product between the embeddings, determining a Euclidean distance metric between the embeddings, and / or other comparison between the embeddings. The comparison of the two embeddings may produce a metric (e.g., the result of an inner product, a distance metric, etc.) indicating the likelihood that the selected descriptor (corresponding to the selected descriptor embedding) describes the region of interest (corresponding to the region embedding).

[0069] At block 462, the system determines whether the comparison of block 460 indicates that the descriptor embedding matches any of the region embeddings (or at least a threshold amount of the region embeddings). For example, when the comparison of block 460 includes generating a measure of similarity for comparison of the selected descriptor embedding to each of the region embeddings, a match may be determined when the measure of similarity satisfies a threshold.

[0070] If, at block 462, the system determines that the descriptor embedding does not match any of the region embeddings (or at least a threshold amount of the region embeddings), the system proceeds to block 468. If, at block 462, the system determines that the descriptor embedding does match at least one of the region embeddings (or at least a threshold amount of the region embeddings), the system proceeds to block 464.

[0071] At block 464, the system adds the selected object descriptor corresponding to the selected descriptor embedding to the current context list. Thus, when the comparison of block 460 indicates that the selected object descriptor is sufficient to describe the object in the environment, the descriptor will be added to the current context list.

[0072] The system stores the estimated map locations for the matched region embeddings in association with the object descriptor at block 466. For example, if the object descriptor is "fruit," it may be stored in association with an estimated map location that captures a particular region of interest for bananas, and in association with other estimated map locations that capture a region of interest for "pears."

[0073] At box 466, the system optionally merges estimated map locations for similar matching region embeddings at subbox 466A. For example, it may be the case that there are five different regions of interest that are each generated based on different visual data instances but each capture the same object. Further, one or more of those five different regions may have different estimated map locations due to, for example, inaccuracies in generating the corresponding estimated map locations. Therefore, the system may identify similar matching region embeddings and merge their corresponding estimated map locations into a single estimated map location. For example, the single estimated map location may be an average of the estimated map locations. In these and other ways, the system may determine that multiple regions of interest are associated with the same object and effectively treat them as a single merged region of interest. In determining that the matching region embeddings are similar to each other, the system may compare the region embeddings themselves, their estimated map locations, and / or their sizes. For example, if the first region embedding and the second region embedding are close to each other (e.g., within a threshold distance in the embedding space), their estimated map locations are close to each other (e.g., within a threshold distance in the Cartesian space), and / or their sizes are close to each other (e.g., within a threshold percentage of each other), then the region embeddings may be determined to be similar to each other. In other words, it may be determined that the region embeddings are associated with the same object.

[0074] At block 468, the system determines whether there are more unprocessed descriptors. If so, the system proceeds to block 458 and selects another descriptor embedding for another object descriptor. If not, the system proceeds to block 470 and the method 400 ends. When a new FF NL instruction is received, the method 400 may be executed again.

[0075] Next go to Figure 5 , providing an illustration of Figure 2 Flow diagram of an implementation of block 500 of the method.

[0076] At block 552, the system processes the object descriptors of the current context list and the FF NL instructions using the LLM to generate an LLM output that models the probability distribution. The object descriptors of the current context list processed by the system at block 552 may be from a Figure 4 The object descriptor of the current context list generated by the iteration of block 464 of method 400. The FF NL instruction processed by the system in block 552 may be Figure 4 The instructions received in block 452 of method 400 .

[0077] At block 554, the system determines whether one of the robot skills is implemented based on the LLM output and the skill description of the robot skill. At a first iteration of block 554, the LLM output is the output generated at block 552. At a further iteration of block 554, the LLM output is the output generated at the most recent iteration of block 562.

[0078] In some implementations, at block 554, the LLM output may model a probability distribution over word components, and the system may generate, for each of the skill descriptions, a corresponding skill-based metric reflecting the probability of the skill description and its corresponding robot skill in the LLM output. In other words, the skill-based metric for the skill description may reflect the probability of this skill description (and the corresponding robot skill) reflected in the probability distribution of the LLM output. In some of those implementations, the system may determine that the robot skill is achieved when the skill-based metric for the skill description of the robot skill is: (a) the skill-based metric with the highest probability among all skill-based metrics generated at the iteration of block 554, and optionally (b) satisfies a threshold. Further, in some of those implementations, when all skill-based metrics generated at the iteration of block 554 do not meet the threshold, the system may determine that no robot skill is achieved. In some additional or alternative implementations, at block 554, the system also generates a base metric for a "finished" description (e.g., a description indicating that a task is completed / ended and / or indicating that a task is not complete). In some of those implementations, when the underlying metric for the "done" description is the highest probability and / or meets the threshold, the system may determine that no robot skill is achieved. Regardless of the technique utilized in determining that no robot skill is achieved, when such a determination is made at the first iteration of block 554, this may indicate that the robot is unable to perform the task (e.g., the required environmental objects are not available and / or the robot cannot achieve the required robot skill). When such a determination is made at subsequent iterations of block 554, this may indicate that performance of the task is complete (e.g., the robot skill that has been achieved has completed the task).

[0079] Block 554 may include subblock 554A. At block 554A, in determining based on the LLM output and the skill description, the system compares the LLM output to the skill description for each of the N candidate robot skills, where N is a subset of a superset of the M robot skills that can be performed by the robot. Subblock 554A may include optional further subblock 554A1, in which the system determines based on the skill description for the superset of the M robot skills and (e.g., in Figure 4 464 iteration of method 400) and / or an object descriptor of the current context list (e.g., Figure 3The system may select N candidate robot skills from a superset of the M robot skills based on comparison of region embeddings for regions of interest (determined at an iteration of block 360 of method 300). For example, the system may compare text embeddings of at least a portion of skill descriptions (e.g., describing at least a portion of an object with which the robot skill can interact) in the superset of the M robot skills with text embeddings of object descriptors and / or with region embeddings. Further, the system may select N candidate robot skills whose comparison indicates at least a threshold degree of similarity. In these and other ways, a subset of the N candidate robot skills may include only those skills that are relevant to the object descriptors of the current context list and / or to objects in the environment (as indicated by the region embeddings). Such selection of a subset of the N candidate robot skills may prevent erroneous determination of robot skills that are not task-related and / or not executable given the current environment objects at block 554. Such selection of a subset of the N candidate robot skills may additionally or alternatively enable efficient comparison of the LLM output with corresponding candidate skill descriptions.

[0080] As a specific example of block 554A, assume that the object descriptors of the current context list include "banana" but exclude "bottle," "beverage container," or any similar descriptors. Further assume that "pick up fruit" is a skill descriptor for a candidate robot skill, and that "pick up bottle" is a skill descriptor for a separate candidate robot skill. In such an example, the "pick up fruit" candidate robot skill may be selected at block 554A based on a comparison of the text embedding for "banana" and the text embedding for "fruit" indicating a threshold degree of similarity. However, the "pick up bottle" candidate robot skill may be excluded at block 554A based on a comparison of the text embedding for "bottle" with the text embeddings of the object descriptors of the current list failing to indicate a threshold degree of similarity.

[0081] At box 556, when the determination at box 554 does not achieve any of the robot skills, the system determines to proceed to box 558. When the system proceeds to box 558 at the first iteration of box 554, the system determines an "error" indicating that the task cannot be performed. Optionally, the system provides a user interface output (e.g., audible, visual, and / or tactile) to indicate to a human user that the task cannot be performed. When the system proceeds to box 558 at a subsequent iteration of box 554, the system determines that the task is completed. Optionally, the system provides a user interface output (e.g., audible, visual, and / or tactile) to indicate to a human user that the task is completed.

[0082] At block 556 , when the determination at block 554 achieves one of the robot skills, the system determines to proceed to blocks 560 and 562 .

[0083] At block 560, the system implements the determined robot skill (determined at the most recent iteration of block 554). For example, if the determined robot skill has a skill descriptor of "navigate to bananas," the system may implement the determined robot skill so that the robot navigates to bananas in the environment.

[0084] In some implementations, box 560 includes sub-box 560A, in which the system utilizes the determined map locations for the corresponding areas of interest in implementing the robot skill (e.g., those map locations corresponding to matching region embeddings determined at iterations of box 466 of method 400). For example, if the determined robot skill has a skill descriptor of "navigate to bananas," the system may determine a region embedding for the area of ​​interest that has a threshold degree of similarity to the text embedding for "bananas." Further, the system may utilize the map location for one of those region embeddings in implementing the "navigate to bananas" robot skill (e.g., the robot skill may navigate to the map location). The map location may be, for example, Figure 4 The merged estimated map position is determined in iteration 466A of sub-block 466 of method 400 .

[0085] At block 562, the system processes the skill description of the implemented robot skill using the LLM to generate additional LLM outputs and then proceeds back to block 554 to perform another iteration of block 554 that considers the additional LLM outputs. In some implementations, block 554 includes sub-block 554A in which the system processes the skill description of the implemented robot skill and the object descriptors and FF NL instructions of the current context list using the LLM.

[0086] Figure 6 An example architecture of a robot 620 is schematically depicted. The robot 620 includes a robot control system 660, one or more operating components 640a-640n, and one or more sensors 642a-642m. The sensors 642a-642m may include, for example, vision sensors, light sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors, accelerometers, gyroscopes, thermometers, barometers, etc. Although the sensors 642a-m are depicted as being integral to the robot 620, this is not meant to be limiting. In some implementations, the sensors 642a-m may be located external to the robot 620, for example, as a standalone unit.

[0087] The operating components 640a-640n may include, for example, one or more end effectors and / or one or more servo motors or other actuators to achieve movement of one or more components of the robot. For example, the robot 620 may have multiple degrees of freedom, and each of the actuators may control actuation of the robot 620 within one or more of the degrees of freedom in response to a control command. As used herein, the term "actuator" encompasses a mechanical or electrical device (e.g., a motor) that produces motion, in addition to any driver that may be associated with the actuator and converts a received control command into one or more signals for driving the actuator. Thus, providing a control command to an actuator may include providing a control command to a driver that converts the control command into an appropriate signal to drive the electrical or mechanical device to produce the desired motion.

[0088] The robot control system 660 can be implemented in one or more processors, such as a CPU, GPU, and / or other controller of the robot 620. In some implementations, the robot 620 can include a "brain box" that can include all or aspects of the control system 660. For example, the brain box can provide real-time data bursts to the operating components 640a-n, where each of the real-time bursts includes a set of one or more control commands that specify, among other things, motion parameters (if any) for each of one or more of the operating components 640a-n. In some implementations, the robot control system 660 can perform the methods described herein (such as Figure 2 Method 200, Figure 3 Method 300, Figure 4 Method 400, and / or Figure 5 One or more aspects of method 500).

[0089] As described herein, in some implementations, in controlling the robot during performance of a robotic task, all or aspects of the control commands generated by the control system 660 may be generated based on the robotic skills determined to be relevant to the robotic task and, optionally, based on the determined map locations for environmental objects. Figure 6 60 is shown as an integral part of the robot 620, but in some implementations, all or aspects of the control system 660 may be implemented in a component separate from but in communication with the robot 620. For example, all or aspects of the control system 660 may be implemented on one or more computing devices (such as the computing device 710) in wired and / or wireless communication with the robot 620.

[0090] Figure 77 is a block diagram of an example computing device 710 that may optionally be used to perform one or more aspects of the techniques described herein. The computing device 710 typically includes at least one processor 714 that communicates with a plurality of peripheral devices via a bus subsystem 712. These peripheral devices may include a storage subsystem 724 (including, for example, a memory subsystem 725 and a file storage subsystem 726), a user interface output device 720, a user interface input device 722, and a network interface subsystem 716. The input devices and output devices allow a user to interact with the computing device 710. The network interface subsystem 716 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0091] The user interface input device 722 may include a keyboard, a pointing device (such as a mouse, trackball, touchpad, or graphics tablet), a scanner, a touch screen incorporated into a display, an audio input device (such as a voice recognition system, a microphone), and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computing device 710 or onto a communication network.

[0092] The user interface output device 720 may include a display subsystem, a printer, a fax machine, or a non-visual display (such as an audio output device). The display subsystem may include a cathode ray tube (CRT), a flat panel device (such as a liquid crystal display (LCD)), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. Generally speaking, the use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 710 to a user or another machine or computing device.

[0093] Storage subsystem 724 stores programming and data structures that provide some or all of the functionality of the modules described herein. For example, storage subsystem 724 may include a Figure 2 Method 200, Figure 3 Method 300, Figure 4 Method 400 and / or Figure 5 The logic of selected aspects of method 500.

[0094] These software modules are typically executed by the processor 714 alone or in combination with other processors. The memory 725 used in the storage subsystem 724 may include multiple memories, including a main random access memory (RAM) 730 for storing instructions and data during program execution and a read-only memory (ROM) 732 in which fixed instructions are stored. The file storage subsystem 726 may provide persistent storage for program and data files, and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations may be stored by the file storage subsystem 726 in the storage subsystem 724 or in other machines accessible by the processor 714.

[0095] The bus subsystem 712 provides a mechanism for the various components and subsystems of the computing device 710 to communicate with each other as intended. Although the bus subsystem 712 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0096] The computing device 710 can be of different types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Figure 7 The depicted description of computing device 710 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computing device 710 are possible, and these configurations are similar to those of FIG. Figure 7 Computing devices are depicted as having more or fewer components than those shown.

[0097] Other implementations may include a non-transitory computer-readable storage medium storing instructions that are executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform a method, such as one or more of the methods described herein. Other implementations may include a system of one or more computers and / or one or more robots that include one or more processors that are operable to execute stored instructions to perform a method, such as one or more of the methods described herein.

[0098] In some implementations, a method is provided that includes identifying visual data instances captured throughout an environment of at least one robot using one or more visual components. The method further includes processing the visual data instances to identify regions of interest in the environment, and determining, for each of the regions of interest: an estimated location of the region of interest, and a region embedding for the region of interest in a natural language embedding space and semantically corresponding to visual features of the region of interest. The method further includes, for each of the regions of interest, storing an association of the estimated location of the region of interest with the region embedding for the region of interest. The method further includes identifying instructions for the robot to perform a task. The instructions are free-form natural language instructions generated based on user interface inputs provided by a user via one or more user interface input devices. The method further includes determining, based on the instructions, object descriptors that each describe a corresponding candidate environment object associated with the performance of the task. The method further includes comparing the object descriptor embeddings for the object descriptors with the region embeddings for the regions of interest to identify: a subset of object descriptors that each describe a corresponding object that may be present in the environment. The method further includes, in response to identifying a subset of object descriptors, processing the subset of object descriptors and the instructions using a large language model (LLM) to generate an LLM output. The LLM output may model a probability distribution over candidate word composition that depends on the object descriptor and depends on the instruction. The method further includes determining to implement the robot skill based on the LLM output and the skill description that is a natural language description of the robot skill that can be performed by the robot. The method further includes, in response to determining to implement the robot skill: causing the robot to implement the robot skill in the environment.

[0099] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0100] In some implementations, the natural language description of the robotic skill includes a skill action descriptor and a skill object descriptor (e.g., a skill object descriptor that conforms to one of the object descriptors in the subset). In some versions of those implementations, the method further includes (a) identifying a given region of interest based on comparing a skill object descriptor embedding for the skill object descriptor to a region embedding for a given region of interest in the regions of interest, and (b) in response to identifying the given region of interest, causing the robot to implement the robotic skill in the environment using an estimated position of the region of interest. In some variants of those versions, the robotic skill is a navigation skill, and causing the robot to implement the robotic skill in the environment using the estimated position of the region of interest includes causing the robot to navigate to a specific position determined based on the estimated position. In some additional or alternative variations of those versions, the method further includes: identifying the additional given regions of interest based on comparing the skill object descriptor embedding for the skill object descriptor with the region embedding for the additional given regions of interest in the regions of interest; determining that the region of interest and the additional regions of interest correspond to the same object based on the estimated position of the region of interest and the estimated position of the additional regions of interest; and in response to determining that the region of interest and the additional regions of interest correspond to the same object, using the estimated position of the region of interest and the estimated position of the additional regions of interest in causing the robot to implement the robotic skill in the environment. In some of those additional or alternative variations, the robotic skill is a navigation skill, and using the estimated position of the region of interest and the estimated position of the additional regions of interest in causing the robot to implement the robotic skill in the environment includes: (a) determining a specific location based on the estimated position of the region of interest, the estimated position of the additional regions of interest; and (b) causing the robot to navigate to the specific location. Further, in some of those additional or alternative variations, determining that the region of interest and the additional regions of interest correspond to the same object is further based on comparing a first size of the first region of interest with a second size of the second region of interest.

[0101] In some implementations, processing the visual data instances to identify regions of interest in an environment and determining an estimated location and a region embedding for each of the regions of interest includes, for a given one of the visual data instances: processing the given visual data instance using a class-agnostic object detection model to identify a given region of interest in the visual data instance; determining an estimated location for the given region of interest based on the given region of interest and a pose of a visual component when the given visual data instance was captured; and generating a region embedding for the given region of interest based on processing a portion of the given visual data instance corresponding to the given region of interest. Processing the portion may be using a visual language model (VLM) encoder trained to predict a natural language description of an image.

[0102] In some implementations, the method further includes, in response to determining to implement the robot skill: processing a subset of object descriptors, instructions, and skill descriptions of the robot skill using the LLM to generate additional LLM outputs, the additional LLM outputs modeling additional probability distributions on candidate word composition that depend on the object descriptors, instructions, and skill descriptions; determining to implement the additional robot skill based on the additional LLM outputs and the additional skill descriptions that are additional natural language descriptions of the additional robot skill that can be performed by the robot; and in response to determining to implement the additional robot skill: causing the robot to implement the additional robot skill in the environment and after implementing the robot skill in the environment. In some of those implementations, the method further includes, in response to determining to implement the additional robot skill: processing a subset of object descriptors, instructions, skill descriptions of the robot skill, and additional skill descriptions of the additional robot skill using the LLM to generate further LLM outputs, the further LLM outputs modeling additional probability distributions on candidate word composition that depend on the object descriptors, instructions, skill descriptions, and additional skill descriptions; and determining completion of performance of the task by the robot based on the further LLM outputs.

[0103] In some implementations, the method further includes, in response to determining to implement a robot skill: processing a subset of object descriptors, instructions, and a skill description of the robot skill using an LLM to generate additional LLM outputs that model additional probability distributions over candidate word compositions that depend on the object descriptors, instructions, and skill descriptions; and determining completion of performance of the task by the robot based on the additional LLM outputs.

[0104] In some implementations, the method further includes generating object descriptor embeddings. In some of those implementations, generating each of the object descriptor embeddings includes processing a corresponding object descriptor in the object descriptors using a text encoding model to generate a corresponding object descriptor embedding in the object descriptor embeddings.

[0105] In some implementations, the object descriptor includes one or more object descriptors that are not explicitly specified in the instruction. In some versions of those implementations, determining, based on the instruction, the object descriptors that each describe a corresponding candidate environment object associated with the execution of the task includes: processing the instruction using an LLM or an additional LLM to generate an alternating LLM output; and determining one or more object descriptors in the object descriptor based on the alternating LLM output. In some variants of those versions, the method further includes: determining a category descriptor for a category based on the alternating LLM output; and determining the given descriptor based on that a given descriptor in the object descriptor is a descriptor of a specific object that is a member of the category and based on that the category descriptor is determined based on the alternating LLM output. In some additional or alternative variants of those versions, the method further includes: identifying a category descriptor present in the instruction of the category; and determining the given descriptor based on that a given descriptor in the object descriptor is a descriptor of a specific object that is a member of the category and based on that the category descriptor is present in the instruction.

[0106] In some implementations, determining based on the LLM output and the skill description as a natural language description of the robot skill to achieve the robot skill includes: determining that the probability distribution of the LLM output indicates a skill description having a probability that satisfies a threshold probability degree, and the probability is greater than other probabilities determined for other candidate skill descriptions of other candidate robot skills that can be performed by the robot. In some versions of those implementations, the method further includes: selecting only the robot skill and the other candidate robot skills from a superset of skills that can be performed by the robot; and in response to the selection, determining the probabilities and other probabilities only for the robot skill and the other candidate robot skills. In some of those versions, selecting only the robot skill and the other candidate robot skills is based on comparing the skill descriptors and the other skill descriptors to a subset of the object descriptors and / or to region embeddings for the region of interest.

[0107] In some implementations, a method is provided that includes generating, based on processing visual data instances captured throughout an environment of one or more robots: regions of interest, and an estimated map location and a corresponding region embedding for each of the regions of interest. The method further includes receiving free-form (FF) natural language (NL) instructions provided via one or more user interface input devices and instructing the robot to perform a task. The method further includes determining, based on the FF NL instructions and the region embeddings for the regions of interest, object descriptors that each describe an object that is relevant to performing the task and that may be present in the environment. The method further includes utilizing the determined object descriptors in determining a robot skill implemented by at least one of the robots in performing the task.

[0108] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0109] In some implementations, the method further includes causing at least one of the robots to perform a robotic skill in the environment.

[0110] In some implementations, utilizing the determined object descriptors in determining the robotic skill implemented by at least one of the robots in performing the task includes utilizing the determined object descriptors in robotic planning based on a large language model (LLM). In some versions of those implementations, utilizing the determined object descriptors in determining the robotic skill implemented by at least one of the robots in performing the task includes generating an instance of an LLM output based on processing the determined object descriptors and FF NL instructions using the LLM. In some versions of those versions, utilizing the determined object descriptors in determining the robotic skill implemented by at least one of the robots in performing the task includes using an instance of an LLM output in determining the robotic skill implemented by the robot in performing the task in an environment.

[0111] In some implementations, the method further includes utilizing at least one of the determined map locations in implementing one or more of the determined robotic skills.

Claims

1. A method implemented by one or more processors, the method comprising: capturing, using one or more vision components, instances of visual data throughout an environment of at least one robot; The visual data instances are processed to identify regions of interest in the environment, and for each of the regions of interest, determine: the estimated location of the region of interest, and a region embedding for the region of interest in a natural language embedding space and corresponding semantically to visual features of the region of interest; for each of the regions of interest, storing an association of the estimated position of the region of interest with the region embedding for the region of interest; identifying instructions for the robot to perform a task, the instructions being free-form natural language instructions generated based on user interface input provided by a user via one or more user interface input devices; determining, based on the instructions, object descriptors that each describe a corresponding candidate environmental object relevant to performance of the task; Comparing an object descriptor embedding for the object descriptor with the region embedding for the region of interest to identify: a subset of said object descriptors that each describe a corresponding object that may be present in said environment; In response to identifying the subset of object descriptors, processing the subset of object descriptors and the instructions using a large language model (LLM) to generate an LLM output that models a probability distribution over candidate word compositions depending on the object descriptors and on the instructions; determining to implement the robot skill based on the LLM output and a skill description that is a natural language description of a robot skill that can be performed by the robot; as well as In response to determining to implement the robot skill: The robot is enabled to implement the robot skill in the environment.

2. The method of claim 1, wherein the natural language description of the robot skill comprises a skill action descriptor and a skill object descriptor.

3. The method according to claim 2, further comprising: identifying a given region of interest among the regions of interest based on comparing the skill object descriptor embedding for the skill object descriptor with the region embedding for the given region of interest; In response to identifying the given region of interest, the estimated position of the region of interest is used in causing the robot to perform the robotic skill in the environment.

4. The method of claim 3, wherein the robotic skill is a navigation skill, and wherein using the estimated position of the region of interest in causing the robot to implement the robotic skill in the environment comprises: The robot is caused to navigate to a specific location determined based on the estimated location.

5. The method according to claim 3, further comprising: identifying an additional given region of interest among the regions of interest based on comparing the skill object descriptor embedding for the skill object descriptor with the region embedding for the additional given region of interest; determining, based on the estimated position of the region of interest and the estimated position of the additional region of interest, that the region of interest and the additional region of interest correspond to the same object; as well as In response to determining that the region of interest and the additional region of interest correspond to the same object, the estimated position of the region of interest and the estimated position of the additional region of interest are used in causing the robot to perform the robotic skill in the environment.

6. The method of claim 5, wherein the robotic skill is a navigation skill, and wherein using the estimated position of the region of interest and the estimated position of the additional region of interest in causing the robot to implement the robotic skill in the environment comprises: determining a specific location based on the estimated location of the region of interest and the estimated location of the additional region of interest; as well as The robot is caused to navigate to the specific location. 7 . The method of claim 5 , wherein determining that the region of interest and the additional region of interest correspond to the same object is further based on comparing a first size of a first region of interest to a second size of a second region of interest.

8. The method of claim 3, wherein the skill object descriptor corresponds to one of the object descriptors of the subset.

9. The method of claim 1 , wherein processing the visual data instances to identify the regions of interest in the environment and determining the estimated location and the region embedding for each of the regions of interest comprises: For a given visual data instance in the visual data instances: processing the given visual data instance using a class-agnostic object detection model to identify a given region of interest in the visual data instance; determining the estimated position for the given region of interest based on the given region of interest and a pose of the visual component when the given visual data instance was captured; as well as The region embedding for the given region of interest is generated based on processing a portion of the given visual data instance corresponding to the given region of interest, wherein processing the portion is using a visual language model (VLM) encoder trained to predict natural language descriptions of images.

10. The method of claim 1, further comprising, in response to determining that the robot skill is achieved: processing the subset of object descriptors, the instructions, and the skill description of the robot skill using the LLM to generate additional LLM outputs that model additional probability distributions over the candidate word compositions that depend on the object descriptors, the instructions, and the skill description; determining to implement the additional robot skills based on the additional LLM output and an additional skill description that is an additional natural language description of an additional robot skill that can be performed by the robot; as well as In response to determining to implement the additional robotic skill: The robot is caused to achieve the additional robot skill in the environment and after achieving the robot skill in the environment.

11. The method of claim 10, further comprising, in response to determining to implement the additional robotic skill: processing the subset of object descriptors, the instructions, the skill descriptions of the robot skills, and the additional skill descriptions of the additional robot skills using the LLM to generate a further LLM output that models additional probability distributions over the candidate word compositions that depend on the object descriptors, the instructions, the skill descriptions, and the additional skill descriptions; and Completion of performance of the task by the robot is determined based on the further LLM output.

12. The method of claim 1 , further comprising, in response to determining that the robot skill is achieved: processing the subset of object descriptors, the instructions, and the skill description of the robot skill using the LLM to generate additional LLM outputs that model additional probability distributions over the candidate word compositions that depend on the object descriptors, the instructions, and the skill description; and Completion of performance of the task by the robot is determined based on the additional LLM output.

13. The method of claim 1, further comprising generating the object descriptor embedding.

14. The method of claim 13, wherein generating each of the object descriptor embeddings comprises: A corresponding object descriptor in the object descriptors is processed using a text encoding model to generate a corresponding object descriptor embedding in the object descriptor embeddings.

15. The method of claim 1, wherein the object descriptors include one or more object descriptors not explicitly specified in the instruction.

16. The method of claim 15, wherein determining, based on the instructions, object descriptors each describing a corresponding candidate environmental object associated with performance of the task comprises: processing the instruction using the LLM or an additional LLM to generate an alternate LLM output; as well as One or more of the object descriptors are determined based on the alternating LLM output.

17. The method according to claim 16, further comprising: determining a class descriptor for a class based on the alternating LLM output; as well as The given one of the object descriptors is determined based on it being a descriptor of a particular object that is a member of the class and based on the class descriptor being determined based on the alternating LLM output.

18. The method according to claim 16, further comprising: a class descriptor present in the instruction identifying the class; as well as The given descriptor is determined based on the given one of the object descriptors being a descriptor of a particular object that is a member of the class and based on the class descriptor being present in the instruction.

19. The method of claim 1, wherein determining to achieve the robot skill based on the LLM output and the skill description as the natural language description of the robot skill comprises: Determining that the probability distribution of the LLM output indicates the skill description having a probability of satisfying a threshold degree of probability, and the probability is greater than other probabilities determined for other candidate skill descriptions of other candidate robot skills that can be performed by the robot.

20. The method according to claim 19, further comprising: selecting only the robot skill and the other candidate robot skills from a superset of skills that can be performed by the robot; as well as In response to the selection, the probability and the other probabilities are determined only for the robot skill and the other candidate robot skills.

21. The method of claim 20, wherein selecting only the robot skill and the other candidate robot skills is based on comparing the skill descriptor and the other skill descriptors with the subset of object descriptors and / or with the region embedding for the region of interest.

22. A method comprising: generating, based on processing instances of visual data captured throughout an environment of the one or more robots: regions of interest, and an estimated map location and a corresponding region embedding for each of the regions of interest; receiving free-form (FF) natural language (NL) instructions provided via one or more user interface input devices and instructing the robot to perform a task; determining, based on the FF NL instructions and the region embedding for the region of interest, object descriptors each describing an object that is relevant to performing the task and that may be present in the environment; as well as The determined object descriptors are utilized in determining a robot skill implemented by at least one of the robots in performing the task.

23. The method according to claim 22, further comprising: The at least one of the robots is enabled to perform the robot skill in the environment.

24. The method of claim 22, wherein utilizing the determined object descriptor in determining the robot skill implemented by at least one of the robots in performing the task comprises: The identified object descriptors are utilized in large language model (LLM) based robot planning.

25. The method of claim 24, wherein utilizing the determined object descriptors in determining the robot skill implemented by at least one of the robots in performing the task further comprises: An instance of an LLM output is generated based on the object descriptor determined using the LLM process and the FF NL instruction.

26. The method of claim 24, wherein utilizing the determined object descriptors in determining the robot skill implemented by at least one of the robots in performing the task further comprises: The instance of using the LLM output in determining the robot skill achieved by the robot in performing the task in the environment.

27. The method of claim 22, further comprising: At least one of the determined map locations is utilized in implementing one or more of the determined robotic skills.

28. A method implemented by one or more processors, the method comprising: capturing, using one or more vision components, instances of visual data throughout an environment of at least one robot; For each of the visual data instances: processing the visual data instance using a class-agnostic object detection model to identify any regions of interest in the visual data instance; For each region of interest in a plurality of regions of interest identified from the visual data instance: determining an estimated location of the region of interest; generating a region embedding for the region of interest based on processing visual data from a corresponding visual data instance of the visual data instance and corresponding to the region of interest, wherein generating the region embedding comprises processing the visual data using a visual language model encoder trained to predict a natural language description of an image; as well as generating an entry associating the estimated position with the region embedding; After generating said entry: identifying instructions for the robot to perform a task, the instructions being free-form natural language instructions generated based on user interface input provided by a user via one or more user interface input devices; determining, based on the instructions, one or more object descriptors that each describe a corresponding object associated with performance of the task; processing each of the one or more object descriptors using a text encoding model to generate a corresponding object descriptor embedding; comparing the object descriptor embeddings to the region embeddings of the entries to identify a subset of the object descriptor embeddings that each corresponds to at least one of the entries; In response to identifying the subset of object descriptors, processing the subset of object descriptors and the instructions using a large language model (LLM) to generate an LLM output that models a probability distribution over candidate word compositions depending on the object descriptors and on the instructions; making a determination based on the LLM output and a skill description that is a natural language description of a robot skill that can be performed by the robot to implement the robot skill; as well as In response to determining to implement the robotic skill: The robot is enabled to implement the robot skill in the environment.

29. A system comprising a memory storing instructions and one or more processors operable to execute the instructions to perform the method of any preceding claim.

30. The system of claim 29, wherein the system comprises one or more robots.

31. One or more computer readable media storing instructions which, when executed by one or more processors, perform the method of any one of claims 1 to 28.

Images