Training a machine learning model for use in robotic control

Abstract

Some implementations relate to capturing a vision data instance; processing the vision data instance to generate at least one object descriptor that describes at least one respective object within a scene represented by the vision data instance; causing a language model (LM) to process a first LM prompt that has been generated based on the at least one object descriptor and a selected training policy descriptor to generate a first LM output; and causing the LM or a different LM to process a second LM prompt that is based on the first LM output to generate a second LM output. Based on second LM output a training policy, corresponding to a selected training policy descriptor, is used by a robot in attempting to perform a task, and corresponding training data is acquired and utilized.

Description

TRAINING A MACHINE LEARNING ODEL FOR USE IN ROBOTIC CONTROLBackground

[0001] Many robots are programmed to perform certain tasks. For example, a robot on an assembly line can be programmed to recognize certain objects and perform particular manipulations to those certain objects. Further, some robots can perform certain tasks in response to explicit user interface input that corresponds to the certain task. For example, a vacuuming robot can perform a general vacuuming task in response to a spoken utterance of "robot, clean".

[0002] A machine learning model may be used for controlling the robot to perform the various tasks, such as manipulation tasks involving one or more objects near the robot. The machine learning model may be trained over time by allowing the robot to perform tasks under human supervision, collecting data from those tasks, and training the machine learning model using the collected data. Through training, the robot (via the machine learning model) may gain the ability to perform new tasks and / or may become better at performing previously- learned tasks successfully.

[0003] Training a machine learning model for robotic control can require a large quantity of data to be acquired, therefore a large number of robots could be used simultaneously to perform the training tasks. This may speed up the rate at which training data is acquired and the machine learning model is trained. However, the requirement of human involvement in supervising the robots during the training can be a constraint on the collection of training data. One human may be able to safely supervise only a small number of robots, such as two robots, at any given time. As the number of robots simultaneously training the machine learning model is scaled up, the number of humans supervising the robots will often scale up in proportion. As a result, a large number of humans may be required to supervise the robots, which may result in the training being labor-intensive, costly and inefficient.

[0004] Furthermore, in some cases the robots may be trained in a supervised training environment of a predetermined nature. That is, the robots may be trained in an environment only containing objects that have been carefully pre-selected by a human supervisor. The human supervisor may have provided the robots with information regarding the pre-selected objects and / or tasks that the human supervisor wants the robots to complete. Furthermore,the robots may be at fixed locations within the environment. As such, the tasks performed by the robots may be simple in nature, with little variety, which in turn may limit the training of the machine learning model. Outside of the carefully managed training environment (i.e. 'in the wild'), the robots may find difficulty in successfully performing tasks to train the machine learning model, particularly where the robots encounter unfamiliar scenes and / or objects.Summary

[0005] Language models (LMs) have been developed that can be used to process natural language (NL) content and / or other input(s), to generate LM output that that reflects NL content and / or other content that is responsive to the input(s). For example, large language models (LLMs) have been developed that are trained on massive amounts of data and are able to be utilized to robustly process a wide range of NL inputs and generate corresponding LM output that reflects corresponding NL content that is accurate and responsive to the NL input. An LLM can include at least hundreds of millions of parameters and can often include at least billions of parameters, such as one hundred billion or more parameters. An LLM can, for example, be a sequence-to-sequence model, Transformer-based, and / or include an encoder and / or a decoder. One non-limiting example of an LLM is GOOGLE'S Pathways Language Model (PaLM). Another non-limiting example of an LLM is GOOGLE'S Language Model for Dialogue Applications (LaMDA).

[0006] Separately, efforts have been made in attempting to enable robust free-form (FF) NL control of a robot. For example, to enable a robot to react appropriately in response to any one of a variety of different typed or spoken instructions from a human that are directed to the robot. For instance, in response to a FF NL instruction of "put the blocks in the toy bin", to be able to perform a robotic task that includes (a) navigating to the "blocks", (b) picking up the "blocks", (c) navigating to the "toy bin", and (d) placing the "blocks" in the "toy bin".

[0007] Implementations disclosed herein recognize that LMs, such as LLMs, can encode a wealth of semantic knowledge about the world, and that such knowledge can be useful to robots in acting upon high-level, temporally extended instructions expressed in a FF NL instruction. Implementations disclosed herein further recognize that a drawback of LLMs is that they lack real-world experience, such as real-world experience in robotic control, and / or that they lack any grounding to current real-world condition(s). For example, prompting anLLM with "I spilled my drink on the table, can you help" can result in generation of LLM output that reflects NL content that is descriptive of reasonable step(s) for cleaning a spill. For instance, the highest probability decoding of the LLM output can reflect NL content of "you could try using a vacuum". However, "you could try using a vacuum" may not be applicable to a particular agent, such as a robot, that needs to perform this task in a particular environment. For example, the robot may not have an integrated vacuum and may not have access to any separate vacuum in the particular environment, or may be incapable of controlling a separate vacuum in the particular environment.

[0008] Where an LLM output describes a task that is applicable to a particular robot, the LLM output may be provided to one or more downstream machine learning models that have been trained to translate the particular robotic task described in the LLM output into instructions for controlling the robot to perform the task. For example, the machine learning model(s) may take the LLM output as an input and provide as output one or movement parameters that are provided to one or more actuators of the robot to cause it to move and perform the task.

[0009] Efforts have been made to improve the downstream machine learning model(s) by acquiring training data as the robot performs various tasks, with the training data used to train the downstream machine learning model(s). Through training, the machine learning model(s) may learn to perform new tasks and / or may become better able to convert various received LLM outputs into appropriate movement parameters for controlling the robot to perform tasks with a higher rate of success.

[0010] A large number of robots may be used simultaneously in a given environment to acquire training data for training the machine learning model(s), thereby increasing the rate at which training data is acquired and accelerating the training of the machine learning model(s). However, the robots will often need to be supervised by one or more humans, either remotely or in close proximity to the environment of the robots. This is to ensure the robots are operating safely and as expected. In some cases the humans may also be used to teleoperate (i.e. remotely control) one or more of the robots to perform a particular task so that training data may be acquired for that teleoperated task and used to train the machine learning model(s). By teleoperating the robot, the human may be able to 'teach' the machine learningmodel(s) how to perform a new task, how to perform a previously-learned task in a different manner, or how to perform a previously-learned task more successfully. However, human supervision of the robots can be labor-intensive, costly and inefficient. As the number of robots is scaled up, so is the number of human supervisors. In scenarios where the robots require human guidance to perform a task, for example through teleoperation, the robots may be idle while soliciting and / or awaiting the human guidance, which results in inefficient operation of the robots.

[0011] In view of these and / or other considerations, some implementations disclosed herein are directed to capturing a vision data instance in an environment of at least one robot. The vision data instance may be captured using one or more vision components of the at least one robot, or may be captured by one or more vision components that are separate from the robot but which are able to transmit the vision instance data to the robot or another part of the system disclosed herein. The vision data instance is processed to generate at least one object descriptor, wherein each at least one object descriptor describes at least one respective object within a scene represented by the vision data instance. As an example, the vision data instance may be processed by a visual language model (VLM) to generate the object descriptor(s), wherein the VLM receives the vision data instance as an input and provides the object descriptor(s) as an output. The object descriptor(s) may be a natural language descriptor such as a natural language text string. Optionally, the vision data instance may also be processed to generate at least one scene descriptor, wherein each at least one scene descriptor describes the scene represented by the vision data instance. As an example, the vision data instance may be processed by a VLM (such as the same VLM that generated the object descriptor(s), or a different VLM) to generate the scene descriptor(s), wherein the VLM receives the vision data instance as an input and provides the scene descriptor(s) as an output. The scene descriptor(s) may be a natural language descriptor such as a natural language text string. A language model (LM) is caused to process a first LM prompt that has been generated based on the object descriptor(s), a selected training policy descriptor, and optionally the scene descriptor(s), to generate a first LM output. The selected training policy descriptor has been selected from a plurality of training policy descriptors, with each training policy descriptor corresponding to a respective training policy from among a plurality of training policies. The LMcould be an LLM, for example. The first LM output comprises a plurality of task descriptors, each task descriptor corresponding to a respective task for the at least one robot to perform. Further, an LM is caused to process a second LM prompt that is based on the plurality of training policy descriptors and the plurality of task descriptors to generate a second LM output. The LM that is caused to process the second LM prompt may be the same LM used to process the first LM prompt, or may be a different LM. The LM caused to process the second LM prompt may be an LLM in some examples. The second LM output classifies each task descriptor as corresponding to at least one respective training policy descriptor of the plurality of training policy descriptors. A task descriptor from a subset of the plurality of task descriptors that have been classified as corresponding to the selected training policy descriptor is selected. Based on the training policy corresponding to the selected training policy descriptor, the robot is caused to attempt to perform the task corresponding to the selected task descriptor, in the environment of the robot. Training data is acquired as the robot attempts to perform the task corresponding to the selected task descriptor and the training data may be recorded. The training data may be any data acquired during performance of the task by the robot that may be used to train a machine learning model, for example video data of the robot performing the task, movement parameters used to control the robot to perform the task, a measure of how successfully the robot performed the task etc. The training data may be acquired using one or more of the sensors 742a-742m discussed in relation to FIG. 7, for example. The ML model is caused to be trained using the acquired training data.

[0012] Some implementations disclosed herein relate to selecting a training policy descriptor from a plurality of training policy descriptors based on selection data, wherein each training policy descriptor corresponds to a respective training policy for training a machine learning (ML) model. The plurality of training policy descriptors may comprise at least one training policy descriptor corresponding to a teleoperation training policy and at least one training policy descriptor corresponding to a non-teleoperation training policy. The teleoperation training policy may be a policy for controlling the robot to acquire training data using human involvement, with the human teleoperating the robot to perform the task and acquire training data. The non-teleoperation training policy may be an autonomous policy for controlling the robot to acquire training data during a task without human involvement, forexample without a human teleoperating the robot to perform the task. As examples, the non- teleoperation policy may comprise a scripted training policy or a learned training policy. The selection data is indicative of whether a human should be involved with training the machine learning (ML) model. The selection data is indicative of which training policy should be used to perform a task. In these implementations, a first LM prompt is generated based on the selected training policy descriptor and at least one object descriptor. Optionally, the first LM prompt may be further generated based on at least one scene descriptor. The object descriptor(s) and / or scene descriptor(s) may have been generated using any of the methods described herein. An LM is caused to process the first LM prompt to generate a first LM output comprising a plurality of task descriptors, each task descriptor corresponding to a respective task for the at least one robot to perform. The LM could be an LLM, for example. A task descriptor is selected from the plurality of task descriptors. Based on the training policy corresponding to the selected training policy descriptor, the robot is caused to attempt to perform the task corresponding to the selected task descriptor, in the environment of the robot. Training data is acquired as the robot attempts to perform the task and the training data may be recorded. The ML model is caused to be trained using the training data.

[0013] Some implementations disclosed herein relate to causing an LM to process an LM prompt to generate a first LM output comprising a plurality of task descriptors, each task descriptor corresponding to a respective task for at least one robot to perform. In these implementations, the first LM prompt has been generated based on at least one object descriptor and a selected training policy descriptor. Each object descriptor describes at least one respective object within a scene represented by a vision data instance captured using one or more vision components in an environment of the at least one robot. The selected training policy descriptor has been selected from a plurality of training policy descriptors, each training policy descriptor corresponding to a respective training policy of a plurality of training policies. In some examples, the first LM prompt has been further generated based on at least one scene descriptor, wherein each scene descriptor describes the scene represented by the vision data instance. In these implementations, a second LM prompt is generated that is based on the plurality of training policy descriptors and the plurality of task descriptors. The LM or a different LM is caused to process the second LM prompt to generate a second LM output thatclassifies each task descriptor as corresponding to at least one respective training policy descriptor of the plurality of training policy descriptors. A subset of the plurality of task descriptors is determined, the subset of the plurality of task descriptors comprising task descriptors classified as corresponding to the selected training policy descriptor. A task descriptor is selected from the subset of the plurality of task descriptors. Based on the training policy corresponding to the selected training policy descriptor, the robot is caused to attempt to perform the task corresponding to the selected task descriptor. Training data is acquired as the robot attempts to perform the task and the training data may be recorded. An ML model is caused to be trained using the training data.

[0014] Some implementations disclosed herein relate to causing a language model (LM) to process a first LM prompt to generate a first LM output comprising a plurality of task descriptors, each task descriptor corresponding to a respective task for at least one robot to perform, wherein the first LM prompt has been generated based on at least one object descriptor and a selected training policy descriptor. Each object descriptor describes at least one respective object within a scene represented by a vision data instance captured using one or more vision components in an environment of the at least one robot. The selected training policy descriptor has been selected from a plurality of training policy descriptors, each training policy descriptor corresponding to a respective training policy of a plurality of training policies. In some examples, the first LM prompt has been further generated based on at least one scene descriptor, wherein each scene descriptor describes the scene represented by the vision data instance. In these implementations, a task descriptor is selected from the plurality of task descriptors. Based on the training policy corresponding to the selected training policy descriptor, the robot is caused to attempt to perform the task corresponding to the selected task descriptor in the environment. Training data is recorded as the robot attempts to perform the task and the training data may be recorded. An ML model is caused to be trained using the training data. At least one diversity measure corresponding to the task being performed by the robot is determined and the at least one diversity measure is recorded.

[0015] The above description is provided as an overview of only some implementations disclosed herein. These and other implementations are described in more detail herein, including in the detailed description and the claims.

[0016] It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.Brief Description of the Drawings

[0017] FIG. 1A illustrates an example of a human providing a free-form (FF) natural language (NL) instruction to an example robot.

[0018] FIG. IB illustrates a simplified birds-eye view of an example environment in which the human and the robot of FIG. 1A are located, and illustrates an example vision data instance captured in the environment.

[0019] FIG. 2 illustrates an example environment in which implementations described herein can be implemented, showing an example of how components may interact in accordance with various implementations described herein.

[0020] FIG. 3 is a flowchart illustrating a first example method of training a machine learning model for controlling a robot, according to implementations disclosed herein.

[0021] FIG. 4 is a flowchart illustrating a second example method of training a machine learning model for controlling a robot, according to implementations disclosed herein.

[0022] FIG. 5 is a flowchart illustrating a third example method of training a machine learning model for controlling a robot, according to implementations disclosed herein.

[0023] FIG. 6 is a flowchart illustrating a fourth example method of training a machine learning model for controlling a robot, according to implementations disclosed herein.

[0024] FIG. 7 schematically depicts an example architecture of a robot.

[0025] FIG. 8 schematically depicts an example architecture of a computer system.Detailed Description

[0026] Some implementations disclosed herein use a robot to generate training data that can be used train a machine learning model for controlling the robot, or for controlling a different robot.

[0027] Turning now to the Figures, FIG. 1A illustrates an example of a human 101 providing a free-form (FF) natural language (NL) instruction 105 of "get the fruit ready to wash" to anexample robot 110.

[0028] The robot 110 illustrated in FIG. 1A is a particular mobile robot. However, additional and / or alternative robots can be utilized with techniques disclosed herein, such as additional robots that vary in one or more respects from robot 110 illustrated in FIG. 1A. For example, a mobile forklift robot, an unmanned aerial vehicle ("UAV"), a non-mobile robot, and / or a humanoid robot can be utilized instead of or in addition to robot 110, in techniques described herein.

[0029] Robot 110 includes a base 113 with wheels provided on opposed sides thereof for locomotion of the robot 110. The base 113 may include, for example, one or more motors for driving the wheels of the robot 110 to achieve a desired direction, velocity, and / or acceleration of movement for the robot 110. The robot 110 also includes robot arm 114 with an end effector 115 that takes the form of a gripper with two opposing "fingers" or "digits."

[0030] Robot 110 also includes a vision component 111 that can generate vision data (e.g., images) related to shape, color, depth, and / or other features of object(s) that are in the line of sight of the vision component 111. The vision component 111 can be, for example, a monocular camera, a stereographic camera (active or passive), and / or a 3D laser scanner. A 3D laser scanner can include one or more lasers that emit light and one or more sensors that collect data related to reflections of the emitted light. The 3D laser scanner can generate vision component data that is a 3D point cloud with each of the points of the 3D point cloud defining a position of a point of a surface in 3D space. A monocular camera can include a single sensor (e.g., a charge-coupled device (CCD)), and generate, based on physical properties sensed by the sensor, images that each includes a plurality of data points defining color values and / or grayscale values. For instance, the monocular camera can generate images that include red, blue, and / or green channels. Each channel can define a value for each of a plurality of pixels of the image such as a value from 0 to 255 for each of the pixels of the image. A stereographic camera can include two or more sensors, each at a different vantage point. In some of those implementations, the stereographic camera generates, based on characteristics sensed by the two sensors, images that each includes a plurality of data points defining depth values and color values and / or grayscale values. For example, the stereographic camera can generate images that include a depth channel and red, blue, and / or green channels.

[0031] Robot 110 also includes one or more processors that, for example, can perform one or more methods, or parts of methods, described herein. For example, one or more processors of robot 110 can implement all or aspects of method 300, 400, 500, and / or 600 described herein, and / or any method described in relation to FIG. 2. Additional descriptions of some examples of the structure and functionality of various robots is provided herein.

[0032] Turning now to FIG. IB, a simplified birds-eye view of an example environment, in which the human 101 and the robot 110 of FIG. 1A are located, is illustrated. The human 101 and the robot 110 are represented as circles in FIG. IB. Further, environmental features 191, 192, 193, and 194 are illustrated in FIG. IB. The environmental features 191, 192, 193, and 194 illustrate outlines of landmarks in the environment. For example, the environment could be an office kitchen or a work kitchen, and features 191 and 192 can be countertops, feature 193 can be a kitchen island, and feature 194 can be a round table.

[0033] Also illustrated in FIG. IB is an example vision data instance 180 that was captured in the environment using vision component 111. In some examples, the vision data instance 180 may have been captured during a previous exploration of the environment of FIG. IB by the robot 110. The vision data instance 180 represents a scene of the environment that has been viewed by the vision component 111. The vision data instance 180 captures objects within the scene. In this example, the objects comprise a pear and keys that are both present on the round table represented by feature 194. It is noted that, in the birds-eye view, the pear, the keys, and other objects of the environment are not illustrated for the sake of simplicity.

[0034] A region of interest 184A, of the vision data instance 180, is also illustrated and encompasses the pear in the vision data instance. A region of interest 184B, of the vision data instance 180, is also illustrated and encompasses the keys in the vision data instance 180. The vision data instance 180 can be processed, using a class-agnostic object detection model, to identify the regions of interest 184A and 184B. Further, the vision data, of the vision data instance that corresponds to the region of interest 184A, can be processed using a visual language model (VLM) encoder to generate a region embedding for the region of interest 184A. An estimated map location, represented by the circle at the end of the line connecting the region of interest 184A to the round table represented by feature 194, can also be determined for the region of interest 184A. The region embedding for the region of interest184A can be stored in association with the estimated map location for the region of interest 184A and optionally in association with a size of the region of interest.

[0035] Yet further, the vision data, of the vision data instance that corresponds to the region of interest 184B, can be processed using the VLM encoder to generate a region embedding for the region of interest 184B. An estimated map location, represented by the circle at the end of the line connecting the region of interest 184B to the round table represented by feature 194, can also be determined for the region of interest 184B. The region embedding for the region of interest 184B can be stored in association with the estimated map location for the region of interest 184B and optionally in association with the size of the region of interest.

[0036] Only a single vision data instance 180 is illustrated in FIG. IB for sake of simplicity. However, it is noted that many additional vision data instances may have been previously captured in the environment, and regions of interest and corresponding region embeddings and estimated map locations similarly determined for those vision data instances, and associated data stored for utilization in techniques disclosed herein.

[0037] FIG. 2 illustrates an example environment in which implementations described herein can be implemented. FIG. 2 shows a robot 110, which could in some examples be the robot 110 described previously in relation to FIG. 1A. The robot 110 is located in an environment and has a vision component 111 for capturing one or more vision data instances 180 of a scene in the environment. The robot 110 may in some examples also include a base 113 and a robot arm 114 with an end effector 115, as described previously in relation to FIG. 1A. FIG. 2 further shows an LLM engine 200, an LLM 220, an affordance engine 230, another LLM 240 which in some instances may be the same as the first LLM 220, a task selection engine 250, an implementation engine 260, a training data engine 270, a training engine 280, a training policy selection engine 208, training policies 210, and one or more machine learning models 290. One or more of the illustrated components can be implemented by the robot 110 (e.g., utilizing processor(s) and / or memory thereof) and / or utilizing remote computing device(s) (e.g., cloud-based server(s)) that are in network communication with the robot 110. The system, or aspects thereof, shown in the environment of FIG. 2 may be used to implement all or aspects of the methods described herein, such as method 300, 400, 500, and / or 600. Itshould be noted that where reference in this disclosure to a 'large language model', the reference may equally be applied to a 'language model'. For example, the LLM engine 200 shown in FIG. 2 may instead be referred to as an LM engine and / or the first LLM prompt 202 may be a first LM prompt.

[0038] In FIG. 2, the LLM engine 200 generates a first LLM prompt 202 based on at least one or more object descriptors 204 and a selected training policy descriptor 206a. The LLM engine 200 can generate the first LLM prompt 202 as a natural language LLM prompt 202. The first LLM prompt 202 is for causing the LLM 220 to output a plurality of task descriptors, each task descriptor corresponding to a respective task for the robot 110 to perform.

[0039] The object descriptor(s) 204 can include NL descriptor(s) of object(s) currently or recently detected in the environment with the robot 110, such as descriptor(s) of object(s) determined based on processing image(s) or other vision data using object detection and classification machine learning model(s). As an example, the object descriptor(s) may be generated using a suitable machine learning model(s) such as a visual language model (VLM) that may receive a visual data instance 180 captured by the vision component 111 of the robot 110 as input and output the object descriptor(s) 204 corresponding to object(s) in the scene captured in the visual data instance 180. The VLM may form part of the system shown in FIG. 2 or could be external to the system but in communication with the system. As non-limiting examples, the object descriptor(s) 204 can include "soap", "napkin", "snack", "cloth" and "sponge" and the LLM engine 200 can generate the first LLM prompt 202 to incorporate one or more of such object descriptors 204. For instance, the first LLM prompt 202 can include "Soap, a napkin, a snack, a cloth and a sponge are nearby". The object descriptor(s) can be prepended or otherwise incorporated into first LLM prompt 202. By incorporating the object descriptor(s) into the first LLM prompt 202, the LLM 220 can be encouraged to suggest tasks in its output that involve manipulation of one or more of the described objects.

[0040] The selected training policy descriptor 206a has been selected by a training policy selection engine 208 from among a plurality of training policy descriptors 206. Each training policy descriptor 206 describes the capabilities of a corresponding training policy from among a plurality of training policies 210. The training policies 210 may include a teleoperation training policy and at least one non-teleoperation training policy such as a scripted policy or a learnedpolicy. The teleoperation training policy may be a policy for controlling the robot to acquire training data using human involvement, with a human teleoperating the robot to perform a task and acquire training data. The non-teleoperation training policy may be an autonomous policy for controlling the robot to acquire training data during a task without human involvement, for example without a human teleoperating the robot to perform the task.

[0041] The training policy selection engine 208 may select the selected training policy descriptor 206a based on selection data 212. The selection data may be indicative of whether a human should be involved with training the machine learning (ML) model (i.e. whether a training policy descriptor corresponding to a teleoperation training policy should be selected over training policy descriptors corresponding to a non-teleoperation training policies).

[0042] In some examples, the selection data indicates, for each training policy descriptor of the plurality of training policy descriptors, a respective probability for selecting that training policy descriptor. As an example, there may be a scenario in which one human is in the environment supervising five robots. Furthermore, the training policies may comprise a teleoperation training policy and a non-teleoperation policy. The human may only be able to teleoperate one of the five robots at any given time. The teleoperation training policy (or the training policy description corresponding to the teleoperation training policy) may therefore have been assigned a probability of 0.2 (i.e. 1 in 5) of being selected from among all the training policies. Any other training policies (i.e. non-teleoperation training policies) will be assigned probabilities of being selected that together total 0.8 (i.e. 4 in 5). As such, the likelihood of a teleoperation training policy being selected is comparatively small and so the likelihood of a human supervisor being required to teleoperate more than one of the five robots at any given time is reduce. By tailoring the selection data (e.g. probabilities) to a given situation (e.g. the number of human supervisors per robot in an environment), training using the robots may be made more efficient. In some examples, the selection data may indicate the availability of a human to supervise the robot. As an example, the availability of a human to supervise the robot may be determined using one or more sensors in the environment of the robot, for example one or more cameras. In some examples, the selection data indicates a ratio of humans to robots in the environment.

[0043] In some implementations, the LLM engine 200 can optionally generate the first LLMprompt 202 further based on one or more scene descriptor(s) 214, which describe the scene of a current environment of the robot 110 as represented by the vision data instance 180. The scene descriptor(s) 214 may be a NL descriptor(s) of the scene and may be generated using a suitable machine learning model(s), such as a VLM, for example the same VLM used to generate the object descriptor(s) 204. A non-limiting example of a NL scene descriptor 214 can be "You are in a small kitchen with low lighting and objects scattered on a table". The scene descriptor(s) can be prepended or otherwise incorporated into first LLM prompt 202. By incorporating the scene descriptor(s) into the first LLM prompt 202, the LLM 220 can be encouraged to suggest tasks in its output that are suitable for the environment in which the robot 110 is located.

[0044] The first LLM prompt 202 includes an NL request for the LLM 220 to generate tasks for the robot 110 to perform. As an example, the request may be "You are a robot. Generate a list of tasks you can perform using the following objects". In some examples, the request may be a request for a specific number of tasks.

[0045] In some implementations, the LLM engine 200 can optionally generate the first LLM prompt 202 further based on one or more additional rules 216. The one or more additional rules 216 can be prepended or otherwise incorporated into first LLM prompt 202. As an example, the one or more additional rules 216 may comprise one or more safety rules that indicate which types of task would be considered safe and / or unsafe for the robot to perform. By incorporating the safety rules into the first LLM prompt 202, the LLM 220 can be encouraged to suggest tasks in its output that are considered safe according to the safety rules. The safety rules may comprise one or more NL text strings to be prepended or otherwise incorporated into first LLM prompt 202. As a non-limiting example, a safety rule could be "This robot shall not attempt tasks involving humans, animals or living things. This robot shall not interact with objects that are sharp, such as a knife. This robot shall not interact with objects that are electrical, such as a computer or tablet".

[0046] Additionally or alternatively, as a further example, the one or more additional rules 216 may comprise one or more capability rules that indicate capabilities of the robot 110 that will be performing the task. The capability rule(s) may comprise one or more NL text strings to be prepended or otherwise incorporated into first LLM prompt 202. As a non-limiting example,a capability rule could be "This robot shall not attempt to lift objects that are heavier than a book". As another non-limiting example, a capability rule could be "This robot only has one arm, and thus cannot perform tasks requiring two arms". By incorporating the capability rule(s) into the first LLM prompt 202, the LLM 220 can be encouraged to suggest tasks in its output for which the robot has the physical capabilities to perform.

[0047] Additionally or alternatively, as a further example, the one or more additional rules 216 may comprise one or more prompt examples such as a manually engineered example(s) of desired output style(s). The prompt example(s) may comprise one or more NL text strings to be prepended or otherwise incorporated into first LLM prompt 202. For example, the first LLM prompt 202 can include a prompt example of "in a numbered list format" or "in the style of 1. first task, 2. second task, 3. third task, etc.". The prompt example(s) can be prepended or otherwise incorporated into first LLM prompt 202 to encourage prediction, by the LLM 220, of content that is in the desired output style(s).

[0048] As a non-limiting example, the first LLM prompt 202 can be "You are a robot. You have only one arm, and thus cannot perform tasks requiring two arms. You must not interact with objects that are sharp, such as a knife. You are operating in a small kitchen with low lighting and objects scattered on a table. The objects include soap, a napkin, a snack, a cloth and a sponge. Generate a list of 7 tasks you could perform using the objects, in the style of 1. first task, 2. second task, 3. third task, etc. You may only suggest pick tasks. A pick task is a task that has two components only: a verb, here pick, and an object. The task starts with "pick" immediately followed by an object to pick up, for example "pick orange"".

[0049] The LLM engine 200 processes the generated first LLM prompt 202, using the LLM 220, to generate a first LLM output 222. In some examples the LLM 220 forms part of the same system as the other components shown in FIG. 2 such as the LLM engine 200. However, in some examples the LLM 220 is instead external to the system, for example hosted by one or more remote, external servers. The LLM engine 200 may cause the LLM 220 to process the first LLM prompt 202 by providing the first LLM prompt 202 as input to the LLM 220. Responsive to receiving the first LLM prompt 202 as an input, the LLM 220 processes the first LLM prompt 202 to provide a first LLM output 222, which is received by the system.

[0050] The first LM output 222 is a NL output that includes a plurality of task descriptors.Each task descriptor in the first LM output 222 is a semantic description of a corresponding task for the at least one robot to perform. As a non-limiting example, the first LM output 222 may comprise a plurality of task descriptors such as "pick soap", "place snack on napkin", "throw sponge", "wipe soap on napkin", "tear sponge in two", "pick up a shadow".

[0051] FIG. 2 shows an affordance engine 230 which may be used to assist with filtering the task descriptors provided by the LLM 220 in the first LLM output 222. Although a selected training policy descriptor 206a was used to generate the first LLM prompt 202, to guide the LLM 220 to output task descriptors of tasks that conform to the selected training policy descriptor 206a, it is possible that the LLM 220 may nevertheless include in its first LLM output 222 task descriptors of tasks that do not conform to the selected training policy descriptor 206a, or may be better performed by a different training policy to the training policy corresponding to the selected training policy descriptor 206a. The affordance engine 230 can assist with filtering task descriptors in the first LLM output 222 that do not conform to the selected training policy 206a, or that may be better performed by a different training policy to the training policy corresponding to the selected training policy descriptor 206a.

[0052] The affordance engine 230 generates a second LLM prompt 232 based on the task descriptors provided in the first LLM output 222 and the training policy descriptors 206. The affordance engine 230 can generate the second LLM prompt 232 as a natural language second LLM prompt 232. The second LLM prompt 232 include the plurality of task descriptors provided in the first LLM output 222. The second LLM prompt 232 further includes the plurality of training policy descriptors 206. The second LLM prompt 232 may include a prompt instructing the LLM 240 to classify each of the task descriptors as belonging to a respective one of the training policy descriptors, or as being rejected (for example due to being non-compliant with the safety rules). The second LLM prompt 232 may comprise any of the additional rules 216 or prompts described previously, to assist the LLM 240 in the classification.

[0053] As a non-limiting example of a second LLM prompt 232, the second LLM prompt 232 can be "You are a robot. You have only one arm, and thus cannot perform tasks requiring two arms. You must not interact with objects that are sharp, such as a knife. You are operating in a small kitchen with low lighting and objects scattered on a table. The objects include soap, a napkin, a snack, a cloth and a sponge. You can operate in multiple policy modes. In a first policymode, you can only pick up objects. In a second policy mode, you can ask a human for help by teleoperation. In a third policy mode, you can use a learned policy that can pick, move near, knock, place upright and open / close. If a task cannot be performed due to an aforementioned limitation, you must classify it as rejected. You are asked to perform the following tasks: "pick soap", "place snack on napkin", "throw sponge", "wipe soap on napkin", "tear sponge in two", "pick up a shadow". Classify the tasks according to the policy modes, or as rejected: Output as: 1. 'Task 1' - 'classification'; 2. 'Task 2' - 'classification'; etc.".

[0054] The affordance engine 230 causes the generated second LLM prompt 205A to be processed using an LLM 240, which LLM 240 may be the same LLM 220 used to process the first LLM prompt 202 or may be a different, to generate a second LLM output 242. The affordance engine 230 (or another component of the system) may cause the LLM 240 to process the second LLM prompt 232 by providing the second LLM prompt 232 as input to the LLM 240. Responsive to receiving the second LLM prompt 232 as an input, the LLM 240 processes the second LLM prompt 232 to provide a second LLM output 242, which is received by the system. The LLM 240 may form part of the same system as the other components shown in FIG. 2, or may instead be external to the system, for example hosted by one or more remote, external servers.

[0055] The second LLM output 242 includes the task descriptors provided to the LLM 240 as part of the second LLM prompt 232, however the second LLM output 242 will further specify, for each task descriptor, which of the training policies 210 may be used to perform the respective task associated with each task descriptor. If a task cannot be performed, for example because it contravenes a rule provided to the LLM 240 as part of the second LLM prompt 232, the second LLM output 240 will classify that particular task descriptor as being rejected. The second LM output 242 therefore classifies each task descriptor as corresponding to at least one respective training policy descriptor of the plurality of training policy descriptors 206, or optionally as being rejected. A non-limiting example of the second LLM output 242 may be "1. pick soap - first policy mode; 2. place snack on napkin - third policy mode; 3. throw sponge - second policy mode; 4. wipe soap on napkin - third policy mode; 5. tear sponge in two - second policy mode; 6. pick up a shadow - rejected".

[0056] FIG. 2 shows a task selection engine 250 which may be used to select a task to beperformed by the robot 110 from among the tasks suggested by the LLM 220 or LLM 240. As discussed previously, it is possible that the LLM 220 include in its first LLM output 222 task descriptors of tasks that do not conform to the selected training policy descriptor 206a, or may be better performed by a different training policy to the training policy associated with the selected training policy descriptor 206a. The second LM output 242 has now classified each task descriptor provided in the first LLM output 222 as corresponding to at least one respective training policy descriptor of the plurality of training policy descriptors 206. This can allow the task selection engine 250 to filter out task descriptors in the second LLM output 242 that have not been classified in the second LLM output 242 as corresponding to the selected training policy descriptor.

[0057] The task selection engine 250 receives the second LLM output 242 as an input and filters the task descriptors in the second LLM output 242 by determining a subset of the task descriptors contained in the second LLM output 242, the subset of the task descriptors comprising those task descriptors that have been classified in the second LLM output 242 as corresponding to the selected training policy descriptor 206a. The task selection engine 250 then selects a selected task descriptor 252 from the subset of task descriptors. The selected task descriptor 252 corresponds to a task to be performed, or attempted to be performed, by the robot 110.

[0058] If the subset contains a single task descriptor then that task descriptor will be selected as the selected task descriptor 252. However, if the subset contains more than one task descriptor then the task selection engine 250 will select the selected task descriptor 252 from among the plurality of task descriptors in the subset. For example, the selected task descriptor 252 may be chosen at random from the subset. This may help ensure a diverse range of tasks are selected to be performed by the robot for training the machine learning model(s). However, other methods of selecting the selected task descriptor 252 from among the subset of task descriptors may be used. For example, the task selection engine 250 may select the task descriptor having a predetermined position in the subset, for example the first task descriptor listed in the subset or the last task descriptor listed in the subset. In other examples, the task selection engine 250 may select the selected task descriptor 252 based on determining a relative uniqueness of each of the task descriptors in the subset and selectingthe most unique task descriptor in the subset. In other examples, the task selection engine 250 may select the selected task descriptor 252 based on whether the task corresponding to the task descriptor (or a similar task to that task) has been performed previously by the robot 110. In other examples, the task selection engine 250 may select the selected task descriptor 252 based on whether a task similar to the task corresponding to a particular task descriptor has previously resulted in failure when performed by the robot 110, which may be indicative that further training based on the task may be required.

[0059] FIG. 2 further shows an implementation engine 260. In response, to receiving the selected task descriptor 252 that has been selected by the task selection engine 250, the implementation engine 260 causes the robot 110 to perform, or attempt to perform, the task corresponding to the selected task descriptor 252. The implementation engine 260 may control the robot 110 to perform the task based on the selected training policy. In some examples, the selected training policy uses one of the machine learning models 290.

[0060] FIG. 2 further shows a training data engine 270, which is used to record training data 272 as the robot performs the task. The training data 272 is received by a training engine 280, which can use the training data 272 to train the machine learning model(s) 290. The machine learning model(s) 290 may be used by the robot 110 shown in FIG. 2 or a different robot to perform tasks. In some examples, by training the machine learning model(s) 290 using the training data 280, the machine learning model(s) 290 may gain the ability to control the robot 110 or other robots to perform new tasks. Additionally or alternatively, training the machine learning model(s) 290 using the training data 280 may allow the machine learning model(s) 290 to control the robot 110 or other robots to perform known tasks in an alternative manner, more efficiently, or with a better success rate.

[0061] Turning now to FIG. 3, a flowchart is illustrated of an example method 300 of training one or machine learning models used for controlling a robot, according to implementations disclosed herein. For convenience, the operations of the method 300 are described with reference to a system that performs the operations. This system can include one or more components of a robot, such as a robot processor and / or robot control system of robot 110, robot 720, and / or other robot and / or can include one or more components of a computer system, such as computer system 810. Moreover, while operations of method 300 are shownin a particular order, this is not meant to be limiting. One or more operations can be reordered, omitted or added. For example, one or more operations described in relation to FIG. 2 may be incorporated.

[0062] At block 310, the system captures, using one or more vision components, a vision data instance in an environment of at least one robot, such as the robot 110.

[0063] The environment in which the vision data instances, utilized by the system at block 300, are captured is a constrained and optionally defined space in which the robot 110 at least selectively operates. The environment can be, for example, a room in a building, multiple rooms in a building, an entire floor of a building, and / or the entirety of a building. In some implementations, the environment in which a robot operates can be dictated by human input (e.g., defining areas to which the robot is confined and / or defining areas to which the robot is prohibited from entering) and / or can be dictated by constraint(s) of the robot (e.g., a robot may be incapable of navigating stairs, opening any or certain type(s) of doors, etc.).

[0064] The vision data instances utilized by the system can be generated by one or more vision components that were (and perhaps still are) in the environment. Such vision component(s) can include vision component(s) of the robot 110, vision component(s) of additional robot(s) previously (and perhaps currently) in the environment, and / or other fixed or non-fixed vision component(s) in the environment. In some implementations, the vision data instances include (or are restricted to) images that include multiple color channels (e.g., red, green, and blue (RGB) channels) and / or that include a depth (D) channel. For example, the vision data instances can include RGB images generated by a monocular camera vision component and / or RGB-D images generated by a stereographic camera vision component.One or more of the vision data instances utilized by the system at block 300 can optionally be generated by robot(s), during exploration of the environment, such as exploration using predetermined waypoints in the environment and / or exploration using frontier exploration algorithms.

[0065] At block 320, the system processes the vision data instance captured at block 310 to generate at least one object descriptor. Each object descriptor describes at least one respective object within a scene represented by the vision data instance. Optionally, the system may also process the vision data instance captured at block 310 to generate at least one scenedescriptor. Each scene descriptor describes the scene represented by the vision data instance. The system may use a VLM to process the vision data instance 180 to generate the object descriptor(s) and / or scene descriptor(s).

[0066] At block 330, the system causes a language model (LM) to process a first LM prompt that has been generated based on the at least one object descriptor, a selected training policy descriptor to generate a first LM output and, if it has been generated at block 320, the at least one scene descriptor. The first LM output comprises a plurality of task descriptors each corresponding to a respective task for the at last one robot to perform, wherein the selected training policy descriptor has been selected from a plurality of training policy descriptors, each training policy descriptor corresponding to a respective training policy of a plurality of training policies. The system may select the training policy descriptor based on selection data.

[0067] In some examples, the language model forms part of the system. In other examples, the language model is external to the system, in which case the system causes the language model (LM) to process the first LM prompt by providing the first LM prompt as an input to the language model. In response to receiving the first LM prompt as an input, the language model processes the first LM prompt to provide the first LM output, which is received by the system.

[0068] At block 340, the system causes the LM or a different LM to process a second LM prompt that is based on the plurality of training policy descriptors and the plurality of task descriptors to generate a second LM output that classifies each task descriptor as corresponding to at least one respective training policy descriptor of the plurality of training policy descriptors.

[0069] At block 350, the system selects a task descriptor from a subset of the plurality of task descriptors that have been classified as corresponding to the selected training policy descriptor.

[0070] At block 360, the system causes, based on the training policy corresponding to the selected training policy descriptor, the robot to attempt to perform the task corresponding to the selected task descriptor in the environment.

[0071] At step 370, the system acquires training data as the robot attempts to perform the task corresponding to the selected task descriptor.

[0072] At step 380, the system trains, or causes to be trained, a machine learning (ML)model using the training data.

[0073] Turning now to FIG. 4, a flowchart is illustrated of an example method 400 of training one or machine learning models used for controlling a robot, according to implementations disclosed herein. For convenience, the operations of the method 400 are described with reference to a system that performs the operations. This system can include one or more components of a robot, such as a robot processor and / or robot control system of robot 110, robot 720, and / or other robot and / or can include one or more components of a computer system, such as computer system 810. Moreover, while operations of method 400 are shown in a particular order, this is not meant to be limiting. One or more operations can be reordered, omitted or added. For example, one or more operations described in relation to FIG. 2 may be incorporated.

[0074] At block 410, the system selects a training policy descriptor from a plurality of training policy descriptors based on selection data. Each training policy descriptor corresponds to a respective training policy for training a machine learning (ML) model. The plurality of training policy descriptors comprises at least one training policy descriptor corresponding to a teleoperation training policy and at least one training policy descriptor corresponding to a non- teleoperation training policy. The selection data is indicative of whether a human should be involved with training the machine learning (ML) model.

[0075] At block 420, the system generates a first LM prompt based on the selected training policy descriptor and an object descriptor. Each at least one object descriptor describes at least one respective object within a scene represented by a vision data instance, the vision data instance captured using one or more vision components in an environment of at least one robot. Optionally, the system generate the first LM prompt further based on at least one scene descriptor. Each scene descriptor describes the scene represented by the vision data instance. The system may use a VLM to process the vision data instance 180 to generate the object descriptor(s) and / or scene descriptor(s).

[0076] At block 430, the system causes an LM to process the first language model prompt to generate a first LM output comprising a plurality of task descriptors. Each task descriptor corresponds to a respective task for the at last one robot to perform.

[0077] At block 440, the system selects a task descriptor from the plurality of taskdescriptors.

[0078] At block 450, the system causes a robot to perform a task corresponding to the selected task descriptor. The system may be caused to perform the task based on the training policy corresponding to the selected training policy descriptor.

[0079] At block 460, the system acquires training data as the robot performs the task.

[0080] At block 470, the system causes a machine learning model(s) to be trained using the training data.

[0081] As indicated by the dotted arrow, the method 400 may be repeated to further train the machine learning model(s).

[0082] Turning now to FIG. 5, a flowchart is illustrated of an example method 500 of training one or machine learning models used for controlling a robot, according to implementations disclosed herein. For convenience, the operations of the method 500 are described with reference to a system that performs the operations. This system can include one or more components of a robot, such as a robot processor and / or robot control system of robot 110, robot 720, and / or other robot and / or can include one or more components of a computer system, such as computer system 810. Moreover, while operations of method 500 are shown in a particular order, this is not meant to be limiting. One or more operations can be reordered, omitted or added. For example, one or more operations described in relation to FIG. 2 may be incorporated.

[0083] At block 510, the system causes an LM to process a first LM prompt to generate a first LM output comprising a plurality of task descriptors. Each task descriptor corresponds to a respective task for at least one robot to perform. The first LM prompt has been generated based on at least one object descriptor that each describe at least one respective object within a scene represented by a vision data instance, the vision data instance captured using one or more vision components in an environment of the at least one robot. The first LM prompt has been further generated based on a selected training policy descriptor that has been selected from a plurality of training policy descriptors, each training policy descriptor corresponding to a respective training policy of a plurality of training policies.

[0084] At block 520, the system generates a second LM prompt that is based on the plurality of training policy descriptors and the plurality of task descriptors.

[0085] At block 530, the system causes the LM of block 510 or a different LM to process the second LM prompt to generate a second LM output that classifies each task descriptor as corresponding to at least one respective training policy descriptor of the plurality of training policy descriptors.

[0086] At block 540, the system determines a subset of the plurality of task descriptors. The subset of the plurality of task descriptors comprises those task descriptors classified as corresponding to the selected training policy descriptor.

[0087] At block 550, the system selects a task descriptor from the subset of the plurality of task descriptors.

[0088] At block 560, the system causes a robot to attempt to perform the task corresponding to the selected task descriptor. The system causes the robot to attempt to perform the task corresponding to the selected task descriptor based on the training policy corresponding to the selected training policy descriptor.

[0089] At block 570, the system acquires training data as the robot performs the task.

[0090] At block 580, the system causes a machine learning model(s) to be trained using the training data.

[0091] As indicated by the dotted arrow, the method 500 may be repeated to further train the machine learning model(s).

[0092] Turning now to FIG. 6, a flowchart is illustrated of an example method 600 of training one or machine learning models used for controlling a robot, according to implementations disclosed herein. For convenience, the operations of the method 600 are described with reference to a system that performs the operations. This system can include one or more components of a robot, such as a robot processor and / or robot control system of robot 110, robot 720, and / or other robot and / or can include one or more components of a computer system, such as computer system 810. Moreover, while operations of method 600 are shown in a particular order, this is not meant to be limiting. One or more operations can be reordered, omitted or added. For example, one or more operations described in relation to FIG. 2 may be incorporated.

[0093] At block 610, the system causes an LM to process a first LM prompt to generate a first LM output comprising a plurality of task descriptors. Each of the task descriptorscorresponds to a respective task for at least one robot to perform. The first LM prompt has been generated based on at least one object descriptor that each describe at least one respective object within a scene represented by a vision data instance, the vision data instance captured using one or more vision components in an environment of the at least one robot. The first LM prompt has been further generated based on a selected training policy descriptor that has been selected from a plurality of training policy descriptors. Each training policy descriptor corresponds to a respective training policy of a plurality of training policies.

[0094] At block 620, the system selects a task descriptor from the plurality of task descriptors.

[0095] At block 630, the system causes a robot to perform the task corresponding to the selected task descriptor. The system causes the robot to perform the task corresponding to the selected task descriptor based on the training policy corresponding to the selected training policy descriptor.

[0096] At block 640, the system acquires training data as the robot performs the task.

[0097] At block 650, the system causes a machine learning model(s) to be trained using the training data.

[0098] At block 660, the system determines at least one diversity measure corresponding to the task performed by the robot. The at least one diversity measure may comprise a language diversity measure, which is a measure of the diversity of the selected task descriptor compared to one or more task descriptors selected during a previous iteration of method 600. The at least one diversity measure may additionally or alternatively comprise a visual diversity measure, which indicates a diversity of a video of the task being performed as compared to one or more videos of previous tasks being performed during a previous iteration of method 600. The video may have been captured using the vision component 111 and may form part of the acquired training data.

[0099] At block 670, the system records the at least one diversity measure, for example in a memory of the system. In some examples, an indication of the at least one diversity measure may be output by the system. For example, an audio and / or visual indication of the at least one diversity measure may be output using the user interface output device(s) 820 described in relation to FIG. 7.

[0100] As indicated by the dotted arrow, the method 600 may be repeated to further train the machine learning model(s).

[0101] FIG. 7 schematically depicts an example architecture of a robot 720. The robot 720 includes a robot control system 760, one or more operational components 740a-740n, and one or more sensors 742a-742m. The sensors 742a-742m may include, for example, vision sensors, light sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors, accelerometers, gyroscopes, thermometers, barometers, and so forth. While sensors 742a-m are depicted as being integral with robot 720, this is not meant to be limiting. In some implementations, sensors 742a-m may be located external to robot 720, e.g., as standalone units.

[0102] Operational components 740a-740n may include, for example, one or more end effectors and / or one or more servo motors or other actuators to effectuate movement of one or more components of the robot. For example, the robot 720 may have multiple degrees of freedom and each of the actuators may control the actuation of the robot 720 within one or more of the degrees of freedom responsive to the control commands. As used herein, the term actuator encompasses a mechanical or electrical device that creates motion (e.g., a motor), in addition to any driver(s) that may be associated with the actuator and that translate received control commands into one or more signals for driving the actuator. Accordingly, providing a control command to an actuator may comprise providing the control command to a driver that translates the control command into appropriate signals for driving an electrical or mechanical device to create desired motion.

[0103] The robot control system 760 may be implemented in one or more processors, such as a CPU, GPU, and / or other controller(s) of the robot 720. In some implementations, the robot 720 may comprise a "brain box" that may include all or aspects of the control system 760. For example, the brain box may provide real time bursts of data to the operational components 740a-n, with each of the real time bursts comprising a set of one or more control commands that dictate, inter alia, the parameters of motion (if any) for each of one or more of the operational components 740a-n. In some implementations, the robot control system 760 may perform one or more aspects of method(s) described herein, such as method 300 of FIG.3, method 400 of FIG. 4, method 500 of FIG. 5, method 600 of FIG. 6, and / or any methoddescribed in relation to FIG. 2.

[0104] In some implementations, all or aspects of the control commands generated by control system 760, in controlling a robot during performance of a robotic task, can be generated based on robotic skill(s) determined to be relevant for the robotic task and, optionally, based on determined map location(s) for environmental object(s). Although control system 760 is illustrated in FIG. 7 as an integral part of the robot 720, in some implementations, all or aspects of the control system 760 may be implemented in a component that is separate from, but in communication with, robot 720. For example, all or aspects of control system 760 may be implemented on one or more computing devices that are in wired and / or wireless communication with the robot 720, such as computing device 810.

[0105] FIG. 8 is a block diagram of an example computing device 810 that may optionally be utilized to perform one or more aspects of techniques described herein. Computing device 810 typically includes at least one processor 814 which communicates with a number of peripheral devices via bus subsystem 812. These peripheral devices may include a storage subsystem 824, including, for example, a memory subsystem 825 and a file storage subsystem 826, user interface output devices 820, user interface input devices 822, and a network interface subsystem 816. The input and output devices allow user interaction with computing device 810. Network interface subsystem 816 provides an interface to outside networks and is coupled to corresponding interface devices in other computing devices.

[0106] User interface input devices 822 may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen incorporated into the display, audio input devices such as voice recognition systems, microphones, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and ways to input information into computing device 810 or onto a communication network.

[0107] User interface output devices 820 may include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices. The display subsystem may include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual display such as via audio output devices. In general,T1use of the term "output device" is intended to include all possible types of devices and ways to output information from computing device 810 to the user or to another machine or computing device.

[0108] Storage subsystem 824 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 824 may include the logic to perform selected aspects of the method 300 of FIG. 3, the method 400 of FIG. 4, the method 500 of FIG. 5, the method 600 of FIG. 6, and / or any method described in relation to FIG. 2.

[0109] These software modules are generally executed by processor 814 alone or in combination with other processors. Memory 825 used in the storage subsystem 824 can include a number of memories including a main random access memory (RAM) 830 for storage of instructions and data during program execution and a read only memory (ROM) 832 in which fixed instructions are stored. A file storage subsystem 826 can provide persistent storage for program and data files, and may include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges. The modules implementing the functionality of certain implementations may be stored by file storage subsystem 826 in the storage subsystem 824, or in other machines accessible by the processor(s) 814.

[0110] Bus subsystem 812 provides a mechanism for letting the various components and subsystems of computing device 810 communicate with each other as intended. Although bus subsystem 812 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0111] Computing device 810 can be of varying types including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device 810 depicted in FIG. 8 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computing device 810 are possible having more or fewer components than the computing device depicted in FIG. 8.

[0112] Other implementations can include a non-transitory computer readable storagemedium storing instructions executable by one or more processor(s) (e.g., a central processing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s)), and / or tensor processing unit(s) (TPU(s))) to perform a method such as one or more of the methods described herein. Yet other implementations can include a system of one or more computers and / or one or more robots that include one or more processors operable to execute stored instructions to perform a method such as one or more of the methods described herein.

Claims

CLAIMSWhat is claimed is:

1. A method implemented by one or more processors, the method comprising: capturing, using one or more vision components, a vision data instance in an environment of at least one robot; processing the vision data instance to generate at least one object descriptor, wherein each at least one object descriptor describes at least one respective object within a scene represented by the vision data instance; causing a language model (LM) to process a first LM prompt that has been generated based on the at least one object descriptor and a selected training policy descriptor to generate a first LM output comprising a plurality of task descriptors each corresponding to a respective task for the at last one robot to perform, wherein the selected training policy descriptor has been selected from a plurality of training policy descriptors, each training policy descriptor corresponding to a respective training policy of a plurality of training policies; causing the LM or a different LM to process a second LM prompt that is based on the plurality of training policy descriptors and the plurality of task descriptors to generate a second LM output that classifies each task descriptor as corresponding to at least one respective training policy descriptor of the plurality of training policy descriptors; selecting a task descriptor from a subset of the plurality of task descriptors that have been classified as corresponding to the selected training policy descriptor; causing, based on the training policy corresponding to the selected training policy descriptor, the robot to attempt to perform the task corresponding to the selected task descriptor in the environment; acquiring training data as the robot attempts to perform the task; and causing a machine learning (ML) model to be trained using the training data.

2. The method of claim 1, wherein the plurality of training policies comprises a teleoperated training policy and a non-teleoperated training policy.

3. The method of claim 2, wherein the selected training policy descriptor corresponds to the teleoperated training policy, wherein the method further comprises outputting, at a user interface output device, an indication that teleoperation of the robot is required.

4. The method of claim 2 or claim 3, further comprising outputting, at the user interface output device, an indication of the task corresponding to the selected task descriptor for verification by a human and, in response to outputting the indication of the task, receiving, at a user interface input device, user input verifying that the task can be performed by the robot.

5. The method of any one of claims 2 to 4, further comprising receiving, at a user interface input device, user input to teleoperate the robot to perform the task.

6. The method of any preceding claim, wherein processing the vision data instance to generate the least one object descriptor comprises providing the vision data instance to a visual language model (VLM) to generate the at least one object descriptor.

7. The method of any preceding claim, further comprising processing the vision data instance to generate at least one scene descriptor, wherein each at least one scene descriptor describes the scene represented by the vision data instance, wherein the first LM prompt has been further generated based on the at least one scene descriptor.

8. The method of claim 6, wherein processing the vision data instance to generate at least one scene descriptor comprises providing the vision data instance to a visual language model (VLM) to generate the at least one scene descriptor.

9. The method of any preceding claim, further comprising selecting the selected training policy descriptor from the plurality of training policy descriptors based on selection data.

10. The method of claim 9, wherein the selection data indicates the availability of a human to supervise the robot.

11. The method of claim 8 or claim 9, wherein the selection data indicates a ratio of humans to robots in the environment.

12. The method of any one of claims 8 to 11, wherein the selection data indicates, for each training policy descriptor of the plurality of training policy descriptor, a respective probability for selecting that training policy descriptor.

13. The method of any preceding claim, further comprising causing the at least one robot to drive to a location in the environment prior to capturing the vision data instance.

14. A method implemented by one or more processors, the method comprising: selecting a training policy descriptor from a plurality of training policy descriptors based on selection data, wherein each training policy descriptor corresponds to a respective training policy for training a machine learning (ML) model, wherein the plurality of training policy descriptors comprises at least one training policy descriptor corresponding to a teleoperation training policy and at least one training policy descriptor corresponding to a non-teleoperation training policy, and wherein the selection data is indicative of whether a human should be involved with training the machine learning (ML) model; generating a first language model (LM) prompt based on the selected training policy descriptor and at least one object descriptor, wherein each at least one object descriptor describes at least one respective object within a scene represented by avision data instance, the vision data instance captured using one or more vision components in an environment of at least one robot; causing an LM to process the first LM prompt to generate a first LM output comprising a plurality of task descriptors each corresponding to a respective task for the at last one robot to perform; selecting a task descriptor from the plurality of task descriptors; causing, based on the training policy corresponding to the selected training policy descriptor, the robot to attempt to perform the task corresponding to the selected task descriptor in the environment; acquiring training data as the robot attempts to perform the task; and causing the machine learning (ML) model to be trained using the training data.

15. A method implemented by one or more processors, the method comprising: causing a language model (LM) to process an LM prompt to generate a first LM output comprising a plurality of task descriptors each corresponding to a respective task for at least one robot to perform, wherein the first LM prompt has been generated based on: at least one object descriptor that each describe at least one respective object within a scene represented by a vision data instance, the vision data instance captured using one or more vision components in an environment of the at least one robot; and a selected training policy descriptor that has been selected from a plurality of training policy descriptors, each training policy descriptor corresponding to a respective training policy of a plurality of training policies; generating a second LM prompt that is based on the plurality of training policy descriptors and the plurality of task descriptors; causing the LM or a different LM to process the second LM prompt to generate a second LM output that classifies each task descriptor as corresponding to at least one respective training policy descriptor of the plurality of training policy descriptors;determining a subset of the plurality of task descriptors, the subset of the plurality of task descriptors comprising task descriptors classified as corresponding to the selected training policy descriptor; selecting a task descriptor from the subset of the plurality of task descriptors; causing, based on the training policy corresponding to the selected training policy descriptor, the robot to attempt to perform the task corresponding to the selected task descriptor; acquiring training data as the robot attempts to perform the task; and causing a machine learning (ML) model to be trained using the training data.

16. A method implemented by one or more processors, the method comprising: causing a language model (LM) to process a first LM prompt to generate a first LM output comprising a plurality of task descriptors each corresponding to a respective task for at least one robot to perform, wherein the first LM prompt has been generated based on: at least one object descriptor that each describe at least one respective object within a scene represented by a vision data instance, the vision data instance captured using one or more vision components in an environment of the at least one robot; and a selected training policy descriptor that has been selected from a plurality of training policy descriptors, each training policy descriptor corresponding to a respective training policy of a plurality of training policies; selecting a task descriptor from the plurality of task descriptors; causing, based on the training policy corresponding to the selected training policy descriptor, the robot to attempt to perform the task corresponding to the selected task descriptor in the environment; acquiring training data as the robot attempts to perform the task; causing a machine learning (ML) model to be trained using the training data; determining at least one diversity measure corresponding to the task being performed by the robot; andrecording the at least one diversity measure.

17. A system comprising memory storing instructions and one or more processors that are operable to execute the instructions to perform the method of any preceding claim.

18. The system of claim 17, wherein the system comprises one or more robots.

19. One or more computer readable media storing instructions that, when executed by one or more processors, perform the method of any of claims 1 to 16.

Images