Human in-loop task and action planning in imitation learning

Abstract

The invention discloses human in-the-loop task and action planning in imitation learning. In various examples, systems and methods are disclosed that relate to training a machine learning model using an artificial presentation of sections of a task, where other sections of the task are performed by a planning method, such as a task and motion planning (TAMP) system. A method may include segmenting a task performed by a robot into segments, determining a first set of instructions in a plurality of sets of instructions for operating the robot to perform a first target of the first segment, determining that the plurality of sets of instructions are insufficient to perform a second target of a second segment, a second set of instructions for operating the robot for a second section after the end of the first section is received from the user device, and a machine learning model for controlling the robot is updated using the second set of instructions for the second section.

Description

Background Art

[0001] Imitation learning is a method that uses human demonstrations of complex manipulation skills to train robots or other autonomous systems to learn those skills. However, imitation learning does not scale well because providing long manipulation demonstrations is time-consuming and labor-intensive. Summary of the invention

[0002] Embodiments of the present disclosure relate to using imitation learning to train (e.g., update one or more parameters, such as weights and biases) and deploy machine learning models, e.g., to perform segments of a task. For example, systems and methods are disclosed that enable decomposing a task into segments, identifying segments for which imitation learning is required, and requesting human demonstrations for those segments.

[0003] Combining robotic motion planning methods with imitation learning allows robots to complete tasks beyond the scope of planning methods, while greatly improving training throughput relative to imitation learning of the entire task. Conventional robotic motion planning methods typically require accurate dynamic models and precise perception (which are often unavailable), which limits their effectiveness in contact-rich and low-tolerance operations. The system can plan behaviors for a wide range of multi-step manipulation tasks by searching over effective combinations of a small number of primitive skills. Traditionally, each skill is hand-engineered; however, some skills, such as closing a spring cap or inserting a rod into a hole, are extremely difficult to model in a productive way.

[0004] The embodiments discussed herein provide for training a machine learning model to execute an instruction set to provide or fine-tune a lacking skill. By training a machine learning model to perform skills that a planning method does not provide, while allowing a planning method to perform skills that a planning method can provide, the embodiments herein provide the benefits of both imitation learning and planning methods. A planning method may include a machine learning model to supplement its existing skills with the skills provided by a machine learning model trained by imitation learning, and imitation learning may be limited to segments of skills or tasks that cannot be performed using the existing skills of the planning method. This greatly reduces the amount of human demonstrations required, reduces the time and burden on human operators, and increases the throughput of training compared to conventional imitation learning.

[0005] In an example implementation, a processor (e.g., one or more processors or processing units) includes one or more circuits for segmenting a task to be performed by a robot. For each segment, the one or more circuits may select an instruction set from a plurality of instruction sets to be applied and apply the selected instruction set to each segment to enable the robot to perform a portion of the task, wherein one or more of the instruction sets from the selected instruction sets are executed by a machine learning model, and wherein the machine learning model has been trained. The training may include, for one iteration of performing the task, applying a first instruction set from a plurality of instruction sets to move a first segment of the robot for a first segment in the segments, requesting a human to demonstrate a second instruction set from the plurality of instruction sets, and updating the second instruction set based on the human demonstration.

[0006] The plurality of instruction sets may include coordinates and vectors for controlling the robot. The machine learning model may be trained to execute a second instruction set based on determining that a subset of the plurality of instruction sets is insufficient to execute an objective of the second segment, wherein the objective includes moving an object from a first pose to a second pose. The subset of the plurality of instruction sets may be insufficient to execute the objective based on the subset of the plurality of instruction sets being insufficient to control the robot to move the object from a first pose to a second pose within a predetermined time period. The subset of the plurality of instruction sets may be insufficient to execute the objective based on the subset of the plurality of instruction sets being insufficient to control the robot to move the object from a first pose to a second pose without contacting the second object. The subset of the plurality of instruction sets may be insufficient to execute the objective based on the subset of the plurality of instruction sets being insufficient to control the robot to move the object from a first pose to a second pose with a predetermined level of accuracy. For a first segment of an iteration, a human demonstration may or may not be requested. The selected instruction set may include a transition pose for transitioning to a second instruction set executed by the machine learning model. One or more transition poses may be determined by mapping the segment to a human demonstration of the task and identifying one or more poses of the robot at a portion of the human demonstration corresponding to the transition from the first segment to the second segment.

[0007] In an example implementation, the processor may include one or more circuits for: segmenting a task to be performed by the robot, determining a first instruction set of a plurality of instruction sets for operating the robot to perform a first goal of a first segment in the segment, determining that the plurality of instruction sets are insufficient to perform a second goal of a second segment in the segment, sending a request for a human demonstration of the second segment to a user device of an operator, receiving a second instruction set from the user device for operating the robot to perform the second segment after the first segment ends, and updating a machine learning model for controlling the robot using the second instruction set for the second segment. The plurality of instruction sets may be coordinates and vectors for controlling the robot. The second goal may include moving an object from a first pose to a second pose. Based on the plurality of instruction sets being insufficient to control the robot to move the object from the first pose to the second pose within a predetermined time period, the plurality of instruction sets may be insufficient to perform the second goal. Based on the plurality of instruction sets being insufficient to control the robot to move the object from the first pose to the second pose without contacting the second object, the plurality of instruction sets may be insufficient to perform the second goal. Based on the plurality of instruction sets being insufficient to control the robot to move the object from the first pose to the second pose with a predetermined level of accuracy, the plurality of instruction sets may be insufficient to perform the second goal. The one or more circuits may send a request for a human demonstration of a second segment by adding the second segment to a queue of segments for which demonstration is requested. The segment queue may be associated with a human operator, and the one or more circuits may add the second segment to the segment queue based at least on an expected throughput of the human operator. A first goal for the first segment may include reaching a transition gesture for transitioning between the first segment and the second segment. The transition gesture may be determined by mapping the segments to a human demonstration of a task and identifying one or more gestures of the robot at a portion of the human demonstration corresponding to a transition from the first segment to the second segment. The segment may include a plurality of transition gestures for transitioning to a segment associated with the human demonstration. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present system and method for training and deploying a machine learning model using imitation learning is described in detail below with reference to the accompanying drawings, wherein:

[0009] Figure 1 is a block diagram of an example system according to at least some embodiments of the present disclosure;

[0010] Figure 2 is a block diagram of an example task partitioning according to at least some embodiments of the present disclosure;

[0011] Figure 3 depicts example executions of partitioned tasks in accordance with at least some embodiments of the present disclosure;

[0012] Figure 4is a flow chart of an example process for training a machine learning model in accordance with at least some embodiments of the present disclosure;

[0013] Figure 5 depicts an example queuing system for human demonstrations in accordance with at least some embodiments of the present disclosure;

[0014] Figure 6 An example bootstrapping process for determining transition poses between a planning method and a human demonstration is shown in accordance with at least some embodiments of the present disclosure; and

[0015] Figure 7 is a flow chart illustrating the operation of an example method for training a machine learning model in accordance with at least some embodiments of the present disclosure;

[0016] Figure 8 Example pseudo code in accordance with at least some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0017] Systems and methods are disclosed that involve using imitation learning to train and deploy machine learning models, for example, to perform one or more segments of a task.

[0018] The systems and methods described herein may be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machine tools, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles connected to one or more trailers, spacecraft, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, submarines, drones, and / or other vehicle types. In addition, the systems and methods described herein may be used for a variety of purposes, such as, but not limited to, machine control, machine motion, machine actuation, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, safety and supervision, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twins, generative AI applications, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing, and / or any other suitable application.

[0019] The disclosed embodiments can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aerial systems, medical systems, boating systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems including one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least in part in a data center, systems for performing conversational AI operations, systems for performing generative AI operations, systems implementing one or more language models (such as, one or more large language models (LLMs)), systems for hosting real-time streaming applications, systems for presenting one or more of virtual reality content, augmented reality content, or mixed reality content, systems for performing light transport simulations, systems for performing collaborative content creation of 3D assets, systems implemented at least in part using cloud computing resources, and / or other types of systems.

[0020] Learning from human demonstrations has emerged as a promising way to teach robots complex manipulation skills. However, extending the paradigm of learning from human demonstrations to long-horizon tasks in the real world is difficult—for example, because providing long manipulation demonstrations is time-consuming and labor-intensive. At the same time, not all parts of a task are equally challenging. For example, an important part of complex manipulation tasks (such as parts assembly or making a cup of coffee) is free space motion and object transportation, which can be easily automated by non-learning methods (such as motion planning). However, planning methods typically require accurate dynamic models and precise perception (which are often unavailable), which limits the effectiveness of planning methods for contact-rich and low-tolerance manipulation. The embodiments discussed herein address long-horizon manipulation tasks in the real world by combining the advantages of learning and planning methods.

[0021] Some embodiments discussed herein enhance motion planning methods, such as Task and Motion Planning (TAMP) systems, which have been shown to be significant in solving long span problems. Although various embodiments are discussed about the TAMP system, other planning methods can be used interchangeably. The TAMP method can be used to plan behaviors for a wide range of multi-step manipulation tasks by searching for effective combinations of a small number of original skills. Traditionally, each skill is manually operated; however, some skills, such as closing a spring cover or inserting a rod into a hole, are extremely difficult to model in a productive manner. Some embodiments discussed herein use a combination of human teleoperation and closed-loop learning to implement only these selection skills (skills that are difficult to implement using the TAMP system), while keeping the rest automated (e.g., performed by the TAMP system). These skills use human teleoperation when data is collected, and use strategies or machine learning models trained according to the data when deployed.

[0022] Some embodiments discussed herein are directed to human-in-the-loop TAMP (HITL-TAMP), a system that symbiotically combines TAMP with teleoperation (human demonstrations). The system collects demonstrations by using a TAMP gating control mechanism—the system trades off between the TAMP system and a human teleoperator, who takes over to fill in the gaps delegated by the TAMP. When prompted by the TAMP system, the human operator only needs to participate in selected steps of the mission plan, which means that the human operator can manage a fleet of robots by asynchronously participating in one demonstration session at a time, while the TAMP system controls the rest of the fleet.

[0023] By soliciting human demonstrations only when needed, and allowing humans to participate in multiple parallel sessions, the embodiments discussed herein significantly increase the throughput of data collection while reducing the effort required to collect large datasets on long-span, contact-rich tasks. The data collection system that collects human demonstrations can be combined with an imitation learning framework that trains the TAMP gating policy on the collected human data. This combination results in superior performance in terms of the amount of data and time required for humans to teach the task to the robot, as well as the success rate of the machine learning model, compared to collecting human demonstrations for the entire task.

[0024] Figure 1 is a block diagram of an example system 100 in accordance with at least some embodiments of the present disclosure. System 100 may include computing device 110, network 115, robot 120, and user device 130, as well as additional or alternative components.

[0025] The computing device 110 may include one or more processors 112 and one or more memories 114. The memory 114 may be a non-transitory medium including computer-readable instructions that, when executed by one or more processors such as the processor 112, cause the processor to perform one or more functions described herein. The memory 114 may be a hard drive, an SSD, a RAM, a buffer, a cache, or any other storage device. The memory 114 may include one or more storage devices. The processor 112 may include one or more circuits 113. The one or more circuits 113 may be configured to execute instructions (e.g., instructions stored in the memory 114) to perform various functions.

[0026] The network 115 may be or include a wide area network (WAN), a local area network (LAN), a cellular network, and / or any type of network. In one example, the network 115 is the Internet.

[0027] The robot 120 may be a robot configured to perform one or more tasks. The robot 120 may include one or more sensors. In one example, the robot 120 includes one or more cameras, one or more LiDAR sensors, one or more RADAR sensors, one or more ultrasonic sensors, and / or one or more other sensor types or modalities. The robot 120 may include one or more motors and one or more manipulators, such as grippers or claws. The robot 120 may include one or more memories and one or more processors for storing and executing instructions. The robot 120 may include a communication interface for receiving instructions via the network 115.

[0028] User device 130 may be a device used by a human operator. User device 130 may include a display. User device 130 may include a user interface for receiving user input, such as a mouse and / or keyboard or touch screen. User device 130 may be a notebook, a smart phone, a desktop computer, or any other device for displaying information and receiving input.

[0029] The computing device 110 may send instructions to the robot 120 via the network 115. The computing device 110 may send coordinates and vectors for controlling the robot 120 to the robot 120 via the network. The computing device 110 may send a request for a human demonstration of a task or a task segment to the user device 130 via the network 115. The computing device 110 may segment the task to be performed by the robot into a plurality of segments, as discussed herein, wherein each segment may have an associated goal. The goal may include at least one of changing the pose of at least one part of the robot and / or changing the pose of at least one object. In one example, the goal may be one of: 1) moving a portion of the entire robot from an original pose to a new pose, 2) moving two or more parts of the robot (e.g., different robot manipulators, segments, components) from an original pose to a new pose, 3) moving an object other than the robot from an original pose to a new pose, 4) picking up, engaging, or interacting with an object, 5) releasing an object, (6) causing an interaction between a first object and a second object, and / or (7) changing an object. The pose may include at least one of a position or an orientation. Subsequent segments may have different goals and different types of goals. In one example, the goal of the first segment is to move the robot's manipulator into position to grasp an object, while the goal of the second segment is to move the object from an original pose to a new pose.

[0030] The computing device 110 may select one or more skills or instruction sets stored in the memory 114 or the memory of the robot 120 to perform the first objective of the first segment of the task. The computing device 110 may determine that the skills or instruction sets stored in the memory 114 or the memory of the robot 120 are insufficient to perform the second objective of the second segment of the task. The computing device 110 may send a request for a human demonstration of the second segment of the task to the user device 130. The user device 130 may receive input from a human operator to control the robot 120 to demonstrate the second segment. The computing device 110 may train a machine learning model based on the human demonstration of the second segment. The computing device 110 may store the machine learning model to perform the second segment. The computing device 110 may access the machine learning model to perform the second segment of the task.

[0031] Figure 2 is a block diagram of an example task partitioning according to at least some embodiments of the present disclosure. Task partitioning may be performed by Figure 1 system 100 or one or more components of system 100 (e.g., Figure 1The task may include and / or be defined by the task objective 210. The TAMP system 220 may generate a mission plan 230 based on the task objective 210. Although the TAMP system is discussed herein, other motion planning systems may also be used. The mission plan 230 may include sections, each of which is associated with a goal. The mission plan 230 may include a first section 232, a second section 234, and a third section 236. The first section 232 may be associated with a first goal, the second section 234 may be associated with a second goal, and the third section 236 may be associated with a third goal. When executed, the first, second, and third goals may be equivalent to executing the mission objective 210.

[0032] Each of the first segment 232, the second segment 234, and the third segment 236 can be associated with a skill or instruction set for achieving the corresponding goal. The TAMP system 220 can associate the first segment 232, the second segment 234, and the third segment 236 with a skill or instruction set. The TAMP system 220 can associate the first segment 232, the second segment 234, and the third segment 236 with a corresponding skill or instruction set based on the skill or instruction set being sufficient to achieve the corresponding goal. The first segment 232 can be associated with a first skill 242, the second segment 234 can be associated with a second skill 244, and the third segment 236 can be associated with a third skill 246. The first skill 232, the second skill 244, and the third skill 246, collectively referred to herein as the motor skill 240, can be a skill for operating Figure 1 The instruction set of robot 120.

[0033] The TAMP system 220 can select the first skill 242 and the third skill 246 from a plurality of skills or instruction sets. The plurality of skills or instruction sets can be stored in Figure 1 114. The TAMP system 220 may determine that the plurality of skills or instruction sets are insufficient to perform the objective of the second segment 234. As discussed herein, the TAMP system 220 may request a human demonstration of the second segment 234 for training the machine learning model to perform the second skill 244. The second skill 244 may be added to the plurality of skills or instruction sets for use by the TAMP system 220. During deployment, the TAMP system 220 may apply the motor skill 240 to the corresponding segment to enable the robot to complete the task objective 210.

[0034] In some embodiments, the TAMP system 220 can receive input from a user to decompose the mission objective 210 into the mission plan 230 and determine which of the first section 232, the second section 234, and the third section 236 can be performed by multiple skills or instruction sets, and which section requires human demonstrations to train the machine learning model.

[0035] In some embodiments, the TAMP system 220 can determine which section of the first section 232, the second section 234, and the third section 236 can be performed by multiple skills or instruction sets, and which section needs to use human demonstrations to train the machine learning model by trying to apply one or more candidate skills to the second section 232 to achieve the second goal and determining that one or more candidate skills are not enough to achieve the second goal. In some embodiments, the TAMP system 220 can determine that multiple skills are insufficient by trying to apply one or more candidate skills to the second section 234 in the simulation. The simulation can include a 3D simulation of a robot, a workspace, and one or more objects for completing a task. In one example, the second goal is to move an object from a first posture to a second posture. Based on multiple instruction sets, it is not enough to control the robot to move the robot to move the object from the first posture to the second posture within a predetermined time period, and multiple skills may be insufficient. Based on multiple instruction sets, it is not enough to control the robot to move the robot to move the object from the first posture to the second posture without contacting the second object, and multiple skills may be insufficient. The plurality of skills may be insufficient based on the plurality of instruction sets being insufficient to control the robot to move the robot to move the object from the first pose to the second pose with a predetermined level of accuracy.

[0036] Figure 3 An example execution of a segmented task 330 is depicted in accordance with at least some embodiments of the present disclosure. Task 330 may be performed by Figure 1 The system 100 uses a Figure 2 Task 330 may include a start condition 331 and an end condition 338. The start condition 331 and / or the end condition 338 may be performed by a motion planning system such as the TAMP system 220 of FIG. Figure 2 The task 330 may be defined by a task objective such as the task objective 210 of Figure 2 A TAMP system such as TAMP system 220 of FIG. 30 may be decomposed into multiple sections. Task 330 may include a first section 332 , a second section 333 , a third section 334 , a fourth section 335 , a fifth section 336 , and a sixth section 337 .

[0037] The start condition 331 may be a starting pose of the robot 320 and / or one or more objects performing the task 330. The start condition 331 may be loosely defined based on one or more start parameters. The one or more start parameters may vary to allow flexibility in starting the task 331. In one example, the start condition 331 may include the robot 320 posing over a coffee pod on a table next to a coffee machine. The end condition 338 may be an end pose of the robot 320 and / or one or more objects. In one example, the end condition 338 may include the coffee pod being inserted into the interior of the coffee machine with the lid of the coffee machine closed on the coffee pod.

[0038] The first segment 332 can be performed by the TAMP system using one or more of a plurality of available skills. The TAMP system may have determined that a plurality of available skills are sufficient to perform the first segment 332. However, the TAMP system may have determined that a plurality of available skills are insufficient to perform the second segment 333. During training, the second segment 333 can be performed by a human operator providing a human demonstration of the second segment 333. The TAMP system may request a human demonstration of the second segment 333. The human demonstration of the second segment 333 can be used to train the first machine learning model to perform the second segment 333. During deployment, the first machine learning model can be used to perform the second segment 333.

[0039] The end posture of the robot 320 for the first section 332 can be based on the transition posture for transitioning between the first section 332 and the second section 333 and for transitioning between the control of the robot 320 provided by the TAMP system and the control of the robot provided by the human operator. The end posture of the second section 333 can be defined as the posture from which the TAMP system can successfully control the robot 320. The end posture of the second section 333 can prompt the TAMP system to regain control of the robot 320 to execute the third section 334. The TAMP system can execute the third section 334, request a human demonstration of the fourth section 335, execute the fifth section 336, and request a human demonstration of the sixth section 337. The human demonstrations of the fourth section 335 and the sixth section 337 can be used to train the second machine learning model and the third machine learning model, respectively. During deployment, the second machine learning model can be used to execute the fourth section 335, and the third machine learning model can be used to execute the sixth section 337. In some examples, two or more of the first machine learning model, the second machine learning model, and the third machine learning model are a single machine learning model trained to execute two or more corresponding segments.

[0040] During training, control of the robot 320 is transferred between the TAMP system and the human operator to perform the task 330. During deployment, control of the robot 320 is transferred between the TAMP system and the machine learning model trained using human demonstrations to perform the task 330. The TAMP system can select a machine learning model similar to how the TAMP system selects skills for executing a segment. In this way, the machine learning model can be integrated into the TAMP system to supplement the skills of the TAMP system. Training and / or deployment can be performed in virtual or physical space.

[0041] Task 330 may include any number of segments. Although segments are shown as alternating between being controlled by the TAMP system and being controlled by a human operator, any number of subsequent segments may be performed by either the TAMP system or a human operator. Human demonstration, while discussed as being provided by a single human operator, may be provided by multiple different human operators. In some embodiments, in addition to or alternatively to using a human operator to teach / perform a segment or skill, another robot or computing program previously trained on the skill or segment may be used to teach the current robot / machine learning model the skill / segment. In this way, learning from another robot or computing program may be transferred to the current robot / machine learning model to implement another skill of the robot / machine learning model.

[0042] Figure 4 430 is a flowchart of an example process for training a machine learning model in accordance with at least some embodiments of the present disclosure. Different segments of the task performed by the robot may be performed by the TAMP system or a human operator, such as in combination with Figure 3 discussed.

[0043] Different states of the task corresponding to the objectives of the segments of the task may represent different transition points between control of the robot by the TAMP system and control of the robot by a human operator. The start state of the first segment 431 may be the start condition of the task, similar to Figure 3 The end state 439 of the eighth section 438 may be the end condition of the task, similar to Figure 3 The end condition 338.

[0044] The TAMP system can execute the first section 431. The first section 431 may include a first transition gesture of the robot and / or one or more objects for transferring control from the TAMP system to a human operator. The human operator can execute the second section 432. The second section 432 may include a second transition gesture of the robot and / or one or more objects for transferring control from a human operator to the TAMP system. The first transition gesture may be defined by various gestures of the robot and / or one or more objects, from which the human operator can successfully execute the second section 432, as discussed herein. The second transition gesture may be defined by various gestures of the robot and / or one or more objects, from which the TAMP system can successfully execute the third section 433. The human operator can execute the fourth section 434. The TAMP system can execute the fifth section 435. The human operator can execute the sixth section 436. The TAMP system can execute the seventh section 437. The human operator can execute the eighth section 438 to achieve the end condition 439. Each section may include a transition gesture, as discussed herein.

[0045] In one example, the robot must place a mug on a coffee machine, remove a coffee box from a drawer, insert the box into the coffee machine, and then close the lid. The task has 8 segments - first, the TAMP system grabs the mug and approaches the placement location, then the human (or, once trained, the machine learning model) places the mug on the coffee machine (placement requires precision due to arm size and space constraints). Next, the TAMP system approaches the machine lid, and the human (or, once trained, the machine learning model) opens the lid (requiring prolonged contact with the hinged mechanism). Then, the TAMP system approaches the drawer handle, and the human (or, once trained, the machine learning model) opens the drawer. Finally, the TAMP system grabs the box from the drawer and approaches the machine, then the human (or, once trained, the machine learning model) inserts the box and closes the machine lid.

[0046] Figure 5 An example queuing system 500 for human demonstrations is depicted in accordance with at least some embodiments of the present disclosure. The queuing system 500 may be implemented in hardware, software, or a combination of hardware and software. The queuing system 500 may be part of the system 100 or executed on the system 100.

[0047] The queuing system 500 may include a control switch 550 that switches control from the TAMP system to a human operator. The queuing system 500 may include a user device 540. The user device 540 may be Figure 1The human operator can input commands to the robot using the user device 540. The queuing system 500 can select one or more sections from a plurality of sections of the task for human demonstration, such as the first section 531, the second section 532, the third section 533, the fourth section 534, the fifth section 535, and the sixth section 536. The queuing system 500 can select the third section 533, the fourth section 534, and the sixth section 536 for human demonstration. When the human demonstration of the third task 533 is being performed, the fourth section 534 and the sixth section 536 can be placed in the queue 560 for human demonstration of the fourth section 534 and the sixth section 536. When the human demonstration of the third section 533 has been performed, the switch 550 can grant control to the user device 540 to perform the human demonstration of the fourth task 534. When the human demonstration of the fourth section 534 has been performed, the switch 550 can grant control to the TAMP system to perform the fifth section 535. When the TAMP system has executed the fifth segment 535 , the switch may grant control to the user device 540 to perform a human demonstration of the sixth segment 536 .

[0048] In some embodiments, the TAMP system places these segments in a queue 560 based on the need for human demonstrations in the segments. In some embodiments, the queue 560 receives segments from multiple TAMP systems simultaneously. In this way, a human operator can perform human demonstrations on segments of multiple tasks. This can reduce the downtime of the human operator between human demonstrations and increase the throughput of human demonstrations. The shared queue 560 can allow a human operator to provide more human demonstrations to the task segments.

[0049] In one example, each of the multiple tasks is defined by a goal formula G. In each TAMP iteration, the TAMP system observes the current state s. If the current state s satisfies G, the event terminates, otherwise, the TAMP system solves a plan from the current state s to the goal G. The TAMP system then issues joint pose commands to execute the planned motion until the action a required for human demonstration is achieved. Next, control switches to teleoperation mode, in which the human has full 6-DOF control of the robot's end effector. The robot end effector is controlled using an action space controller. The TAMP system monitors whether the state s satisfies the planned action postcondition a.effects. Once satisfied, control switches back to the TAMP system, which re-plans. Figure 8 An example of pseudocode for this process is shown in .

[0050] In one example, the queuing system 500 is used to increase the throughput of human demonstrations for training one or more machine learning models to perform a segment of a task. Since the TAMP system requires human assistance only for a small portion of the task, human operators have the opportunity to manage multiple robots and data collection sessions simultaneously. To this end, the queuing system 500 allows each operator to interact with the fleet of robots. This can be accomplished by using several (N robot ) robot process, a single human process, and a queue 560. Each robot process runs asynchronously and spends time in one of three modes - (1) being controlled by the TAMP system, (2) waiting for human control, or (3) being controlled by a human. This allows the TAMP system to operate multiple robots in parallel. When the TAMP system wants to prompt a human to take control, the TAMP system enqueues the environment into a shared queue 560. The human process communicates with the human user device and sends control commands from the user device to one robot process at a time. When the human operator completes a segment, the TAMP system resumes control of the robot and the human process dequeues the next session from the queue 560.

[0051] In some embodiments, queue 560 receives segments based on the expected throughput of human operators. The expected throughput can be referred to as the average queue consumption rate. Assume that the average queue consumption rate (the number of task demonstrations completed per unit time) of human operators is R H , and the average queue generation rate (the number of successfully executed task segments per unit time) of the TAMP system is R T , then the effective production rate should match or exceed the consumption rate, as shown in expression (1):

[0052] R T (N robot - 1) ≥R H (1)

[0053] In expression (1), 1 is subtracted because a robot is controlled by a human. Rearranging expression (1) yields expression (2):

[0054]

[0055] Therefore, the size of the fleet should be at least one greater than the ratio of the human rate of generating demonstration segments to the TAMP rate of solving and executing those segments. This number is typically limited by the amount of system resources (in simulation) or the availability of hardware (in the real world). In practice, human operators also need to take breaks and have an effective "duty cycle" where the human operator is kept busy X% of the time. The embodiments discussed herein can also support this scaling (taking into account duty cycles). Assume that the human operator is busy at T on In-house operating system and in Toff rest within T, the human operator on During the period, the effective rate R H -R T (N robot -1) Consume items from the queue and off During the period, the rate R T (N robot -1) fills the queue. In some examples, expression (3) is used to ensure that the human consumption rate is less than or equal to the production rate:

[0056] N robot T off R T (Nrobot-1) (3)

[0057] Rearranging expression (3) yields expression (4):

[0058]

[0059] In expression (4), the human duty cycle X is defined in expression (5) as:

[0060]

[0061] The embodiments described herein result in task execution including segments controlled by TAMP and segments controlled by humans. The machine learning model can be trained on segments controlled by humans, such as using behavioral cloning. In order to deploy the trained machine learning model, the TAMP gating control loop identical to the switching logic can be used, which is used to use strategies rather than people to hand control over to human operators. Therefore, the TAMP system can pass control to the trained machine learning model instead of passing control to a human operator.

[0062] Figure 6An example bootstrap process for determining a transition gesture between a planning method and a human demonstration according to at least some embodiments of the present disclosure is shown. As discussed herein, a segment of a task includes one or more transition gestures for transferring control between a TAMP system and a human operator. In some embodiments, the TAMP system determines at least one transition gesture for transferring control to a human operator for a first segment 621 and at least one transition gesture for transferring control to a human operator for a second segment 622 based on a group of human demonstrations 610 of the task. In one example, the group of human demonstrations may include 1-5 human demonstrations of the task. The TAMP system can determine the transition gesture from a group of human demonstrations so that a human operator can successfully control the robot for a human-controlled segment. In one example, a human operator provides a group of demonstrations of a tool suspension task, so the TAMP system can guide one or more transition gestures for transferring control to a human operator for a selected segment, as described herein.

[0063] Tool hanging example

[0064] In this example, the robot must perform a tool hanging task, where the robot must assemble a structure by inserting a first part into a base and then placing a second part on top of the first part. The two parts are placed relative to the base, which never moves. The task consists of four segments, where the TAMP system grabs each part and approaches the insertion point while a human handles each insertion. For the initial condition, the components of the structure can be placed anywhere in the predefined workspace.

[0065] In this example, the robot acts in a discrete-time Markov decision process (MDP)<X,U,T(x′|x,u),R(x),P0> A robot is defined by a state space X, an action space U, a transition distribution T, a reward function R, and an initial state distribution P0. An offline dataset of N partial demonstration trajectories is provided. The offline dataset can be collected using the HITL-TAMP system discussed herein and can include a set of human demonstrations 610. TAMP policy π t(u|x) can be used to control the robot. TAMP policies can plan sequences of actions that will be tracked using a high-frequency feedback controller. The PDDLStream planning framework, which is a logic-based action language that supports planning with continuous values, can be used to model the TAMP domain. States and actions can be described using predicates, Boolean functions, which can have discrete and continuous parameters. A predicate paired with the value of its parameter is called a literal. The TAMP domain can use the following parameters: o is the object, g is the 6-DoF object grasp posture, p is the object placement posture, q is the robot configuration with d DoF, and τ is the robot trajectory consisting of a sequence of robot configurations. The planning state s is a set of true literals of fluent predicates, whose truth values ​​can change over time. The following stream predicates are defined as follows: AtPose(o,p) is true when object o is placed at position p; AtGrasp(o,p) is true when object o is grasped using grasp g; AtConf(q) is true when the robot is in configuration q; Empty() is true when the robot's end effector is empty; Attached(o,o') is true when object o is attached to object o'. In the "Tool Hanging" task, the robot must insert a frame into a stand and then hang a tool on the frame. The target system state set X * It is expressed as a logical formula through words. In expressions (6) and (7), s0 is the initial state s0 and G is the target formula.

[0066]

[0067] G=Attached(frame,stand)∧Attached(tool,frame)∧Empty() (7)

[0068] Planned actions a are represented using action patterns. An action pattern is defined by: 1) a name, 2) a list of parameters, 3) a list of static (non-fluent) text constraints (con) that valid parameter values ​​satisfy, 4) a list of fluent text preconditions (pre) that must hold for the action to execute correctly, and 5) a list of fluent text effects (eff) that specify state changes. A move action propels the robot from configuration q1 to q2 via trajectory τ. The constraint Motion(q1,τ,q2) is satisfied if q1 and q2 are the start and end of τ. In a pick action, the constraint Grasp(o,g) holds if g is a valid grasp of object o, and the constraint Pose(o,p) holds if p is a valid placement of object o. The explicit constraint f(q)*g=p represents the kinematics, i.e., the forward kinematics f from configuration q multiplied by the grasp g produces pose p.

[0069] move(q1,τ,q2):

[0070] con:[Motion(q1,τ,q2)]

[0071] pre:[AtConf(q1),Safe(τ)]

[0072]

[0073] pick(o,g,p,q):

[0074] con:[Grasp(o,g),Pose(o,p),[f(q)*g=p]]

[0075] pre:[AtPose(o,p),Empty(),AtConf(q)]

[0076]

[0077] To account for teleoperation by a human operator during mission planning, an approximate model of the teleoperation process can be used. Constraint learning from TAMP can be used by specifying an action pattern for each skill that identifies which constraints can be modeled using classical techniques (using the TAMP system). The remaining constraints can then be extracted from a small number of teleoperation trajectories (e.g., a set of human demonstrations 610). Frame insertion and tool hanging can be teleoperated in the "Tool Hanging" task.

[0078] The attach action models any skill that involves attaching a movable object to another object (e.g., by placing, inserting, or hanging). Its parameters are the object being held o, the current grasp of o g, the corresponding current pose of o p, the current robot configuration q, and the subsequent pose of o Subsequent robot configuration and the object o' to attach to. The attaching action is random because the human teleoperator "selects" the resulting pose and configuration It is constrained by Rather than explicitly modeling this constraint, it is instead modeled by assuming that humans produce a satisfactory Yes, to be optimistic about the outcome without committing to a specific value.

[0079]

[0080] The key constraints in this example are If the object o in posture p and the object o' satisfy the additional condition of the truth target in G, then is true. The task of the human teleoperator is to achieve a pose that satisfies this constraint This is a post-condition for the action. A set of human demonstrations, such as set of human demonstrations 610, can be used to learn to promote a human operator to achieve the pose prerequisite.

[0081] To complete the action model, AttachGrasp and PreAttach constraints are learned, which involve parameters in the attached preconditions. These constraint models can be guided from a few (~3) human demonstrations, such as a set of human demonstrations 610. These human demonstrations only need to show the actions involved, which are only a small part of the task. But through compositionality, these actions can be deployed in many new tasks without retraining. In some examples where the set of objects is fixed, a pose distribution can be learned per task and object. In some examples where there are new objects at test time, these affordances between objects can be estimated directly from the observations. If p is implemented immediately after the human operator PeAttach(o,p,o′) is true if the robot grasps the object o before GoodAttach. For each human demonstration, the system starts from the first state that satisfies GoodAttach and then searches backward in time for the first state where (1) the robot grasps the object o and (2) objects o and o′ are at least δ cm apart. This minimum distance constraint ensures that o and o′ are not touching in a way that is spatially consistent and robust to perception and control errors. The relative pose p between o and o′ can be recorded as a data point. This process can be iterated on human demonstrations to populate the dataset Similarly, if g is a function that allows one to achieve AttachGrasp(o,g) is true if the human grasps the object o. Not all object grasps enable a human to satisfy the goal condition, e.g., a frame grasp on a tip that needs to be inserted. Similar to PreAttach, for each human demonstration, the relative pose between the robot end-effector and the object o in the first pre-contact state before satisfying GoodAttach can be recorded as a data point, resulting in a dataset G. o ={g|AttachGrasp(o,g)}.

[0082] Figure 7 is a flowchart illustrating the operation of an example method 700 for training a machine learning model according to at least some embodiments of the present disclosure. The method 700 may be performed by Figure 1 System 100 or Figure 1The method 700 may be performed by one or more components of the system 100. The method 700 may include more, fewer, or different operations than shown. The operations may be performed in the order shown, in a different order, or simultaneously.

[0083] At 710, the task to be performed by the robot is divided into a plurality of segments. Figure 2 As discussed, the task can be segmented into multiple segments. The task can be segmented into multiple segments based on the actions to be taken with respect to different objects. The task can be segmented based on the type of motion required by the robot within the segment. The task can be segmented based on the segments controlled by the TAMP and the segments controlled by the human.

[0084] At 720, a first instruction set of a plurality of instruction sets is determined for operating a robot to perform a first goal of a first segment in a segment. The plurality of instruction sets may include coordinates and vectors for controlling the robot. The plurality of instruction sets may include predetermined skills for performing an action or goal.

[0085] At 730, it is determined that the plurality of instruction sets are insufficient to perform a second objective of a second segment in the segment. The second objective may include moving an object from a first pose to a second pose. The plurality of instruction sets may be insufficient to perform the second objective based on the plurality of instruction sets being insufficient to control the robot to move the object from the first pose to the second pose without contacting the second object. The plurality of instruction sets may be insufficient to perform the second objective based on the plurality of instruction sets being insufficient to control the robot to move the object from the first pose to the second pose within a predetermined time period. The plurality of instruction sets may be insufficient to perform the second objective based on the plurality of instruction sets being insufficient to control the robot to move the object from the first pose to the second pose with a predetermined level of accuracy.

[0086] At 740, a request for a human demonstration of the second segment may be sent to the operator's user device. Figure 5 A queuing system such as queuing system 500 of can send a request for a human presentation of a second segment to a user device. The second segment can be added to a queue of segments for which presentation is requested. The queue of segments can be associated with a human operator. The second segment can be added to the queue of segments based at least on an expected throughput of the human operator. As discussed herein, the second segment can be added to the queue of segments based on a duty cycle of the human operator.

[0087] The first goal of the first section may include achieving a transition gesture for transitioning between the first section and the second section. The transition gesture may be determined by mapping the sections to a human demonstration of the task and identifying one or more gestures of the robot at a portion of the human demonstration corresponding to the transition from the first section to the second section. The transition gesture may be determined by identifying one or more constraints for transitioning between the first section and the second section. The transition gesture may be determined by identifying one or more gestures according to which a human operator may achieve the second goal of the second section. For example, the transition gesture may include a grasping gesture for a tool to be suspended from a structural member, as discussed herein in the tool suspension example.

[0088] A segment may include multiple transition gestures for transitioning to a segment associated with a human demonstration. In one example, each TAMP-controlled segment may include a transition gesture for transferring control to a human operator. Multiple transition gestures may be predefined. Multiple transition gestures may be predefined with variances and / or tolerances.

[0089] At 750, a second set of instructions for operating the robot for a second segment after the end of the first segment is received from the user device. The human operator may input the second set of instructions in real time as the human operator controls the robot to execute the second segment and execute the objectives of the second segment. The user device may receive data from the robot to facilitate input of the second set of instructions. In one example, the user device displays one or more camera feeds from the robot and receives movement instructions from the user device. In one example, the user device is a smartphone that displays a camera feed from the robot on a display screen of the smartphone, and when the human operator moves the smartphone in a 3D space with six degrees of freedom, it receives a second set of instructions for transmission to the robot.

[0090] At 760, a machine learning model for controlling the robot is updated using a second instruction set for the second segment. The machine learning model can be updated using the second instruction set so that the machine learning model can perform the objective of the second segment. The machine learning model can be trained using one or more instruction sets including the second instruction set. In some implementations, multiple machine learning models can be trained to perform the objectives of various segments of the task. In some implementations, a single machine learning model can be trained to perform various segments of the task.

[0091] Method 700 can be used to train one or more machine learning models to perform various robot-performed tasks. By combining planning methods such as the TAMP method with imitation learning using human demonstrations, method 700 can be applied to efficiently train machine learning models to perform a variety of long-span, touch-intensive tasks.

[0092] In one example, the robot must stack three randomly placed cubes. The task consists of four segments in total. The TAMP system is responsible for grabbing each cube and approaching the stack, while a human (or once trained, a machine learning model) is responsible for placing two cubes on top of the stack. The cubes can be initialized in any position on the work surface.

[0093] In one example, the robot must pick up a nut and place it on a pin. The nut is initialized in a small area and the pin never moves. This task consists of two segments. The TAMP system grabs the nut and approaches the pin, and a human (or, once trained, a machine learning model) inserts the nut on the pin. The nut and pin can be initialized anywhere on the work surface.

[0094] In one example, the robot must pick up a coffee box, insert it into a coffee machine, and then put the lid on. The coffee box starts at a random location in a small box-shaped area, and the coffee machine is fixed. The task has two sections. The TAMP system grabs the coffee box and approaches the coffee machine, and the human (or once trained, the machine learning model) inserts the coffee box and puts the lid on. In some implementations of this example, the coffee box and the coffee machine have significantly larger initialization areas. There is a 50% probability that the coffee box is placed on the left side of the table and the machine is placed on the right side, or vice versa. Once a side is selected for each, the machine position and box position will be further randomized in significant areas.

[0095] In one example, a robot must assemble a structure by inserting a first part into a base and then placing a second part on top of the first. The two parts are placed around the base, which never moves. The task consists of four segments. The TAMP system grabs each part and approaches the insertion point of each of the two parts while a human (or once trained, a machine learning model) handles each insertion. The two parts can be placed anywhere in a predefined workspace.

[0096] In one example, the robot must insert an L-shaped hook into a base piece to assemble a frame, and then twist a wrench off the frame. The L-shaped hook and the wrench have slightly different poses, and the base never moves. The task has four segments. The TAMP system handles grasping the L-shaped hook and wrench and approaching the insertion / hanging point, while a human (or once trained, a machine learning model) handles the insertion. All three parts can be initialized in any position in a predefined workspace.

[0097] The machine learning model updated in method 700 can be trained in a variety of ways. In some implementations, the machine learning model is trained on low-dimensional observations or image observations. This provides flexibility, which is advantageous because training the machine learning model on low-dimensional observations or image observations reduces the perception burden of deploying the TAMP system in the real world. The low-dimensional observations include true object postures, and the image observations include RGB images from the front view camera and the wrist-worn camera. Both observations include proprioception (end effector posture and gripper finger width). In simulation, the image resolution can be 84*84, while in real-world tasks, for Stack Three, Coffee and Coffee Broad, a resolution of 120*160 can be used, and for Tool Hang, a resolution of 240*240 can be used. Real-world agents can all be image-based to avoid the need for motion tracking of objects in real-world applications. The TAMP system can estimate posture at the beginning of each episode or when performing an iteration of a task. A simple perception pipeline can be used, including RANSAC plane estimation for segmenting tables from point clouds, DBSCAN for clustering objects, color-based statistics for associating objects, and iterative closest point (ICP) for estimating object poses. For image-based agents, pixel shift randomization (up to 10% in each image dimension) can be applied as a data augmentation technique.

[0098] With an increased learning rate of 10 -3 The BC-RNN with default hyperparameters can be used to train policies on low-dimensional observations to train policies from segments of human demonstrations in each dataset.

[0099] The present disclosure may be described in the general context of computer code or machine-usable instructions, including computer-executable instructions (e.g., program modules) executed by a computer or other machine (e.g., a personal data assistant or other handheld device). Typically, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. The present disclosure may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. The present disclosure may also be practiced in a distributed computing environment, where tasks are performed by remote processing devices linked through a communications network.

[0100] As used herein, the expression "and / or" about two or more elements should be interpreted as meaning only one element or a combination of elements. For example, "element A, element B, and / or element C" can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, element B and element C.

[0101] The subject matter of the present disclosure is specifically described herein to meet statutory requirements. However, the description itself is not intended to limit the scope of the present disclosure. On the contrary, the inventors have contemplated that the claimed subject matter may also be embodied in other ways to include steps different from the steps described in this document or combinations of similar steps, as well as other current or future technologies. In addition, although the terms "step" and / or "box" may be used herein to refer to different elements of the method employed, unless the order of the various steps is explicitly described, these terms should not be interpreted as implying any particular order between the various steps disclosed herein.

Claims

1. A processor, comprising: One or more circuits for using one or more machine learning models to perform at least a first segment of a task corresponding to a robot and a second segment of the task, wherein the one or more machine learning models are trained to perform the first segment using imitation learning, and the one or more machine learning models are trained to perform the second segment using a method other than the imitation learning.

2. The processor of claim 1, wherein the first section of the task and the second section of the task are performed based on at least coordinates and vectors associated with control of the robot.

3. The processor of claim 1 , wherein the one or more machine learning models are trained to execute the first segment using the imitation learning based at least on a determination that a method other than the imitation learning has not successfully learned to execute the first segment.

4. The processor of claim 3, wherein determining that the method other than imitation learning was unsuccessful is based at least on the robot being unable to move the object from the first pose to the second pose within a predetermined time period.

5. The processor of claim 3, wherein determining that the method other than imitation learning was unsuccessful is based at least on the robot being unable to move a first object from a first pose to a second pose without contacting a second object.

6. The processor of claim 3, wherein determining that the method other than imitation learning was unsuccessful is based at least on an inability of the robot to move the object from the first pose to the second pose with a predetermined level of accuracy.

7. The processor of claim 1, wherein for the second section, no human demonstration is requested. 8 . The processor of claim 1 , wherein a transition gesture between the first segment and the second segment is determined, the transition gesture corresponding to an end or a start of imitation learning.

9. A processor as described in claim 1, wherein an indication is sent to a computing device to indicate that human demonstration is required for the imitation learning based at least on a determination that methods other than imitation learning have not successfully trained the one or more machine learning models to perform the first segment.

10. A system comprising: One or more processing units for: Divide the tasks to be performed by the robot into segments; determining a first set of instructions among a plurality of instructions for operating the robot to perform a first goal of a first segment in the segments; determining that the plurality of instruction sets are insufficient to execute a second target of a second section of the sections; sending a request to a user device of an operator for a human presentation of the second segment; receiving from the user device a second set of instructions for operating the robot for the second segment after the first segment ends; and One or more parameters of a machine learning model used to control the robot are updated using the second set of instructions for the second section.

11. The system of claim 10, wherein the plurality of instruction sets include coordinates and vectors for controlling the robot.

12. The system of claim 10, wherein the second goal comprises moving the object from a first pose to a second pose.

13. The system of claim 12, wherein the plurality of instruction sets are insufficient to perform the second target based at least on the plurality of instruction sets being insufficient to perform at least one of: controlling the robot to move the object from the first posture to the second posture within a predetermined time period; controlling the robot to move the object from the first posture to the second posture without contacting the second object; or The robot is controlled to move the object from the first pose to the second pose with a predetermined level of accuracy.

14. The system of claim 10, wherein the one or more processing units are to send a request for a human presentation of the second segment by adding the second segment to a queue of segments for which presentation is requested.

15. The system of claim 10, wherein the queue of segments is associated with a human operator, and wherein the one or more processing units are configured to add the second segment to the queue of segments based at least on an expected throughput of the human operator.

16. The system of claim 10, wherein the first goal of the first segment includes achieving a transition pose for transitioning between the first segment and the second segment.

17. The system of claim 16, wherein the transition gesture is determined at least in part by: mapping the segments to human demonstrations of the task; and One or more gestures of the robot at a portion of the human demonstration corresponding to a transition from the first segment to the second segment are identified.

18. The system of claim 16, wherein the segment includes a plurality of transition gestures for transitioning to a segment associated with a human presentation.

19. The system of claim 10, wherein the system is included in at least one of: control systems for autonomous or semi-autonomous machines; Perception systems for autonomous or semi-autonomous machines; A system for performing simulation operations; Systems for performing digital twin operations; A system for performing light transport simulations; A system for performing collaborative content creation of 3D assets; Systems for performing deep learning operations; A system for generating or presenting at least one of augmented reality content, virtual reality content, or mixed reality content; A system for hosting one or more real-time streaming applications; Systems implemented using edge devices; Systems implemented using robots; Systems for performing conversational AI operations; A system implementing one or more language models; A system implementing one or more large language models (LLMs); A system for performing one or more generative AI operations; Systems for generating synthetic data; A system comprising one or more virtual machines VM; A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

20. A method comprising: Use one or more machine learning models to perform at least a first segment of a task corresponding to a robot and a second segment of the task, wherein the one or more machine learning models are trained to perform the first segment using imitation learning, and the one or more machine learning models are trained to perform the second segment using a method other than imitation learning.

Images