Neural network for generating robot task demonstrations

Abstract

The present disclosure relates to neural networks for generating robot task demonstrations. One technique for training a neural network includes generating a plurality of input vectors based on a first plurality of task demonstrations associated with a first robot performing a first task in a simulated environment, where each input vector included in the plurality of input vectors specifies a sequence of poses of an end effector of the first robot, and training the neural network to generate a plurality of output vectors based on the plurality of input vectors. Another technique for generating a task demonstration includes generating a simulated environment including a robot and at least one object, causing the robot to at least partially perform a task associated with the at least one object in the simulated environment based on a first output vector generated by the trained neural network, and recording demonstration data of the robot at least partially performing the task in the simulated environment.

Description

[0001] Cross-references to related applications

[0002] This disclosure claims the benefit of U.S. Provisional Patent Application No. 63 / 291205, filed December 17, 2021, entitled "Techniques for Implementing Transformers to Collect Robotic Task Demonstrations". The subject matter of the relevant application is incorporated herein by reference. Technical Field

[0003] The embodiments of this disclosure generally relate to machine learning and robotics, and more specifically, to implementing neural networks to generate robotic task demonstrations. Background Technology

[0004] Robots can automatically perform physical tasks in various real-world environments, such as industrial, home, or nursing settings. Typically, machine learning techniques are used to train robots to perform these physical tasks in these types of real-world environments. For example, machine learning techniques can be used to train robots to automatically perform assembly tasks, tasks involving picking and placing objects, packaging tasks, object stacking tasks, and / or other similar physical tasks. Many machine learning implementations involve training neural networks that enable robots to perform specified tasks without human input and / or human-robot interaction.

[0005] Training a neural network for a given task typically requires a large training dataset, including a sufficient number of robot demos performing the task. Often, these robot demos are manually generated by a user in a simulated environment using a simulated robot. To generate each demo, the user must manually control the simulated robot and successfully complete the given task. Therefore, traditional techniques for generating enough robot demos to train a neural network can be tedious, inefficient, and / or difficult to scale.

[0006] Furthermore, techniques for training neural networks using robot task demos are typically based on configuration parameters for a specific simulator and robot, which serve as training data for generating robot task demos. Therefore, traditional techniques for training neural networks generally only allow the use of a single type of simulator and a single type of robot to generate robot task demos for training the neural network. This restriction on the type of simulator and robot used to generate robot task demos effectively limits or prevents the ability to collect robot task demos from multiple users who can implement different types of simulators and different types of robots. This further increases the workload and efficiency of generating and collecting large numbers of robot task demos for training neural networks.

[0007] As mentioned above, what is needed in the art is a more efficient technique for training neural networks and for collecting robot task demonstrations used to train neural networks. Summary of the Invention

[0008] One embodiment of the present invention illustrates a technique for training a neural network to perform a robotic task. The technique includes generating multiple input vectors based on a first plurality of task demonstrations associated with a first robot performing a first task in a simulated environment, wherein each of the multiple input vectors specifies a pose sequence of the first robot's end-effector. The technique also includes training the neural network to generate multiple output vectors based on the multiple input vectors.

[0009] Compared to existing technologies, one technical advantage of this disclosure is that, using the disclosed technique, a neural network enabling a simulated robot to perform a given task can be trained based on the posture of the robot's end effector as demonstrated in a robot task demonstration. Therefore, unlike conventional methods, using the disclosed technique, the neural network is not trained based on configuration parameters associated with the simulator and / or the simulated robot performing the given task in the robot task demonstration. Thus, robot task demonstrations for training the neural network can be generated using any type of simulator and / or any type of simulated robot. Therefore, the disclosed technique makes it easier and more efficient to generate and collect robot task demonstrations for training the neural network than using conventional techniques. These technical advantages provide one or more technical improvements compared to existing methods.

[0010] One embodiment of the present invention illustrates a technique for generating robot task demonstrations. The technique includes generating a simulated environment comprising a robot and at least one object via a simulator engine. The technique also includes generating a first output vector based on a trained neural network, enabling the robot to at least partially perform a task associated with at least one object in the simulated environment. The technique further includes recording demonstration data of the robot at least partially performing the task in the simulated environment.

[0011] Compared to existing technologies, one technical advantage of the disclosed technology is that, through the disclosed technology, the simulator implements a trained neural network that enables the simulated robot to automatically perform tasks, thereby generating robot demonstrations of those tasks. Another technical advantage is that, utilizing the disclosed technology, the simulator displays trajectory predictions that graphically illustrate the posture of the robot's end effector in a future time-step sequence. Based on the trajectory predictions, the user can interrupt the simulated robot's automatic operation, provide user input for manual operation, and then allow the simulated robot's automatic operation to resume. In this way, the disclosed technology provides automatic and / or automated assisted operation of the simulated robot when performing tasks requiring only minimal (if any) manual user intervention. Therefore, the disclosed technology makes it easier and more efficient to generate and collect robot task demonstrations for training neural networks than using conventional techniques. These technical advantages provide one or more technical improvements compared to existing methods. Attached Figure Description

[0012] To gain a more detailed understanding of the above-described features of the various embodiments, the inventive concepts briefly summarized above can be described in more detail by referring to various embodiments (some of which are shown in the accompanying drawings). However, it should be noted that the drawings illustrate only typical embodiments of the inventive concepts and should not be construed as limiting the scope in any way, and that other equally effective embodiments exist.

[0013] Figure 1 A computing device configured to implement one or more aspects of various embodiments is shown.

[0014] Figure 2 According to various embodiments Figure 1 More detailed descriptions of the training engine, execution engine, and simulator engine.

[0015] Figure 3 This is a flowchart of method steps for training a converter according to various embodiments.

[0016] Figure 4 This is a flowchart of method steps for training multiple dedicated transformers according to various embodiments.

[0017] Figure 5The diagram illustrates various embodiments of the relationship between the two. Figure 2 The transformer is associated with the input vector and the output vector.

[0018] Figure 6 The diagram illustrates trajectory predictions associated with industrial tasks performed by a first simulated robot, according to various embodiments.

[0019] Figure 7 Trajectory predictions associated with industrial tasks performed by a second simulated robot, according to various embodiments, are shown.

[0020] Figure 8 Trajectory predictions associated with household tasks performed by a third simulated robot, according to various embodiments, are shown.

[0021] Figure 9 Trajectory predictions associated with household tasks performed by a fourth simulated robot, according to various embodiments, are shown.

[0022] Figure 10 This is a flowchart illustrating the method steps for generating tasks using a transformer according to various embodiments. Detailed Implementation

[0023] In the following description, numerous specific details are set forth to provide a more thorough understanding of the various embodiments. However, it will be apparent to those skilled in the art that the concepts of the invention can be practiced without one or more of these specific details.

[0024] As used herein, "transformer" refers to a neural network, including transformer-based neural networks. As used herein, "basic transformer" refers to a transformer trained on a large training dataset (numerous robot task demonstrations) to perform one or more tasks. Large training datasets can be generated using first-class simulators and first-class simulated robots. As used herein, "specialized transformer" refers to a basic transformer retrained on a smaller training dataset (fewer robot task demonstrations) relative to the large training dataset used to train the basic transformer to perform a specified task. Smaller training datasets can be generated using second-class simulators and second-class simulated robots.

[0025] As used herein, the converter receives a "converter input vector" as input. The converter input vector includes converter input information for a sequence of past time steps. During the training phase, when the converter is trained, the converter input information includes the state vector and motion vector for each past time step. During the runtime phase, when the trained converter is executed, the converter input information includes the state vector, motion vector, and position vector for each past time step. As used herein, "manual mode converter input information" includes converter input information for a sequence of past time steps collected during manual mode operation of the robot.

[0026] As used in this paper, the “transformer output vector” is generated by the transformer as its output. The transformer output vector includes transformer output information for a sequence of future time steps. When training the transformer during the training phase and executing the training transformer during the runtime phase, the transformer output information includes the state vector and action vector for each future time step.

[0027] As used herein, "execution time step" refers to the simulator's operation of the simulated robot based on time step-based converter output information (in automatic mode) or time step-based user control input (in manual mode). As used herein, "execution time step" refers to the time step that has occurred relative to the current time in the task demonstration.

[0028] As used herein, a “state vector” specifies the state of the robot end effector pose, the state of the end effector grip, and the pose of at least one object in the simulated environment of the robot task demonstration during a specific time step. The state of the end effector pose, end effector grip state, or object pose at a specific time step (t) includes the end effector pose, end effector grip state, or object pose at any point in time within the specific time step (t). In some embodiments, the state of the end effector pose, end effector grip state, or object pose at a specific time step (t) includes the end effector pose, end effector grip state, or object pose at the beginning of the specific time step (t).

[0029] As used herein, a “motion vector” specifies the action of the target pose and target grip state of the robot end effector in a simulated environment demonstrating the robot task during a specific time step. The action of the target end effector pose or target grip state at a specific time step (t) includes the end effector pose or end effector grip state, which will be realized by the next time step (t+1). In some embodiments, the action of the target end effector pose or target grip state at a specific time step (t) includes the end effector pose or end effector grip state at the end of the specific time step (t).

[0030] As used in this paper, a “position vector” for a specific time step specifies the distance and rotation traversed by the robot end effector within a specific time step in the simulated environment of the robot task demonstration.

[0031] As used herein, the “pose” of an end effector or object specifies the position and orientation of the end effector or object, respectively. Position can be specified in three-dimensional (3D) coordinates, and orientation can be specified using four-dimensional (4D) quaternions. Therefore, pose can be specified by a seven-dimensional (7D) vector. In some embodiments, the pose of the end effector includes a global pose defined relative to a fixed reference point / frame in the simulation environment (e.g., the base of a simulated robot). In some embodiments, the pose of the object includes a local pose defined relative to the pose of the robot's end effector.

[0032] As used herein, a “simulator” includes a software application that provides a 3D simulation environment in which a simulated robot performs a task on at least one object included in the 3D simulation environment. For example, a 3D simulation environment may include a virtual reality (VR) environment. A simulator can simulate a specific type of robot based on a set of configuration parameters associated with that specific type of robot, in order to generate and operate a simulated robot within the simulation environment. Different types of robots have different configuration parameters.

[0033] As used in this article, when the simulator is running in "automatic mode", the simulator automatically operates the simulated robot based on the transformer output vector of the transformer.

[0034] As used in this article, when the simulator is running in "manual mode", the simulator operates the robot according to the user's control input.

[0035] As used in this article, when using a combination of automatic and manual modes to operate the simulated robot to successfully complete the task, the simulator is already running in "automatic assist mode".

[0036] Overview

[0037] A training technique is implemented for training a transformer to operate a simulated robot (in conjunction with a simulator) to perform tasks involving interaction with at least one object in a simulated environment using robot task demonstrations. The simulated robot includes an end effector for interacting with at least one object during task performance. The training technique involves training the transformer based on the pose of the robot end effector and the pose of at least one object shown in the robot task demonstration. Each robot task demonstration includes an overall sequence of time steps with all time steps for task performance. During training using the robot task demonstration, a transformer input vector is received via the transformer, which includes transformer input information transformers for past time step sequences contained in the overall time step sequence of the robot task demonstration. For each past time step, the transformer input information includes a state vector and a motion vector. The state vector specifies the pose of the robot end effector, the state of the end effector grip, and the pose of at least one object associated with the past time step. The motion vector specifies the target pose of the robot end effector and the target state of the end effector grip associated with the past time step.

[0038] Using a robotic task demonstration, a transformer is trained to generate a transformer output vector that includes transformer output information based on a predicted / future timestep sequence of input vectors. Throughout the entire timestep sequence of the robotic task demonstration, the predicted / future timestep sequence appears exactly after the past timestep sequence. For each future timestep, the transformer output information includes a state vector and an action vector.

[0039] In some embodiments, the training technique trains the converter without relying on any configuration parameters of the simulator and / or simulated robot used to generate and collect robot task demonstrations. Therefore, when training the converter, the training technique does not consider / take into account the configuration parameters of the simulator and / or simulated robot, and is unaware of the type of simulator and / or simulated robot used to generate the robot task demonstrations. Thus, robot task demonstrations (training datasets) for training the converter can be generated using any type of simulator and / or any type of simulated robot. Therefore, compared to conventional techniques, this training technique makes it easier and more efficient to generate and collect robot task demonstrations for training the converter.

[0040] Initially, training techniques are implemented to train a basic transformer using a large training dataset (a large number of robot task demonstrations) to perform one or more general tasks. In some embodiments, the basic transformer is not trained to perform a specified task. Next, training techniques are implemented to retrain the basic transformer to generate specialized transformers for performing the specified tasks using a smaller training dataset relative to the large training dataset used to train the basic transformer. The specified task associated with a specialized transformer may include tasks not included in the general task group associated with the basic transformer. The smaller training dataset contains fewer robot task demonstrations compared to the large training dataset used to train the basic transformer. Each robot task demonstration in the smaller training dataset contains a demonstration of the robot performing the specified task. The basic transformer can be retrained in this way to generate any number of specialized transformers, each trained to perform a different specified task.

[0041] Robot task demonstrations on both large and small training datasets can be generated using any type of simulator and / or any type of simulated robot. In some embodiments, a first-type simulator and a first-type simulated robot are used to generate robot task demonstrations on a large training dataset, while a second-type simulator and a second-type simulated robot are used to generate robot task demonstrations on a small training dataset. The first-type simulator may differ from the second-type simulator. The first-type simulated robot may differ from the second-type simulated robot.

[0042] Besides being unaware of the simulator and robot types during the training phase, the training technique also allows the dedicated converter to be unaware of the types of simulators and robots used during the execution / running phase. Specifically, when executing a trained dedicated converter in a simulation environment to generate additional robot task demonstrations, any type of simulator and / or robot can be used. In some embodiments, additional robot task demonstrations are generated using a third-type simulator and a third-type simulated robot. The third-type simulator may differ from the first-type simulator and / or the second-type simulator. The third-type simulated robot may differ from the first-type simulated robot and / or the second-type simulated robot.

[0043] In a simulated environment, the execution technique is implemented when a dedicated converter trained for a specified task is executed to collect new / additional robot task demonstrations for that specified task. The simulator implements a trained dedicated converter for automatically maneuvering a simulated robot to perform the specified task, thereby generating a robot demonstration of the specified task. When generating a robot task demonstration containing the entire time-step sequence, the dedicated converter receives a converter input vector containing converter input information for past time-step sequences already executed / completed in the robot task demonstration. For each past time step, the converter input information includes a state vector, a motion vector, and a position vector. The state vector specifies the pose of the robot end effector, the state of the end effector grip, and the pose of at least one object associated with the past time step. The motion vector specifies the target pose of the robot end effector and the target state of the end effector grip associated with the past time step. The position vector specifies the distance traversed and rotation of the robot end effector in the past time step.

[0044] Based on the transformer input vector, a dedicated transformer generates a transformer output vector containing transformer output information for predicting / future timestep sequences that have not yet been executed / completed in the robot task demonstration. The predicted / future timestep sequence appears exactly after the past timestep sequence throughout the robot task demonstration's timestep sequence. For each future timestep, the transformer output information includes a state vector and a motion vector. The simulator receives the transformer output vector from the dedicated transformer and executes the future timestep sequence by manipulating the simulated robot based on the transformer output information of the future timestep sequence. Specifically, for each future timestep, the simulator manipulates the simulated robot in an appropriate manner to achieve the robot's end effector pose and the end effector gripping state specified in the transformer output information of the future timestep. Once executed, the future timestep is contained in the next sequence of past timesteps, which the dedicated transformer receives as input in its next execution process / iteration.

[0045] In some embodiments, the simulator implements multiple execution processes / iterations of a dedicated transformer to automatically operate a simulated robot to perform a specified task. In this way, the trained dedicated transformer can automatically generate and collect a large number of robot task demonstrations without user input / intervention or manual operation of the simulated robot. Therefore, compared to traditional technologies that require extensive manual operation of the simulated robot, the execution technology can generate and collect robot task demonstrations more easily and efficiently.

[0046] When running in automatic mode, the simulator operates the simulated robot based on the converter output vector of a dedicated converter. However, the simulator can also operate in manual mode when operating the simulated robot based on user control input instead of the converter output vector of the dedicated converter. In these embodiments, when the simulator receives the converter output vector from the dedicated converter, it also displays trajectory predictions, including a graphical representation of the future time step sequence specified in the converter output vector. In particular, the trajectory predictions graphically show the pose of the robot's end effector in the future time step sequence. Therefore, if the simulator continues to operate in automatic mode without user intervention, the trajectory predictions graphically illustrate the future poses of the robot's end effector that will occur in the future time steps.

[0047] Based on trajectory prediction, users can decide to set the simulator to manual mode to momentarily intervene in and interrupt / stop the simulated robot's automatic operation. In manual mode, users provide user control input to manually operate the simulated robot, for example, pushing the simulated robot in a direction that the user believes is more suitable for successful task execution than the trajectory prediction. In manual mode, the simulator executes the time-step sequence in the robot demonstration based on the user control input. In manual mode, the simulator also collects transformer input information for the time-step sequence (referred to as "manual mode transformer input information"), including the state vector, motion vector, and position vector for each time step in the sequence.

[0048] Once the user believes the simulated robot is back on track and has successfully completed its task, the user can switch the simulator back to automatic mode to continue simulating the robot's automated operation, such as completing the task, based on the output of a dedicated transformer instead of user control input. Upon returning to automatic mode, the simulator inputs the next transformer input vector to the dedicated transformer, which includes at least a portion of the manual mode transformer input information. Based on this next transformer input vector, the dedicated transformer generates the next transformer output vector when the simulator returns to automatically operating the simulated robot based on the transformer output vector. When a combination of automatic and manual modes is used to operate the simulated robot to successfully complete a specified task, the simulator is referred to as in "automatic-assisted mode."

[0049] Successful robot task demonstrations include those in which a simulated robot has successfully performed a specified task within a predetermined time period. When the specified task is completed entirely in the simulator's automatic mode, robot task demonstrations can be generated using a dedicated transformer with a high success rate. When the specified task is completed in the simulator's automatic-assisted mode, robot task demonstrations can be generated using a dedicated transformer with an even higher success rate than when executed entirely in automatic mode. In this way, the execution technology provides automated and / or automated-assisted operation for the simulated robot to successfully perform tasks that require little or no manual intervention from the user compared to conventional techniques. Therefore, the disclosed technology makes it easier and more efficient to generate and collect robot task demonstrations for training neural networks with less manual input compared to conventional techniques.

[0050] System Overview

[0051] Figure 1 A computing device 100 configured to implement one or more aspects of various embodiments is illustrated. In one embodiment, the computing device 100 includes a desktop computer, laptop computer, smartphone, personal digital assistant (PDA), tablet computer, or any other type of computing device configured to receive input, process data, and optionally display images and adapted to practice one or more embodiments.

[0052] Computing device 100 is configured to execute training engine 122, execution engine 124, and simulator engine 125 residing in memory 116. It is important to note that the computing device described herein is illustrative, and any other technically feasible configuration is within the scope of this disclosure. For example, multiple instances of training engine 122, execution engine 124, and simulator engine 125 may execute on a set of nodes in a distributed and / or cloud computing system to implement the functionality of computing device 100.

[0053] In one embodiment, computing device 100 includes, but is not limited to, interconnect (bus) 112 connecting one or more processors 102, input / output (I / O) device interface 104 coupled to one or more input / output (I / O) devices 108, memory 116, storage 114, and network interface 106. The one or more processors 102 can be any suitable processor implemented as a central processing unit (CPU), graphics processing unit (GPU), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), artificial intelligence (AI) accelerator, any other type of processing unit, or combination of different processing units, such as a CPU configured to run alongside a GPU. Typically, the one or more processors 102 can be any technically feasible hardware unit capable of processing data and / or executing software applications. Furthermore, in the context of this disclosure, the computing elements shown in computing device 100 can correspond to a physical computing system (e.g., a system in a data center) or can be a virtual computing instance performing in a computing cloud.

[0054] Input / output device 108 includes devices capable of receiving input, such as a keyboard, mouse, touchpad, and / or microphone, and devices capable of providing output, such as a display device and / or speaker. Furthermore, I / O device 108 may include devices capable of receiving input and providing output, such as a touchscreen, a Universal Serial Bus (USB) port, etc. I / O device 108 can be configured to receive various types of input from end users of computing device 100 and also provide various types of output to end users of computing device 100, such as displayed digital images, digital video, or text. I / O device 108 may also include virtual reality (VR) device 170, such as VR headsets, VR controllers, and tracking devices.

[0055] In some embodiments, one or more of the I / O devices 108 are configured to couple computing device 100 to network 110. In one embodiment, network 110 is any technically feasible type of communication network that allows data exchange between computing device 100 and external entities or devices (such as a web server or another network computing device). For example, network 110 may include a wide area network (WAN), a local area network (LAN), a wireless (WiFi) network, and / or the Internet.

[0056] In one embodiment, storage 114 includes non-volatile storage for applications and data, and may include fixed or removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other magnetic, optical, or solid-state storage devices. Training engine 122, execution engine 124, and simulator engine 125 may be stored in storage 114 and loaded into memory 116 during execution. Furthermore, storage 114 may store additional robotic task demonstrations 150 generated and collected during the runtime phase of a dedicated converter.

[0057] In one embodiment, memory 116 includes a random access memory (RAM) module, flash memory cells, or any other type of memory cell or a combination thereof. One or more processors 102, I / O device interfaces 104, and network interfaces 106 are configured to read data from memory 116 and write data to memory 116. Memory 116 includes various software programs executable by one or more processors 102 and application data associated with said software programs, including a training engine 122, an execution engine 124, and a simulator engine 125.

[0058] Training engine 122 includes functionality to train a machine learning model on a training dataset (robot task demonstration), and execution engine 124 includes functionality to execute one or more portions of the machine learning model to generate additional data not found in the training dataset (additional robot task demonstration 150). For example, training engine 122 may train multiple neural networks included in the machine learning model on a set of robot task demonstrations, and execution engine 124 may execute a portion of the trained neural networks to generate additional robot task demonstrations 150 not found in the set of robot task demonstrations. In turn, additional robot task demonstrations 150 can be used to retrain the machine learning model and / or other neural networks. In some embodiments, the machine learning model includes a transformer-based neural network (transformer). In other embodiments, the machine learning model includes other types of neural networks implemented to generate robot task demonstrations.

[0059] Simulator 125 operates in conjunction with execution engine 124 to generate and collect additional robot task demonstrations 150 by providing a three-dimensional simulation environment (e.g., a VR environment). This environment includes a simulated robot and at least one simulated object on which the simulated robot performs a specified task. Tasks can include industrial tasks (e.g., stacking blocks, assembling nuts, assembling kits, etc.), domestic tasks (e.g., manipulating bowls, opening drawers, closing doors, etc.), personal care tasks (e.g., assisting with drinking, massaging, scratching, etc.), and any other type of task. Simulated objects can represent any type of object, such as boxes, bowls, furniture, people, animals, etc. In some embodiments, the simulation environment includes a VR environment that a user can interact with via VR device 170, such as a VR headset for displaying the VR environment and a VR controller for manually operating the simulated robot.

[0060] Emulator 125 can include any type of emulator, such as NVIDIA Omniverse. TM Examples of simulators include MuJoCo, pyBullet, Gazebo, etc. In other embodiments, simulator 125 may include any other type of simulator. Each different type of simulator is defined by different configuration parameters, which specify, for example, the control method, data collection rate, simulation method, simulation time step (dt), and the solver or optimizer used to advance the simulation. Such parameters may also include parameters related to the objects / scenes defined in the generic scene description (USD) model, such as physical properties including friction, mass, inertia, etc.

[0061] Simulator 125 receives a set of robot configuration parameters 130 that define the configuration of the simulated robot, including an end effector. The end effector is a part of the robot that interacts with an object in the simulated environment to perform a task associated with that object. An end effector is also called a robot gripper or robot hand. The set of robot configuration parameters 130 defines the model of the simulated robot. The set of robot configuration parameters 130 specifies the robot's physical characteristics, such as its geometry, inertia, and mass. For example, the set of robot configuration parameters 130 can specify the physical characteristics of joints, the connections between joints, and the robot's end effector (e.g., the shape and number of fingers in the end effector). Each different type of simulated robot is defined by a different set of robot configuration parameters 130. For example, a first type of robot is defined by a first set of robot configuration parameters, and a second type of robot is defined by a second set of robot configuration parameters different from the first set. Different types of robots include Baxter, Sawyer, Franka, etc.

[0062] When the simulated robot (in automatic and / or manual mode) performs a specified task in a simulated environment, simulator 125 records demonstration data describing the performance of the specified task in the simulated environment, which is then included in a new robot task demonstration 150 for the specified task. In some embodiments, simulator 125 records demonstration data describing an entire sequence of end effector poses, end effector gripping states, and object poses of at least one object during the complete execution of the specified task, which is included in the new robot task demonstration 150 for the specified task. In other embodiments, simulator 125 records additional data describing the performance of the specified task in the simulated environment included in the new robot task demonstration 150. Each new robot task demonstration 150 can then be stored in a database or data store, such as storage 114.

[0063] Training Transformer

[0064] Figure 2 According to various embodiments Figure 1 A more detailed description of the training engine, execution engine, and simulator engine is provided below. As shown in the figure, the execution engine 124 is coupled to the training engine 122 and the simulator engine 125. The simulator engine 125 is also coupled to the VR device 170.

[0065] Training engine 122 includes training dataset 210 and transformer 200. Training dataset 210 includes multiple robot task demonstrations. Typically, training engine 122 trains transformer 200 to generate transformer output information for future time step sequences based on transformer input information from past time step sequences used for the robot task demonstrations (training dataset 210). In some embodiments, transformer 200 includes a basic transformer trained to control a simulated robot to perform a set of general tasks using training dataset 210. In particular, the basic transformer is trained to assist the simulator in operating the simulated robot to perform a set of general tasks. In other embodiments, transformer 200 includes a dedicated transformer trained to control a simulated robot to perform a specific task using training dataset 210. In particular, the dedicated transformer is trained to assist the simulator in operating the simulated robot to perform a specific task.

[0066] When transformer 200 includes a basic transformer, training dataset 210 includes a large training dataset containing numerous robot task demonstrations for a general set of tasks. When transformer 200 includes a specialized transformer, training dataset 210 includes a smaller training dataset (relative to the large training dataset) containing a smaller number of robot task demonstrations (relative to the large number of robot task demonstrations) for a specific task. Specialized transformers can be generated by retraining (fine-tuning) the basic transformer to perform the specified task using the smaller training dataset. In this way, different specialized transformers can be generated by retraining the basic transformer using different smaller training datasets to perform different specified tasks.

[0067] The training techniques described herein can be used to train basic and / or specialized transformers, regardless of the type of simulator and robot used to collect robot task demonstrations. For example, a large training dataset may include robot task demonstrations collected using first-class simulators and first-class robots. For instance, a large training dataset may include the Roboturk dataset, which includes robot task demonstrations collected using the Mujoco simulator and the Baxter robot. Smaller training datasets may include robot task demonstrations collected using second-class simulators and second-class robots, different from the first-class simulators. For instance, a smaller training dataset may contain demonstrations collected using NVIDIA Omniverse simulators. TM Simulator and Franka robot collection robot mission demonstration.

[0068] During training, training engine 122 trains transformer 200 using training dataset 210, which includes multiple robot task demonstrations. Each robot task demonstration comprises a complete time-step sequence Td. For each robot task demonstration, training engine 122 samples a subsequence of length Ts (e.g., Ts = 400) from the robot task demonstration Td, where Td equals the number of time steps in the robot task demonstration, Ts equals the number of time steps in the sampled subsequence, and Td > Ts.

[0069] Then, the training engine 122 derives transformer input information for the subsequence Ts demonstrating the robot task, including vector pairs (T) containing the state vector (s) and action vector (a) for each time step of the subsequence Ts. The training engine 122 retains the transformer input information for the first Tp time step of the subsequence Ts and zero-masks the transformer input information for the remaining time steps (Ts–Tp) of the subsequence Ts. Therefore, the training engine 122 retains the transformer input information for the first set of initial time steps of the subsequence Ts and zero-masks the transformer input information for the second set of remaining time steps of the subsequence Ts. The first set of initial time steps corresponds to past time steps, and the second set of remaining time steps corresponds to future time steps. The value of Tp can be randomly selected during training to force the transformer 200 to make future predictions of arbitrary length. The transformer input information for the second set of remaining time steps of the subsequence Ts is stored for later use when executing the loss function.

[0070] The converter input vector received by converter 200 as input includes converter input information for the first set of initial time steps of the subsequence Ts. Therefore, the converter input vector includes the sequence / trajectory of state-action pairs for the first set of initial time steps of the subsequence Ts, as shown in equation (1):

[0071] T x ={s x a x} (1)

[0072] The transformer 200 is then trained to generate / predict a transformer output vector that includes transformer output information for the second set of remaining time steps (Ts–Tp) of the masked subsequence Ts. Specifically, the transformer 200 is trained to predict vector pairs (T) that include the state vector (s) and action vector (a) for each time step of the second set of remaining time steps (Ts–Tp). Thus, the transformer output vector comprises the sequence / trajectory of state-action pairs for the second set of remaining time steps of the subsequence Ts, as shown in equation (2):

[0073]

[0074] At each specific time step, the state vector specifies the pose of the robot end effector, the state of the end effector grip, and the pose of at least one object in the robot task demonstration simulation environment during that specific time step. In some embodiments, the end effector pose includes a global pose defined relative to a fixed reference point / frame in the simulation environment (e.g., the base of the simulated robot). In some embodiments, the object pose includes a local pose defined / associated with the robot end effector pose. The state of the end effector pose, end effector grip state, or object pose at a specific time step (t) includes the end effector pose, end effector grip state, or object pose during any point in time within the specific time step (t). In some embodiments, the state of the end effector pose, end effector grip state, or object pose at the beginning of the specific time step (t) includes the end effector pose, end effector grip state, or object pose at the start of the specific time step (t).

[0075] End effector posture (s) r The state of the end effector is determined by the position of the end effector at a specific time step (t). r ) and direction (q) r The position can be specified in three-dimensional (3D) coordinates, and the orientation can be specified using four-dimensional (4D) quaternions. Therefore, the state of the end effector's orientation can be specified by a seven-dimensional (7D) vector, as shown in equation (3):

[0076] s r,t =[p r,t ,q r,t (3)

[0077] End effector gripping state (s g The state of ) is specified as on (0) or off (1) for a specific time step (t), as shown in equation (4):

[0078] s g,t ∈[0,1] (4)

[0079] For a specific object (oi), the object pose (s) oi The state of a specific object is determined by its position (p) at a specific time step (t). oi ) and direction (q) oi The state of an object's pose can be specified by a seven-dimensional (7D) vector, as shown in equation (5):

[0080] so i,t =[p oi,t q oi,t (5)

[0081] At each specific time step, within the simulated environment of the robot task demonstration, the motion vector specifies the motion of the target pose and target grip state of the robot end effector. The motion of the target end effector pose or target grip state at a specific time step t includes the end effector pose or end effector grip state that will be realized in the next time step (t+1). In some embodiments, the motion of the target end effector pose or target grip state at a specific time step t includes the end effector pose or end effector grip state at the end of the specific time step t.

[0082] Target end effector pose (a r The action of the end effector is determined by the target position (p) of the end effector at a specific time step (t). rtarget ) and target direction (q) rtarget Definition. The action of the target end effector pose can be specified by a seven-dimensional (7D) vector, as shown in equation (6):

[0083] a r,t =[p rtarget,t q rtarget,t (6)

[0084] Target end effector gripping state (a g The action of ) is specified as turning on (0) or turning off (1) at a specific time step (t), as shown in equation (7):

[0085] a g,t ∈[0,1] (7)

[0086] The state vector sequence of the first set of time steps in the subsequence Ts specifies the pose of the robot end effector, the state of the end effector grip, and the pose of the J object at each time step, where k represents the last time step in the first set of time steps, as expressed by equation (8):

[0087]

[0088] The sequence of motion vectors for the first set of time steps in the subsequence Ts specifies the target pose of the robot's end effector and the target grasping state of the end effector at each time step, where k represents the last time step in the first set of time steps, as expressed by equation (9):

[0089]

[0090] The first set of initial time step state vector sequences and action vector sequences of the subsequence Ts includes a transformer input vector received as input by transformer 200. Based on the transformer input vector, transformer 200 is trained to generate / predict a transformer output vector that includes transformer output information for the second remaining time step set (future steps) of the subsequence Ts. In some embodiments, based on the transformer input vector, a transformer output vector is predicted using a generative pre-trained transformer (GPT) type transformer that includes a language model predicting the next word in the sequence, as shown in equation (10):

[0091]

[0092] The converter output vector includes a second set of predicted future state vector sequences for the remaining time steps of the subsequence Ts, which specifies the pose of the robot end effector, the state of the end effector grip, and the pose of the J object at each time step, where k+1 represents the first time step, and T represents the last time step in the remaining second set of time steps, as expressed by equation (11):

[0093]

[0094] The converter output vector also includes a second set of predicted future motion vector sequences for the subsequence Ts, which specifies the target pose of the robot end effector and the target grasping state of the end effector at each time step, where k+1 represents the first time step and T represents the last time step in the remaining second set of time steps, as expressed by equation (12):

[0095]

[0096] To train transformer 200, training engine 122 implements a loss function that compares the predicted output of transformer 200 with the expected / correct output. Specifically, before zero-masking the transformer input information, the loss function compares the transformer output vector (predicted) for the second set of remaining time steps of the specified subsequence Ts with the actual / correct transformer input information for the second set of remaining time steps of the subsequence Ts. The actual / correct transformer input information for the second set of remaining time steps of the subsequence Ts is stored for use by the loss function before zero-masking.

[0097] The loss function generates a scalar loss value that indicates the training progress of transformer 200. For example, if the predicted output from transformer 200 is very close to the expected / correct output, the loss value is close to zero, indicating that transformer 200 has learned how to accurately map the transformer input vector to the transformer output vector. If the loss value is not zero or falls below a predetermined threshold, training engine 122 adjusts the parameters of transformer 200 until the loss value becomes zero or falls below the predetermined threshold. For example, training engine 122 can use gradient descent and backpropagation of the loss function to adjust the parameters of transformer 200 to update the weights in the encoder and decoder neural networks of transformer 200 in a way that reduces losses.

[0098] The loss function calculates the loss associated with the end effector's pose (position and orientation) state. To this end, for each time step of the predicted output, the end effector pose state is mapped to the eight corners of the 3D bounding box surrounding the end effector, as shown in Equation (13):

[0099]

[0100] For each corresponding time step of the expected output, the state of the end effector pose is also mapped to the 8 corners of the end effector bounding box, where c sr ∈R8×3, as shown in equation (14):

[0101]

[0102] Then, the loss associated with the end effector's attitude state is calculated as the sum of the L2 distances (Euclidean distances) of the corresponding eight angles of the predicted output and the expected output, as shown in equation (15):

[0103]

[0104] Therefore, the loss (Ls) associated with the end effector's attitude state r It includes a one-dimensional scalar value that takes into account the L2 distance of all eight corresponding angles of the predicted output and the expected output.

[0105] The loss function also includes the loss (Ls) related to the calculation of the end effector's attitude state. r The loss associated with the target end effector's posture and motion is calculated in a similar manner (La). r Specifically, for each time step of the predicted output, the action of the target end effector pose is still mapped to the 8 corners of the end effector bounding box. For each corresponding time step of the expected output, the action of the target end effector pose is also mapped to the 8 corners of the end effector bounding box. Then, the loss associated with the action of the target end effector pose (La) is calculated. rThe L2 distance (Euclidean distance) is calculated as the sum of the corresponding eight angles of the predicted output and the expected output. Therefore, the loss associated with the action of the target end effector posture (La) r It includes a one-dimensional scalar value that takes into account the L2 distance of all eight corresponding angles of the predicted output and the expected output.

[0106] The loss function also includes the loss (Ls) related to the calculation of the end effector's attitude state. r The loss (Ls) associated with the pose state of object i is calculated in a similar manner. oi Specifically, for each time step of the predicted output, the object pose state is mapped to the 8 corners of the object bounding box. For each corresponding time step of the expected output, the object pose state is also mapped to the 8 corners of the object bounding box. The loss associated with the object pose state is then calculated as the sum of the L2 (Euclidean) distances between the corresponding 8 corners of the predicted and expected outputs. Therefore, the loss associated with the pose state of object i (Ls...) is... oi This includes a one-dimensional scalar value considering the L2 distances of all eight corresponding angles of the predicted and expected outputs. Then, the total loss of the J-wide object states is summed, as shown in equation (16):

[0107]

[0108] The loss function also calculates the loss (Ls) associated with the end effector's gripping state. g Therefore, the loss function uses binary cross-entropy to calculate the loss in the gripping state, as shown in equation (17):

[0109]

[0110] The loss function also calculates the loss associated with the action of the target end effector's gripping state (La). g Therefore, the loss function uses binary cross-entropy to calculate the loss of the grasping action state, as shown in equation (18):

[0111]

[0112] Then, the loss function sums all the losses calculated above to determine the total loss (L) between the predicted output and the expected / correct output of the converter 200. TOTAL As shown in equation (19):

[0113]

[0114] As described above, the training engine 122 trains the transducer 200 based on the pose of the robot's end effector, the end effector gripping state, and the pose of at least one object used to perform the task, as shown in the robot task demonstration (training dataset 210). In some embodiments, the transducer 200 is not trained based on any configuration parameters of the simulator and / or simulated robot used to generate the robot task demonstration. Therefore, the characteristics and configuration of the simulator and / or simulated robot are abstracted and are independent of the training of the transducer 200. Thus, robot task demonstrations (training dataset 210) can be generated using any type of simulator and / or any type of simulated robot. Furthermore, additional robot task demonstrations can be generated using any type of simulator and / or robot during the execution of the transducer 200.

[0115] Figure 3 This is a flowchart of method steps for training a transformer according to various embodiments. Although combined... Figure 1-2 The system describes the method steps, but those skilled in the art will understand that any system configured to execute the method steps in any order is within the scope of this invention. Method 300 is executed by training engine 122, which trains a transformer 200 for controlling a simulated robot to perform one or more tasks using a training dataset 210. Specifically, the transformer 200 is trained to assist the simulator in operating the simulated robot to perform one or more tasks. In some embodiments, the transformer 200 includes a basic transformer, and the training dataset 210 includes a large dataset. In other embodiments, the transformer 200 includes a dedicated transformer, and the training dataset 210 includes a smaller dataset (relative to a large dataset). The training dataset 210 includes multiple robot task demonstrations of the simulated robot performing one or more tasks (e.g., industrial tasks, household tasks, and / or personal care tasks). The multiple robot task demonstrations can be generated from one or more different types of simulators 125 and one or more different types of robots.

[0116] As shown in the figure, in step 310, the training engine 122 samples a current subsequence of length Ts from the robot task demonstration within the training dataset 210. The robot task demonstration includes the overall sequence of time steps Td, and the training engine 122 can sample a subsequence of length Ts (e.g., Ts = 400) from the robot task demonstration Td, where Ts is equal to the number of time steps in the sampled subsequence Ts.

[0117] In step 320, the training engine 122 derives the transformer input information for the current subsequence Ts. The transformer input information for the current subsequence Ts includes a vector pair (T), which comprises a state vector (s) and an action vector (a) for each time step of the subsequence Ts. At each specific time step, the state vector specifies the pose of the robot end effector, the state of the end effector grip, and the state of the pose of at least one object in the simulated environment during that specific time step. The end effector pose (s) r The state of the end effector is determined by the 3D position (p) of the end effector at a specific time step (t). r ) and 4D direction (q r The definition is shown in equation (3). For a specific time step (t), the end effector gripping state (s) is... g The state of ) is specified as on (0) or off (1), as shown in equation (4). For object (oi), object pose (s) oi The state of an object is determined by its 3D position (p) at a specific time step (t). oi ) and 4D direction (q oi The target end effector pose is defined as shown in equation (5). At each specific time step, the motion vector specifies the action of the target pose and the target grasping state of the robot end effector in the simulated environment during that specific time step. r The action of the end effector is determined by the 3D target position (p) of the end effector at a specific time step (t). rtarget ) and 4D target direction (q rtarget The target end effector gripping state (a) is defined as shown in equation (6). g The action is specified as opening (0) or closing (1) for a specific time step (t), as shown in equation (7).

[0118] In step 330, the training engine 122 retains the transformer input information for the first set of initial time steps Tp of the current subsequence Ts, and zero-masks the transformer input information for the second set of remaining time steps (Ts–Tp) of the current subsequence Ts. The first set of initial time steps corresponds to past time steps, and the second set of remaining time steps corresponds to future time steps. The transformer input information for the second set of remaining time steps of the current subsequence Ts is stored for later use when executing the loss function.

[0119] In step 340, the training engine 122 inputs a transformer input vector to the transformer 200, which includes transformer input information for the first set of initial time steps of the current subsequence Ts. The transformer input vector includes a sequence of state-action pairs for the first set of initial time steps, as shown in Equation (1). The sequence of state vectors for the first set of time steps of the current subsequence Ts specifies the pose of the robot end effector, the state of the end effector grip, and the pose of the J object at each time step, as shown in Equation (8). The sequence of action vectors for the first set of time steps of the current subsequence Ts specifies the target pose of the robot end effector and the target state of the end effector grip at each time step, as shown in Equation (9).

[0120] In step 350, training engine 122 receives a transformer output vector from transformer 200 based on the transformer input vector. Typically, transformer 200 is trained to generate / predict a transformer output vector that includes transformer output information for a second set of remaining time steps of the current subsequence Ts, based on transformer input information for a first set of initial time steps of the current subsequence Ts. Specifically, as shown in equation (2), transformer 200 is trained to predict vector pairs (T) that include state vectors (s) and action vectors (a) for each time step of the second set of remaining time steps. In some embodiments, a transformer of the Generative Pre-trained Transformer (GPT) type is used to predict the transformer output vector based on the transformer input vector, as shown in equation (10). The transformer output vector includes a sequence of predicted future state vectors for the second set of remaining time steps, which specifies the pose of the robot end effector, the state of the end effector grip, and the pose of the J object for each time step, as shown in equation (11). The converter output vector also includes a second set of time-step predicted future motion vector sequences, which specify the target pose of the robot end effector and the target state of the end effector grip at each time step, as shown in Equation (12).

[0121] In step 360, training engine 122 updates the parameters of transformer 200 based on the loss between the predicted output and the expected / correct output from transformer 200. Specifically, training engine 122 implements a loss function to compare the transformer output vector (predicted output) specifying the transformer input information for the second set of remaining time steps of the current subsequence Ts with the actual transformer input information (expected output) for the second set of remaining time steps of the current subsequence Ts (before the transformer input information is zero-masked). For example, the loss function can compute a loss value that includes L2 loss and / or another error metric between the predicted output and the expected / correct output of transformer 200, as shown in equations (13)-(19). Training engine 122 can use gradient descent and backpropagation of the loss function to adjust the parameters of transformer 200, updating the weights in the encoder and decoder neural networks of transformer 200 in a way that reduces the loss value.

[0122] In step 370, training engine 122 determines whether training of transformer 200 is complete. For example, training engine 122 may determine training is complete when one or more conditions are met. These conditions include (but are not limited to) convergence of the parameters of the encoder and decoder neural networks of transformer 200, or reduction of the loss value below a predetermined threshold. If training of transformer 200 is not complete, method 300 continues in step 310, whereby training engine 122 samples the next subsequence of length Ts from the current or next robot task demonstration within training dataset 210. In this way, method 300 can train transformer 200 by generating multiple input vectors based on multiple task demonstrations of a robot performing tasks in a simulated environment, where each input vector at least specifies a pose sequence of the robot's end effector, and train the neural network to generate multiple output vectors based on said multiple input vectors until training is complete. If training of transformer 200 is determined to be complete, method 300 then terminates.

[0123] As described above, the training engine 122 trains the transducer 200 based on the pose of the robot's end effector, the end effector gripping state, and the pose of at least one object shown in the task demonstration (training dataset 210) for performing one or more tasks. In some embodiments, the transducer 200 is not trained based on any configuration parameters of the simulator and / or simulated robot used to generate the task demonstration. Therefore, robot task demonstrations (training dataset 210) can be generated using any type of simulator and / or any type of simulated robot.

[0124] Figure 4 This is a flowchart illustrating the method steps for training multiple dedicated converters according to various embodiments. Although combined... Figure 1-2The system describes the method steps, but those skilled in the art will understand that any system configured to perform the method steps in any order is within the scope of this invention.

[0125] As shown in the figure, in step 410, training engine 122 accesses training dataset 210, which includes a large dataset of robot task demonstrations, such as the Roboturk dataset. Robot task demonstrations can be collected using any type of simulator and any type of robot, such as first-class simulators and first-class robots. Large datasets include task demonstrations of one or more tasks, such as industrial, domestic, and / or care tasks. In step 420, training engine 122 uses... Figure 3 The training technique shown trains a basic transformer to control a simulated robot to perform one or more tasks using a large dataset.

[0126] In step 430, training engine 122 accesses training dataset 210, which includes a first small dataset containing a smaller number of robot task demonstrations compared to the large dataset. The first small dataset includes robot task demonstrations collected using any type of simulator and any type of robot, such as a second type of simulator different from the first type of simulator and / or a second type of robot different from the first type of robot. The first small dataset includes robot task demonstrations for a first specified task, such as an industrial, domestic, or care task. In step 440, training engine 122 uses... Figure 3 The training technique illustrated generates a first dedicated transformer by retraining a basic transformer to control a simulated robot to perform a first specified task using a first small dataset. In some embodiments, the task group associated with the basic transformer includes one or more tasks, and the first specified task is not included in one or more task groups. In other embodiments, the task group associated with the basic transformer includes two or more task groups, and the first specified task is included in two or more task groups.

[0127] In step 450, training engine 122 accesses training dataset 210, which includes a second sub-dataset containing a smaller number of robot task demonstrations relative to the large dataset. The second sub-dataset includes robot task demonstrations collected using any type of simulator and any type of robot, such as a third type of simulator and / or a third type of robot different from the first and / or second types of simulators. The second sub-dataset includes robot task demonstrations for a second specified task, such as industrial, domestic, or caregiving tasks. In step 460, using... Figure 3The training technique illustrated involves training engine 122 generating a second specialized transformer by retraining the basic transformer to control a simulated robot to perform a second specified task using a second small dataset. In some embodiments, the task group associated with the basic transformer includes one or more tasks, and the second specified task is not included in one or more task groups. In other embodiments, the task group associated with the basic transformer includes two or more task groups, and the second specified task is included in two or more task groups.

[0128] The various steps of method 400 can be repeated to generate multiple dedicated transformers for performing multiple different specified tasks. This can be achieved through implementation. Figure 3 The training techniques shown allow different specialized transformers to be trained using different small datasets collected from different types of simulators and / or robots. Furthermore, once trained, any type of simulator and / or robot can be used during the execution of the specialized transformer to generate additional robot task demonstrations for a specified task associated with the specialized transformer during the execution phase, as described below.

[0129] Execute the transformer for collecting new presentations

[0130] Back Figure 2 After training transformer 200 using a small training dataset associated with a specified task, transformer 200 includes a dedicated transformer trained to control a simulated robot to perform the specified task. Execution engine 124 then executes transformer 200 and operates in conjunction with simulator 125 to generate and collect new demonstrations of the simulated robot performing the specified task in a simulated environment. For example, execution engine 124 may receive transformer input vector 225 from simulator 125 and input transformer input vector 225 into transformer 200. Execution engine 124 may also receive transformer output vector 250 from transformer 200 and send transformer output vector 250 to simulator 125.

[0131] Simulator 125 generates a simulation environment and configures a simulated robot according to a set of robot configuration parameters 130. Simulator 125 implements a dedicated transformer for automatically operating the simulated robot to perform a specified task, thereby generating a robot demonstration of the specified task. The type of simulator 125 and / or the type of the simulated robot (defined by the set of robot configuration parameters 130) can be the same as or different from the type of simulator 125 and / or the type of the simulated robot used to generate and collect training dataset 210. After executing transformer 200, training engine 122 can further train transformer 200 using the new robot task demonstration. The new robot task demonstration can also be stored in storage 114 for training other additional neural networks.

[0132] Specifically, during the creation of a new robot task demonstration, the execution engine 124 executes the transformer 200 to generate a series of transformer output vectors 250 based on a series of transformer input vectors 225. Each transformer output vector 250 is generated based on the corresponding transformer input vector 225. Each transformer input vector 225 includes transformer input information for a past time step sequence of the currently executed robot task demonstration. Each transformer output vector 250 includes predicted transformer output information for a future time step sequence of the currently executed robot task demonstration.

[0133] Figure 5 The diagram illustrates various embodiments of the relationship between the two. Figure 2 The transformer is associated with the input vector and the output vector. Figure 2 The example currently demonstrates the robot task being executed at time step (t), where time step (t) is the most recently executed time step of simulator 125.

[0134] In some embodiments, for each execution iteration of converter 200, converter 200 receives a past time step sequence Tp (converter input vector 225) and generates a future time step sequence Tf (converter output vector 250), where the input sequence length is equal to Tp and the output sequence length is equal to Tf. For each execution iteration of converter 200, simulator 125 executes a sub-part of the future time step sequence Tf by executing a future time step sequence Te (where Te is less than Tf) within the future time step sequence Tf. Once executed, the sequence of future time steps Te is included in the sequence of past time steps Tp in the next execution iteration of converter 200.

[0135] As shown in the figure, converter 200 receives converter input vector 225, which includes converter input information of a sequence of past time steps from (t-Tp) to (t) of the currently executed robot task demonstration. For each past time step, the converter input information includes a state vector (s), an action vector (a), and a position vector (n). As shown in the figure, the information of future time steps received as input by converter 200 is zero-padded, forcing converter 200 to generate information of future time steps only from the information of past time steps.

[0136] Based on the transformer input vector 225, the transformer 200 generates a transformer output vector 250, which includes transformer output information for the predicted future time step sequence from (t+1) to (t+Tf) of the currently executed robot task demonstration. For each future time step, the transformer input information includes a state vector (s) and an action vector (a). The future time step sequence comprises two sub-parts: a first sub-part of time step 510 and a second sub-part of time step 520. The first sub-part of time step 510 from (t+1) to (t+Te) includes future time steps that are currently being executed by the simulator 125 and / or have been loaded by the simulator 125 for execution in the current execution iteration of the transformer 200. In some embodiments, the number of time steps Te in the first sub-part of time step 510 includes the number of time steps executed in each pass / iteration of the transformer 200. In some embodiments, the user cannot manually interrupt the execution of the first sub-part of time step 510.

[0137] The second sub-part of time step 520 includes future time steps that occur after the first sub-part of time step 510. The second sub-part of time step 520, from (t+Te+1) to (t+Tf), includes future time steps that are not currently being executed by simulator 125 and / or have not yet been loaded by simulator 125 for execution in the current execution iteration of converter 200. In some embodiments, the user can manually interrupt / stop the execution of the second sub-part of time step 520 (i.e., prevent the execution of the second sub-part of time step 520).

[0138] Each converter input vector 225 (represented by equation (1)) is input to the corresponding embedding function within the input layer of converter 200. Specifically, the state vector (s) of converter input vector 225 is received by the state embedding function, the action vector (a) of converter input vector 225 is received by the action embedding function, and the position vector (n) of converter input vector 225 is received by the position embedding function.

[0139] For each past time step, the state vector specifies the state of the end effector pose, the state of the end effector grip, and the pose of at least one object at each time step. For each time step, the end effector pose (s) r The state of the end effector grip can be specified by a 7D vector, as shown in equation (3). For each time step, the end effector grip state (s) is... g The state of ) is specified as on (0) or off (1), as shown in equation (4). For each time step of a specific object (oi), the object pose (s) oiThe state of the transformer 200 can be specified by a 7D vector, as shown in equation (5). The state embedding function represents a 128-dimensional state vector corresponding to the latent variables / parameters learned by the transformer 200. For each past time step, the state vector(s) is input into the state embedding function (F). s ), which is represented as a single fully connected network layer of converter 200, as shown in equation (20):

[0140] s emb,t =F s (s t (20)

[0141] For each past time step, the action vector (a) specifies the target pose of the end effector and the action of the target state grasped by the end effector. For each time step, the target end effector pose (a) is... r The action of the target end effector can be specified by a 7D vector, as shown in equation (6). For each time step, the target end effector gripping state (a g The action is specified as on (0) or off (1), as shown in equation (7). The action embedding function represents a 128-dimensional action vector (a), which corresponds to the latent variables / parameters learned by the transformer 200. For each past time step, the action vector (a) is input into the action embedding function (F). a ), which is represented as a single fully connected network layer of converter 200, as shown in equation (21):

[0142] a emb,t =F a (a t ) (twenty one)

[0143] For each past time step, the position vector (n) specifies the distance and rotation traversed by the robot's end effector within that time step. Typically, the position vector is calculated using the cumulative distance traversed by the end effector in Euclidean space within the time step. The position vector may include a single-dimensional integer value (integer marker) indicating the traversal distance and rotation of the end effector during the time step. The position vector (n) is represented by equation (22):

[0144]

[0145] In equation (22), each c r Includes the 3D coordinates of the 3D bounding box around the end effector at a given time step. rThe value is a function of the end effector pose at a given time step. Pose representation (position and quaternion) is used because pose specifies rotations in position space, which alleviates the problem of combining heterogeneous terms in the same function. The position embedding function represents a 128-dimensional position vector (n) corresponding to the latent variables / parameters learned by the transformer 200. For each past time step, the position vector (n) is input to the position embedding function (F). n ), which is represented as a single fully connected network layer of converter 200, as shown in equation (23):

[0146] n emb,t =F n (n t ) (twenty three)

[0147] Position embeddings are added to each state embedding and action embedding. The resulting embeddings are combined to produce an input vector (x), where “LN” denotes layer normalization, as shown in equation (24):

[0148]

[0149] Transformer 200 receives an input vector representing a 256-dimensional input vector, which corresponds to a concatenation of 128-dimensional representations from each state vector and action vector. For each input vector, transformer 200 generates a corresponding prediction vector, also represented in 256 dimensions, mirroring the dimensions of the input vector. The prediction vector is then decoded using a linear layer on the output side of transformer 200, which reflects the linear layer on the input side of transformer 200. The decoded prediction vector includes transformer output vector 250, output by transformer 200 at the output / decoding layer of transformer 200.

[0150] Back Figure 5In automatic mode, simulator 125 receives converter output vector 250 from converter 200 and executes the future time step sequence by manipulating the simulated robot based on the converter output information of the future time step sequence. Specifically, for each future time step, simulator 125 manipulates the simulated robot in an appropriate manner to achieve the pose of the robot's end effector and the end effector gripping state specified in the converter output information of the future time step. For example, simulator 125 may use a Riemann Motion Policy (RMP) to manipulate the simulated robot to achieve the pose of the robot's end effector and the end effector gripping state specified in the converter output information of the future time step. RMP technology includes a method for predicting the acceleration and velocity of the simulated robot based on the current joint position and joint target of the simulated robot. In other embodiments, simulator 125 uses another motion technique to manipulate the simulated robot based on converter output information. Once executed, the future time step is included in the next sequence of past time steps, which converter 200 receives as input in the next execution process / iteration of converter 200.

[0151] In some embodiments, the simulator 125 enables multiple execution processes / iterations of the converter 200 to automatically operate the simulated robot to perform a specified task. In this way, the converter 200 is implemented to automatically generate and collect any number of robot task demonstrations without requiring user input / intervention or manual operation of the simulated robot. Therefore, compared to conventional techniques that require extensive manual operation of the simulated robot, the execution technique can generate and collect robot task demonstrations more easily and efficiently.

[0152] When operating in automatic mode, simulator 125 operates the simulated robot based on the converter output vector of converter 200. However, simulator 125 can also operate in manual mode when operating the simulated robot based on user control input instead of the converter output vector of converter 200. In these embodiments, when simulator 125 receives converter output vector 250 from a dedicated converter, the simulator also displays a trajectory prediction that includes a graphical representation of a future time step sequence specified in the converter output vector 250. In particular, the trajectory prediction graphically displays the pose of the robot's end effector in the future time step sequence. Therefore, if the simulator continues to operate in automatic mode without user intervention, the trajectory prediction graphically illustrates the future poses of the robot's end effector, which will appear in the future time steps of the robot task demonstration.

[0153] For example, simulator 125 can generate a simulated environment that includes a VR environment. In these embodiments, simulator 125 is also coupled to a virtual reality device 170 including a virtual reality headset 171, a virtual reality controller 172, and one or more tracking devices 173. VR controller 172 includes a VR tracking device tracked by tracking device 173. VR controller 172 may include controllers that operate in 3D. VR headset 171 includes a VR tracking device tracked by tracking device 173. Tracking device 173 can track the 3D position of VR headset 171 and VR controller 172. Each tracking device 173 may include various components (not shown) for performing the functions described above for tracking device 173. In other embodiments, other types of display devices and input controllers may be used.

[0154] VR headset 171 can display various VR scenes 175 of a 3D VR environment generated by simulator 125. VR scene 175 includes a specific view of the VR environment depending on the viewpoint of VR headset 171. The displayed VR scene 175 includes a simulated robot and at least one object on which the simulated robot performs a task. Virtual reality scene 175 may also include trajectory predictions representing future time-step sequences that will occur during the robot task demonstration if simulator 125 continues to operate in automatic mode.

[0155] However, based on trajectory prediction, the user can decide to set simulator 125 to manual mode (via VR controller 172) to momentarily intervene and interrupt / stop the automatic mode operation of simulator 125 and the simulated robot (i.e., prevent execution of at least a portion of future time steps). During manual mode, the user (via VR controller 172) provides user control input for manually manipulating the simulated robot, for example, pushing the simulated robot in a direction that the user deems more suitable for successful task execution than the trajectory prediction indicates. In manual mode, simulator 125 executes a sequence of time steps throughout the entire time step sequence demonstrated by the robot based on the user control input. In manual mode, simulator 125 also collects transformer input information (referred to as "manual mode transformer input information") for the executed time step sequence, including the state vector, motion vector, and position vector for each time step in the execution sequence.

[0156] Once the user is confident that the simulated robot has returned to the correct track for successfully performing its task, the user can switch simulator 125 back to automatic mode (via VR controller 172) to continue the robot's automatic operation based on the output of transformer 200 instead of user control input. Upon returning to automatic mode, simulator 125 inputs the next transformer input vector to transformer 200, which includes at least a portion of the manual mode transformer input information. Based on this next transformer input vector, transformer 200 generates the next transformer output vector when simulator 125 returns to automatically operating the simulated robot based on the transformer output vector of transformer 200.

[0157] Figure 6 Trajectory predictions associated with industrial tasks performed by a first simulated robot according to various embodiments are illustrated. In some embodiments, robot task demonstration 150 includes demonstrations of industrial tasks such as nut assembly, kit assembly, box stacking, etc. As shown, the robot task demonstration performed in virtual environment 600 includes a demonstration of a first simulated robot 605 having a first robot end effector 610 performing a nut assembly task on a first object 651 and a second object 652 in virtual environment 600.

[0158] If simulator 125 continues to operate the first simulated robot 605 in automatic mode, simulator 125 generates trajectory prediction 620, graphically representing a sequence of future time steps that will occur during the robot task demonstration. Specifically, trajectory prediction 620 graphically illustrates the pose of the first robot end effector 610 during the future time step sequence. Therefore, trajectory prediction 620 graphically illustrates the future poses of the first robot end effector 610 that will occur in future time steps if simulator 125 continues to operate in automatic mode. In some embodiments, trajectory prediction 620 uses simple 3D geometry (e.g., a cube, a sphere, etc.) to graphically represent the pose of the first robot end effector 610. In other embodiments, trajectory prediction 620 uses 3D geometry similar to that of the first robot end effector 610 to graphically represent the pose of the first robot end effector 610. Based on trajectory prediction 620, the user can decide to set simulator 125 to manual mode to manually control the simulated robot 605 (via VR controller 172) and then return simulator 125 to automatic mode.

[0159] As described above, the future time step sequence includes a first sub-part of time step 510 and a second sub-part of time step 520. The first sub-part of time step 510 includes the future time step that simulator 125 is currently executing. The second sub-part of time step 520 includes future time steps that simulator 125 is not currently executing. In some embodiments, trajectory prediction 620 includes a first sub-part of trajectory prediction 621 and a second sub-part of trajectory prediction 622. The first sub-part of trajectory prediction 621 graphically represents the first sub-part of time step 510, and the second sub-part of trajectory prediction 621 graphically represents the second sub-part of time step 520. Simulator 125 may display the first sub-part of trajectory prediction 621 with a first graphical appearance and the second sub-part of trajectory prediction 622 with a second graphical appearance different from the first graphical appearance. For example, simulator 125 may use different colors, shadows, contrasts, and / or three-dimensional geometry to display the first sub-part 621 than the second sub-part 622.

[0160] In a further embodiment, simulator 125 also generates a past trajectory 630, which graphically represents a sequence of past time steps executed by simulator 125. Specifically, past trajectory 630 graphically illustrates the pose of the first robot end effector 610 during the past time step sequence. In some embodiments, past trajectory 630 graphically represents the pose of the first robot end effector 610 using simple 3D geometry (e.g., a cube, a sphere, etc.). In other embodiments, past trajectory 630 represents the pose of the first robot end effector 610 using 3D geometry similar to that of the first robot end effector 610. Simulator 125 may display past trajectory 630 with a third graphical appearance that differs from the first and / or second graphical appearance. For example, simulator 125 may use different colors, shading, contrast, and / or 3D geometry than the first sub-part 621 and / or the second sub-part 622 to display past trajectory 630. For example, the past trajectory 630 can be graphically represented as a yellow cube, the first sub-part 621 can be graphically represented as a green cube, and the second sub-part 622 can be graphically represented as a blue cube.

[0161] Figure 7 The diagram illustrates trajectory predictions associated with industrial tasks performed by a second simulated robot according to various embodiments. As shown, a demonstration of a robotic task performed in a virtual environment 700 includes a demonstration of a second simulated robot 705, which has a second robot end effector 710 performing a nut assembly task on a first object 751, a second object 752, and a third object 753 in the virtual environment 700. Note that the second simulated robot 705 is a different type of robot, configured differently from... Figure 6 The first simulated robot 605 shown is different. Furthermore, the second robot end effector 710 is a different type of robot end effector, and is similar to... Figure 6 The first robot end effector 610 shown has a different configuration.

[0162] Simulator 125 generates a trajectory prediction 720, graphically illustrating the pose of the second robot end effector 710 in a future time-step sequence. The trajectory prediction 720 uses simple 3D geometry (e.g., a cube as shown) or 3D geometry similar to that of the second robot end effector 710 to graphically represent the pose of the second robot end effector 710. Based on the trajectory prediction 720, the user can decide to set simulator 125 to manual mode to manually control the simulated robot 705 (via VR controller 172), and then return simulator 125 to automatic mode. Simulator 125 also generates a past trajectory 730, which graphically shows the pose of the second robot end effector 710 during a previous sequence in the past time steps. Simulator 125 can display the trajectory prediction 720 with a first graphical appearance and the past trajectory 730 with a second graphical appearance different from the first graphical appearance.

[0163] Figure 8 Trajectory predictions associated with household tasks performed by a third simulated robot according to various embodiments are illustrated. In some embodiments, the robot task demonstration 150 includes a demonstration of a household task, such as bowl manipulation, opening a drawer, closing a door, etc. As shown, the robot task demonstration performed in a virtual environment 800 includes a demonstration of a third simulated robot 805 with a third robot end effector 810 performing a bowl manipulation task on a first object 851 in the virtual environment 800. Simulator 125 generates a trajectory prediction 820 that graphically shows the posture of the third robot end effector 810 during a future time step sequence. Based on the trajectory prediction 820, the user can decide to set simulator 125 to manual mode to manually control simulated robot 805 (via VR controller 172) and then return simulator 125 to automatic mode. Simulator 125 also generates a past trajectory 830 that graphically shows the posture of the third robot end effector 810 during a previous sequence in the past time step.

[0164] Figure 9The diagram illustrates trajectory predictions associated with household tasks performed by a fourth simulated robot according to various embodiments. In some embodiments, the robot task demonstration 150 includes demonstrations of personal care tasks, such as assisting with drinking, massaging, scratching, etc. As shown, the robot task demonstration performed in a virtual environment 900 includes a demonstration of a fourth simulated robot 905 with a fourth robotic end effector 910 performing a drinking task on a first object 951 (representing a cup) and a second object 952 (representing a person) in the virtual environment 900. The simulator 125 generates a trajectory prediction 920 that graphically illustrates the posture of the fourth robotic end effector 910 during a future time step sequence. Based on the trajectory prediction 920, the user can decide to set the simulator 125 to manual mode to manually control the simulated robot 905 (via VR controller 172) and then return the simulator 125 to automatic mode. The simulator 125 also generates a past trajectory 930 that graphically illustrates the posture of the fourth robotic end effector 910 during a previous sequence in the past time step.

[0165] When the simulator 125 is operated using a combination of automatic and manual modes to successfully complete a specified task, it is referred to as operating in "automatic-assisted mode". A successful robot task demonstration includes a robot task demonstration in which the simulated robot has successfully executed the specified task within a predetermined time period. When the specified task is completed entirely in the automatic mode of the simulator 125, a dedicated converter can be implemented to generate robot task demonstrations with a high success rate. When the specified task is completed in the automatic-assisted mode of the simulator 125, a dedicated converter can be implemented to generate robot task demonstrations with an even higher success rate than when executed entirely in automatic mode. In this way, upon successful task execution, the execution technology provides automatic operation and / or automatic-assisted operation of the simulated robot, which requires less or no manual operation from the user compared to conventional technologies.

[0166] When the simulated robot performs a specified task in a simulated environment (in automatic or automatic-assisted mode), simulator 125 records demonstration data describing the performance of the specified task in the simulated environment, which is then included in a new robot task demonstration 150 for the specified task. In some embodiments, simulator 125 records data describing a complete sequence of end effector poses, end effector gripping states, and object poses of at least one object during the completion of the specified task, which is included in the new robot task demonstration 150 for the specified task. In other embodiments, simulator 125 records additional data describing the performance of the specified task in a simulated environment, which is included in the new robot task demonstration 150. Each new robot task demonstration 150 can then be stored in a database or data store, such as storage 114.

[0167] Figure 10 This is a flowchart illustrating the method steps for generating a task using a transformer according to various embodiments. Although combined with... Figure 1-2 The system described in sections 5-9 outlines the method steps, but those skilled in the art will understand that any system configured to execute the method steps in any order is within the scope of this invention. In some embodiments, method 1000 is executed by simulator 125 in conjunction with execution engine 124, which executes a dedicated converter (e.g., converter 200) trained for a first task.

[0168] As shown in the figure, in step 1010, simulator 125 initializes a new task demonstration for the first task in the simulation environment. To this end, simulator 125 configures and generates a simulated robot based on a set of robot configuration parameters 130. For the first task, different new task demonstrations need to be generated. In this regard, simulator 125 can determine different initial states of the simulation environment for different task demonstrations by randomly changing the initial pose of the end effector, the initial gripping state of the end effector, and / or the initial pose of at least one object in the simulation environment. For example, simulator 125 can use a standard random function or any other suitable technique to determine the initial pose of the end effector, the initial gripping state of the end effector, and the initial pose of at least one object.

[0169] In step 1010, simulator 125 also displays a simulated environment including the simulated robot and at least one object, for example, through VR headset 171, throughout the execution of method 1000. Simulator 125 also begins recording data describing the performance of the first task performed by the simulated robot, which is subsequently included in a new robot task demonstration 150 for the first task. For example, simulator 125 may record a whole sequence of data describing the end effector pose, end effector gripping state, and object pose throughout the entire completed execution of the first task. Simulator 125 is also initially set to automatic mode.

[0170] In step 1020, simulator 125 determines and executes an initial sequence of Tp time steps for operating the simulated robot during the execution of the first task. The initial order of Tp time steps is determined based on the initial state of the simulated environment. For example, simulator 125 may derive initial input information describing the initial state of the environment, where the motion vector is set to zero, input the initial input information to a dedicated converter to receive predictions from the dedicated converter, concatenate the predictions with the previous state and motion vectors, and repeat until the initial order of Tp time steps is determined. Simulator 125 then executes the initial sequence of Tp time steps for operating the simulated robot.

[0171] In step 1030, simulator 125 determines converter input vector 225 based on the most recently executed past time step sequence Tp (in automatic or manual mode). Converter input vector 225 includes converter input information for the most recently executed past time step sequence Tp, which can be derived by simulator 125 from the simulation environment. The converter input information includes a state vector, motion vector, and position vector for each past time step. The state vector specifies the pose of the robot end effector, the state of the end effector grip, and the state of the pose of at least one object in the simulation environment. The motion vector specifies the motion of the target pose and target grip state of the robot end effector in the simulation environment. The position vector specifies the distance traveled and rotation of the robot end effector within the time step in the simulation environment.

[0172] In step 1040, simulator 125 determines converter output vector 250 based on converter input vector 225 by inputting converter input vector 225 to dedicated converter and receiving converter output vector 250 from dedicated converter. Specifically, simulator 125 can send converter input vector 225 to execution engine 124, which executes dedicated converter based on converter input vector 225. Dedicated converter generates converter output vector 250 based on converter input vector 225, and execution engine 124 sends converter output vector 250 to simulator 125. Converter output vector 250 includes converter output information for future time step sequences Tf that occurred after the most recent Tp past time step sequence. The converter output information includes a state vector and an action vector for each future time step.

[0173] In step 1050, simulator 125 generates and displays a predicted trajectory and a past trajectory in the simulation environment, for example via VR headset 171. The trajectory prediction graphically illustrates the Tf future time step sequence specified in the transformer output vector 250. Specifically, trajectory prediction 620 graphically illustrates the pose of the robot end effector in the future time step sequence. In some embodiments, trajectory prediction 620 includes a first sub-part of trajectory prediction 621 having a first appearance and a second sub-part of trajectory prediction 622 having a second appearance. The first sub-part of trajectory prediction 621 graphically represents the first sub-part of the currently executing future time step 510. The second sub-part of trajectory prediction 621 graphically represents the second sub-part of the currently unexecuted future time step 520. Simulator 125 may also generate a past trajectory 630 with a third appearance, which graphically represents the sequence of past time steps already executed by simulator 125. Specifically, past trajectory 630 graphically illustrates the pose of the robot end effector during the past time step sequence.

[0174] In step 1060, simulator 125 determines whether manual mode input has been received from the user, for example, via VR controller 172. If manual mode input has been received (in 1060 – yes), method 1000 continues in step 1090. If no manual mode input has been received (in 1060 – no), method 1000 continues in step 1070.

[0175] In step 1070, simulator 125 executes at least a portion of the Tf future time step sequence to automatically operate the simulated robot. In some embodiments, simulator 125 executes a sequence of future time steps. Specifically, for each such future time step, simulator 125 operates the simulated robot in an appropriate manner (e.g., using RMP) to achieve the pose of the robot's end effector and the end effector grip state specified in the converter output information of the future time step. Once executed, the future time step is included in the next sequence Tp of the past time step, which is represented by the converter input vector 225 received by the dedicated converter in the next execution process / iteration of the dedicated converter.

[0176] In step 1080, simulator 125 determines whether the simulated robot has successfully completed the first task. If the first task was not successfully completed (at 1080 – No), method 1000 continues in step 1030. If the first task was successfully completed (at 1080 – Yes), method 1000 continues in step 1085. In step 1085, simulator 125 collects recorded data used to perform the first task and includes the recorded demonstration data in a new task demonstration 150 used for the first task. The new task demonstration 150 can be stored in a database or data store, such as storage 114. Then, method 1000 ends.

[0177] If a manual mode input is received (in 1060 – Yes), then in step 1090, simulator 125 does not execute at least a portion of the Tf future time step sequence, such as the second sub-part of the currently unexecuted future time step 520. Instead, simulator 125 manually operates the time step sequence of the simulated robot based on user control input received from the user (e.g., via the VR controller). In manual mode, simulator 125 collects manual mode transformer input information for the manual time step sequence. In step 1095, simulator 125 (via VR controller 172) receives automatic mode input from the user, which sets simulator 125 back to automatic mode. Method 1000 then continues in step 1080.

[0178] In summary, a training technique is implemented for training a transducer to operate a simulated robot (in conjunction with a simulator) to perform tasks related to at least one object in a simulated environment using a robot task demonstration. The simulated robot includes an end effector for interacting with at least one object during task performance. The training technique involves training the transducer based on the pose of the robot's end effector and the pose of at least one object shown in the robot task demonstration.

[0179] Each robot task demonstration includes an overall time-step sequence, encompassing all time steps for performing the task. During training using the robot task demonstration, the transformer receives a transformer input vector containing transformer input information for past time-step sequences included in the overall time-step sequence of the robot task demonstration. For each past time-step, the transformer input information includes a state vector and a motion vector. The state vector specifies the pose of the robot's end effector, the state of the end effector grip, and the pose of at least one object associated with the past time-step. The motion vector specifies the target pose of the robot's end effector and the target state of the end effector grip associated with the past time-step. Using the robot task demonstration, the transformer is trained to generate a transformer output vector containing transformer output information based on a predicted / future time-step sequence of transformer input vectors. For each future time-step, the transformer output information includes a state vector and a motion vector.

[0180] Compared to existing technologies, one technical advantage of the disclosed technology lies in its ability to train a neural network for manipulating a simulated robot to perform tasks, based on the pose (position and orientation) of the robot's end effector as demonstrated in a task-based robot task demonstration. Using the disclosed technology, the neural network is trained without relying on any configuration parameters of the simulator and / or simulated robot associated with the robot task demonstration. Therefore, robot task demonstrations (training datasets) for training the neural network can be generated using any type of simulator and / or any type of simulated robot. Consequently, the disclosed technology allows for the easier and more efficient generation and collection of robot task demonstrations for training the neural network compared to conventional techniques. These technical advantages provide one or more technological improvements compared to existing methods.

[0181] In summary, the execution technique is realized in a simulated environment when a dedicated transformer, trained for a specified task, is executed to collect new / additional robot task demonstrations for that task. The simulator implements a trained dedicated transformer for automatically maneuvering a simulated robot to perform a specified task, thereby generating a robot demonstration for that task. When generating a robot task demonstration, the dedicated transformer receives a transformer input vector containing transformer input information for a sequence of past time steps executed in the robot task demonstration. For each past time step, the transformer input information includes a state vector, a motion vector, and a position vector. The position vector specifies the distance and rotation traversed by the robot's end effector during the past time steps.

[0182] Based on the transformer input vector, a dedicated transformer generates a transformer output vector containing transformer output information for a predicted / future time-step sequence that has not yet been executed / completed in the robot task demonstration. For each future time step, the transformer output information includes a state vector and an action vector. The simulator receives the transformer output vector from the dedicated transformer and executes the future time-step sequence by manipulating the simulated robot based on the transformer output information. The simulator can implement multiple execution processes / iterations of the dedicated transformer to automatically operate the simulated robot to complete a specified task.

[0183] The simulator can also operate in manual mode when operating the simulated robot based on user control inputs rather than the converter output vectors of a dedicated converter. In these embodiments, when the simulator receives the converter output vectors from the dedicated converter, it also displays trajectory predictions, including a graphical representation of the future time-step sequence specified in the converter output vectors. In particular, the trajectory predictions graphically display the posture of the robot's end effector in the future time-step sequence. Based on the trajectory predictions, the user can switch the simulator to manual mode to momentarily intervene and stop the simulated robot's automatic operation. Once the user believes the simulated robot has returned to the correct trajectory and successfully completed the task, the user can switch the simulator back to automatic mode to continue simulating the robot's automatic operation, e.g., completing the task, based on the output of the dedicated converter rather than the user control inputs.

[0184] Compared to existing technologies, one technical advantage of the disclosed technology is that, through the disclosed technology, the simulator implements a trained neural network for automatically operating a simulated robot to perform tasks, thereby generating robot demonstrations of the tasks. Another technical advantage of the disclosed technology compared to existing technologies is that, utilizing the disclosed technology, the simulator displays trajectory predictions, which graphically illustrate the posture of the robot's end effector in a future time-step sequence. Based on the trajectory predictions, the user can decide to interrupt the simulated robot's automatic operation and immediately provide user input for manually operating the simulated robot, then allow the simulated robot's automatic operation to resume to complete the task. In this way, the disclosed technology provides automatic operation and / or automatic assisted operation of the simulated robot when performing tasks that do not require or require minimal user manual operation. Therefore, the disclosed technology makes it easier and more efficient to generate and collect robot task demonstrations for training neural networks than traditional manual techniques. These technical advantages provide one or more technical improvements compared to existing methods.

[0185] The various aspects of the subject matter described herein are listed in the following numbered clauses.

[0186] 1. In some embodiments, a computer-implemented method for generating a robot task demonstration, the method comprising: generating a simulated environment including a robot and at least one object via a simulator engine; causing the robot to at least partially perform a task associated with the at least one object within the simulated environment based on a first output vector generated by a trained neural network; and recording demonstration data of the robot performing the task at least partially within the simulated environment.

[0187] 2. The computer-implemented method according to Clause 1, wherein the first output vector specifies a future pose sequence of the robot's end effector and is generated by the trained neural network based on a first input vector specifying a past pose sequence of the robot's end effector.

[0188] 3. The computer-implemented method according to any one of clauses 1-2, wherein causing the robot to perform the task at least partially comprises: causing the robot to perform a future pose sequence of the end effector specified in the first output vector.

[0189] 4. The computer-implemented method according to any one of claims 1-3, further comprising: receiving at the simulator engine a second output vector generated by the trained neural network, wherein the second output vector specifies a future pose sequence of the robot's end effector; and displaying a trajectory prediction within the simulation environment, wherein the trajectory prediction includes a graphical representation of the future pose sequence of the end effector.

[0190] 5. The computer-implemented method according to any one of clauses 1-4, further comprising: receiving first user input that causes the simulator engine to enter manual mode; and, in response, causing the robot to perform the task at least partially based on one or more user control inputs.

[0191] 6. The computer-implemented method according to any one of clauses 1-5 further includes recording additional demonstration data of the robot performing the task at least partially based on the one or more user control inputs.

[0192] 7. The computer-implemented method according to any one of clauses 1-6, wherein the simulated environment includes a virtual reality (VR) environment, the VR environment being displayed via a VR headset, and the one or more user control inputs being received via a VR controller.

[0193] 8. The computer-implemented method according to any one of claims 1-7, further comprising: receiving second user input that causes the simulator engine to enter an automatic mode; and, in response, causing the robot to perform the task at least partially automatically based on a third output vector generated by the trained neural network.

[0194] 9. A computer-implemented method according to any one of claims 1-8, wherein when the robot performs the task at least partially based on the one or more user control inputs, the third output vector is generated by the trained neural network based on an input vector specifying a past pose sequence of the robot's end effector.

[0195] 10. The computer-implemented method according to any one of claims 1-9, further comprising: receiving at the simulator engine a second output vector generated by the trained neural network, wherein the second output vector specifies a future pose sequence of the robot's end effector; displaying a trajectory prediction within the simulation environment, wherein the trajectory prediction includes a graphical representation of the future pose sequence of the end effector; and displaying a past trajectory within the simulation environment, the past trajectory including a graphical representation of a past pose sequence of the end effector, wherein the past trajectory is displayed with an appearance different from the trajectory prediction.

[0196] 11. In some embodiments, one or more non-transitory computer-readable media storing instructions, when executed by one or more processors, to cause the one or more processors to generate a robot task demonstration by performing the following steps: generating a simulated environment including a robot and at least one object via a simulator engine; causing the robot to at least partially perform a task associated with the at least one object within the simulated environment based on a first output vector generated by a trained neural network; and recording demonstration data of the robot performing the task at least partially within the simulated environment.

[0197] 12. One or more non-transitory computer-readable media as described in Clause 11, wherein the first output vector specifies a future pose sequence of the robot's end effector, further comprising: displaying a trajectory prediction within the simulation environment prior to causing the robot to perform the task at least partially, wherein the trajectory prediction includes a graphical representation of the future pose sequence of the end effector.

[0198] 13. One or more non-transitory computer-readable media according to any one of clauses 11-12, further comprising: enabling the robot to perform the task based on a plurality of output vectors generated by the trained neural network; and additional demonstration data of the robot performing the task within the simulated environment.

[0199] 14. One or more non-transitory computer-readable media according to any one of clauses 11-13, wherein the first output vector specifies a future pose sequence of the robot's end effector and is generated by the trained neural network based on a first input vector specifying a past pose sequence of the robot's end effector.

[0200] 15. One or more non-transitory computer-readable media according to any one of clauses 11-14, further comprising: receiving at the simulator engine a second output vector generated by the trained neural network, wherein the second output vector specifies a future pose sequence of the robot's end effector; and displaying a trajectory prediction within the simulation environment, wherein the trajectory prediction includes a graphical representation of the future pose sequence of the end effector.

[0201] 16. One or more non-transitory computer-readable media according to any one of clauses 11-15, further comprising: receiving a first user input that causes the simulator engine to enter a manual mode; and, in response to the first user input, operating the robot without based on the second output vector.

[0202] 17. One or more non-transitory computer-readable media according to any one of clauses 11-16, further comprising: receiving first user input that causes the simulator engine to enter manual mode; and, in response to the first user input, causing the robot to perform the task at least partially based on one or more user control inputs.

[0203] 18. One or more non-transitory computer-readable media according to any one of clauses 11-17, further comprising: additional demonstration data recording the robot performing the task at least in part based on the one or more user control inputs.

[0204] 19. One or more non-transitory computer-readable media according to any one of clauses 11-18, further comprising: receiving a second user input that causes the simulator engine to enter an automatic mode; and, in response, causing the robot to perform the task at least partially based on a third output vector generated by the trained neural network.

[0205] 20. In some embodiments, a system includes: one or more memories storing instructions; and one or more processors coupled to the one or more memories, and, upon execution of the instructions, generating a robot task demonstration by performing the following steps: generating a simulated environment including a robot and at least one object via a simulator engine; causing the robot to at least partially perform a task associated with the at least one object within the simulated environment based on a first output vector generated by a trained neural network; and recording demonstration data of the robot performing the task at least partially within the simulated environment.

[0206] Any and all combinations of any elements described in any claim and / or in this application, in any manner, are within the intended scope of this embodiment and protection.

[0207] The various embodiments described are given for illustrative purposes and are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

[0208] Aspects of this embodiment can be implemented as a system, method, or computer program product. Therefore, aspects of the invention can take the form of a completely hardware embodiment, a completely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which are generally referred to herein as “module,” “system,” or “computer.” Furthermore, any hardware and / or software technology, process, function, component, engine, module, or system described in this disclosure can be implemented as a circuit or a set of circuits. Additionally, aspects of the invention can take the form of a computer program product contained on one or more computer-readable media, on which computer-readable program code is included.

[0209] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, magnetic storage, electronic devices, optical devices, electronic devices, or combinations thereof. More specific examples (not an exhaustive list) of computer-readable storage media will include: an electrical connection having one or more wires, a portable computer floppy disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the context of this document, a computer-readable storage medium can be any tangible medium that can contain or store programs for use by or associated with an instruction execution system, apparatus, or device.

[0210] The foregoing description of various aspects of the present invention has referenced flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine. When the instructions are executed by the processor of the computer or other programmable data processing apparatus, the functions / actions specified in the flowcharts and / or block diagrams are realized. Such processors can be, but are not limited to, general-purpose processors, special-purpose processors, special-purpose processors, or field-programmable gate arrays.

[0211] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, comprising one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may not conform to the order indicated in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or sometimes in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a system based on dedicated hardware, or a combination of dedicated hardware and computer instructions, that performs the specified function or action.

[0212] While the foregoing describes embodiments of the present invention, other and further embodiments of the present invention may be designed without departing from the basic scope of the present invention, and the scope of the present invention is defined by the following claims.

Claims

1. A computer-implemented method for generating robot task demonstrations, the method comprising: A simulation environment including a robot and at least one object is generated via a simulator engine; Based on a first output vector generated by a trained neural network in response to a first input vector of a past pose sequence of the robot's end effector within the specified simulated environment, the robot at least partially performs a task associated with the at least one object within the simulated environment. as well as Record demonstration data showing that the robot performs the task at least partially within the simulated environment.

2. The computer-implemented method of claim 1, wherein the first output vector specifies a future pose sequence of the robot's end effector.

3. The computer-implemented method of claim 1, wherein causing the robot to perform the task at least partially comprises: The robot executes the future pose sequence of the end effector specified in the first output vector.

4. The computer-implemented method according to claim 1 further includes: The simulator engine receives a second output vector generated by the trained neural network, wherein the second output vector specifies the future pose sequence of the robot's end effector. as well as The trajectory prediction is displayed within the simulation environment, wherein the trajectory prediction includes a graphical representation of the future posture sequence of the end effector.

5. The computer-implemented method according to claim 1, further comprising: Receive the first user input that puts the simulator engine into manual mode; as well as In response, the robot performs the task at least partially based on one or more user control inputs.

6. The computer-implemented method of claim 5, further comprising recording additional demonstration data of the robot performing the task at least partially based on the one or more user control inputs.

7. The computer-implemented method of claim 5, wherein the simulated environment comprises a virtual reality (VR) environment, the VR environment being displayed via a VR headset, and the one or more user control inputs being received via a VR controller.

8. The computer-implemented method according to claim 5, further comprising: Receive a second user input that causes the simulator engine to enter automatic mode; as well as In response, the robot is made to perform the task automatically, at least partially, based on a third output vector generated by the trained neural network.

9. The computer-implemented method of claim 8, wherein when the robot performs the task at least partially based on the one or more user control inputs, the third output vector is generated by the trained neural network based on an input vector specifying a past pose sequence of the robot's end effector.

10. The computer-implemented method according to claim 1, further comprising: The simulator engine receives a second output vector generated by the trained neural network, wherein the second output vector specifies the future pose sequence of the robot's end effector. The trajectory prediction is displayed within the simulation environment, wherein the trajectory prediction includes a graphical representation of the future pose sequence of the end effector; as well as The past trajectory displayed within the simulation environment includes a graphical representation of the past posture sequence of the end effector, wherein the past trajectory is displayed with an appearance different from the trajectory prediction.

11. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to generate a robot task demonstration by performing the following steps: A simulation environment including a robot and at least one object is generated via a simulator engine; Based on a first output vector generated by a trained neural network in response to a first input vector of a past pose sequence of the robot's end effector within the specified simulated environment, the robot at least partially performs a task associated with the at least one object within the simulated environment; and Record demonstration data showing that the robot performs the task at least partially within the simulated environment.

12. The non-transitory computer-readable medium of claim 11, wherein the first output vector specifies a future pose sequence of the robot's end effector, further comprising: Before the robot performs the task at least partially, a trajectory prediction is displayed within the simulation environment, wherein the trajectory prediction includes a graphical representation of the future pose sequence of the end effector.

13. The non-transitory computer-readable medium according to claim 11, further comprising: The robot performs the task based on multiple output vectors generated by the trained neural network. as well as Additional demonstration data of the robot performing the task within the simulated environment is recorded.

14. One or more non-transitory computer-readable media according to claim 11, wherein the first output vector specifies a future pose sequence of the robot's end effector.

15. The non-transitory computer-readable medium according to claim 11, further comprising: The simulator engine receives a second output vector generated by the trained neural network, wherein the second output vector specifies the future pose sequence of the robot's end effector. as well as The trajectory prediction is displayed within the simulation environment, wherein the trajectory prediction includes a graphical representation of the future posture sequence of the end effector.

16. The non-transitory computer-readable medium according to claim 15, further comprising: Receive the first user input that puts the simulator engine into manual mode; as well as In response to the first user input, the robot is not operated based on the second output vector.

17. The non-transitory computer-readable medium of claim 11, further comprising: Receive the first user input that puts the simulator engine into manual mode; as well as In response to the first user input, the robot performs the task at least partially based on one or more user control inputs.

18. The non-transitory computer-readable medium according to claim 17, further comprising: Additional demonstration data is recorded showing how the robot performs the task at least partially based on one or more user control inputs.

19. The non-transitory computer-readable medium according to claim 17, further comprising: Receive a second user input that causes the simulator engine to enter automatic mode; as well as In response, the robot performs the task at least partially based on a third output vector generated by the trained neural network.

20. A system for generating robot task demonstrations, comprising: One or more memories that store instructions; as well as One or more processors, coupled to the one or more memories, generate a robot task demonstration by performing the following steps when executing the instructions: A simulation environment including a robot and at least one object is generated via a simulator engine; Based on a first output vector generated by a trained neural network in response to a first input vector of a past pose sequence of the robot's end effector within the specified simulated environment, the robot at least partially performs a task associated with the at least one object within the simulated environment. as well as Record demonstration data showing that the robot performs the task at least partially within the simulated environment.

Images