Transition between tasks in different domains

Abstract

A system for trajectory imitation for a robotic manipulator is provided. The system includes an interface configured to receive a plurality of task specifications, wherein the interface is configured to communicate with a real-world robot, a memory to store computer executable programs including a robot simulator, a training module, and a transfer module, and a processor connected with the memory. The processor is configured to execute training for the robot simulator against a task specification using the training module to produce a plurality of source policies with sub-goals for the task specification. The processor executes training for the real-world robot against the task specification using the training module to produce a plurality of target policies with sub-goals for the task specification, and updates parameters of the transfer module according to corresponding trajectories with sub-goals for the robot simulator and the real-world robot.

Description

Technical Field

[0001] The present invention generally relates to a robot that performs a task, wherein the task has been trained in a simulation and transferred to the robot, wherein the transfer has been learned from skills demonstrated in both the simulation domain and the real-world domain. Background Technology

[0002] Industrial robots are widely used in a variety of manufacturing operations, including material handling, assembly, inspection, and pick-and-place tasks. Robots need to be programmed to perform their tasks correctly. Robot programs include instructions for end effectors (such as welding tools) to move to a specific 3D location in certain 3D orientations and perform spot welding. The robot can interpret these instructions to generate a desired trajectory, in which the robot controller operates the robot to follow that desired trajectory. However, programming a robot requires a highly trained person. For more complex tasks, programming can be cumbersome, hindering the flexibility of robot deployment. An alternative to programming robots is task learning. Robot learning is an active area of ​​research. Robots can learn simple tasks, such as picking up an object from one location and placing it in another. However, even such simple tasks require lengthy training times. This requires taking the robot offline during training, which can be costly. Furthermore, there is a risk of damaging the robot during training, especially at the very beginning of the training phase. One way to mitigate this risk is to train the robot in a simulated environment, where the robot and workspace are simulated by a computer graphics and physics engine, and then transfer the results from the training to a real robot. Simulations for robotic applications are an attractive solution for training and robotic agents because they provide a cost-effective data source and alleviate many concerns, such as robot safety and wear and tear. Many existing learning methods rely on robot simulators for the training process, and often training is only possible in simulations requiring large sample sizes. Due to the fact that the simulated robot and its environment are approximations of the real world, a so-called gap exists between the simulation and the real world. Because of this gap, the results of training in the simulation need to be refined when transferred to a real robot. If the task is complex, this refinement or fine-tuning itself can be time-consuming. We propose a solution where the real robot aims to imitate the results of training from the simulation by comparing the desired trajectory of the robot in the simulated environment with the trajectory of the robot in the real robot. The trajectory involves a simple task that both the simulated and real robots can perform. The comparison of the desired trajectory provides a mechanism for determining the correspondence between the simulated environment and the real world. This determined correspondence can then be used to enable the real-world robot to imitate the robot in the simulation for a new task that the robot in the real robot has not previously performed. This can significantly reduce the time required for real robots to learn new tasks, and thus reduce the risk of damage and operating costs. Summary of the Invention

[0003] Some embodiments of the present invention are based on the understanding that a system for trajectory mimicking a robot manipulator can be provided. In this case, the system may include: an interface configured to receive multiple task descriptions, wherein the interface is configured to communicate with a real-world robot; a memory for storing a computer-executable program including a robot simulator, a training module, and a transfer module; and a processor connected to the memory, configured to: train the robot simulator against the task descriptions using the training module to generate multiple source policies with sub-objectives for the task descriptions; train the real-world robot against the task descriptions using the training module to generate multiple target policies with sub-objectives for the task descriptions; and update the parameters of the transfer module based on corresponding trajectories with sub-objectives for the robot simulator and the real-world robot.

[0004] Furthermore, according to some embodiments of the present invention, a computer-executable method for trajectory mimicking of a robot manipulator can be provided. The computer-executable method is executed via an interface connected to a processor and a memory storing a computer-executable program, the computer-executable program including a robot simulator, a training module, and a transfer module. The method includes the following steps: receiving a plurality of task descriptions, wherein receiving is performed to communicate with a real-world robot via an interface; training the robot simulator with the training module for the task descriptions to generate a plurality of source policies having sub-objectives for the task descriptions; training the real-world robot with the training module for the task descriptions to generate a plurality of target policies having sub-objectives for the task descriptions; and updating the parameters of the transfer module according to corresponding trajectories having sub-objectives for the robot simulator and the real-world robot.

[0005] Furthermore, some embodiments of the present invention can provide a computer-executable training method for training a transfer module neural network. In this case, the computer-executable training method includes the following steps: acquiring a sequence of simulated trajectory data points of a simulated robot; acquiring a sequence of real trajectory data points of a real robot; mapping between the simulated trajectory data points, wherein the mapping is repeated for all sequences between the simulated robot trajectory and the real robot trajectory; and storing parameters associated with the learned mapping between the simulated robot trajectory and the real robot trajectory for use in the transfer module neural network.

[0006] The accompanying drawings are included to provide a further understanding of the invention, illustrate embodiments of the invention, and, together with the description, explain the principles of the invention. Attached Figure Description

[0007] [ Figure 1 ]

[0008] Figure 1 A block diagram of a trajectory imitation system for determining the correspondence between trajectories from different environments for the purpose of transferring tasks, according to some embodiments of the present invention, is shown.

[0009] [ Figure 2 ]

[0010] Figure 2 Illustrations of physical robot systems according to some embodiments of the present invention are shown.

[0011] [ Figure 3A ]

[0012] Figure 3A Trajectories with sub-targets are depicted according to some embodiments of the present invention.

[0013] [ Figure 3B ]

[0014] Figure 3B The trajectory without any sub-targets is depicted according to some embodiments of the present invention.

[0015] [ Figure 4 ]

[0016] Figure 4 The correspondence between trajectories through sub-targets according to some embodiments of the present invention is depicted.

[0017] [ Figure 5A ]

[0018] Figure 5A A block diagram of a robot controller according to some embodiments of the present invention is shown.

[0019] [ Figure 5B ]

[0020] Figure 5B A block diagram illustrating an embodiment of a trajectory generator according to some embodiments of the present invention is shown.

[0021] [ Figure 6 ]

[0022] Figure 6 A block diagram of a robot simulator according to some embodiments of the present invention is shown.

[0023] [ Figure 7A ]

[0024] Figure 7A The present invention illustrates a processing flow for a training module for training a source policy with sub-objectives according to some embodiments thereof.

[0025] [ Figure 7B ]

[0026] Figure 7B The present invention illustrates a processing flow for a training module for training a source policy with sub-objectives according to some embodiments thereof.

[0027] [ Figure 7C ]

[0028] Figure 7C The present invention illustrates a processing flow for a training module for training a target strategy with sub-targets according to some embodiments of the present invention.

[0029] [ Figure 7D ]

[0030] Figure 7D The present invention illustrates a processing flow for a training module for training a target strategy with sub-targets according to some embodiments of the present invention.

[0031] [ Figure 8 ]

[0032] Figure 8 Training for a transfer module according to some embodiments of the present invention is described.

[0033] [ Figure 9 ]

[0034] Figure 9 According to some embodiments of the invention, a trajectory is generated from a trajectory associated with a new task using a transfer module. Detailed Implementation

[0035] Various embodiments of the invention are described below with reference to the accompanying drawings. It should be noted that the drawings are not drawn to scale, and elements having similar structures or functions are indicated by similar reference numerals throughout the drawings. It should also be noted that the drawings are intended only to aid in the description of specific embodiments of the invention. The drawings are not intended as an exhaustive description of the invention or as a limitation on the scope of the invention. Furthermore, the aspects described in connection with specific embodiments of the invention are not necessarily limited to those embodiments and can be practiced in any other embodiment of the invention.

[0036] Figure 1A block diagram of a trajectory imitation system is shown for determining the correspondence between trajectories from different environments for the purpose of transferring tasks. The trajectory imitation system 100 is configured to learn parameters of a transfer module that can translate trajectories demonstrated by a simulated robot, where the trajectories are task-related, and the translated trajectories are used to perform the task on a real robot. A physical robot system 195 represents an implementation of a real-world robotic arm. The robot system can also be represented as a robotic arm, a robotic manipulator, or simply a robot. A classifier, "real-world" or "physical," compares the physical system to a simulated model of the robot. The trajectory imitation system 100 includes a processor 104 configured to execute stored instructions, and a memory 108 storing instructions executable by the processor. The processor 104 can be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 108 can include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. The processor 104 is connected to one or more input and output devices via a bus 110.

[0037] The trajectory imitation system 100 may include a storage device 106 adapted to store a task description 106a related to the task to be performed by the robot 195 or robot simulator 135. The storage device 106 may be implemented using a hard disk drive, optical drive, thumb drive, drive array, or any combination thereof. The storage device 106 also includes a training module 106d. The training module 106d is responsible for learning a source policy 106b with sub-objectives and a target policy 106c with sub-objectives. The learned source policy 106b with sub-objectives can be used to cause the robot simulator 135 to perform the task described by the task description 106a, and generate a trajectory 170 while doing so. The learned target policy 106c with sub-objectives can be used by the robot controller 138 to cause the robot 195 to perform the task described by the task description 106a, and generate a trajectory 170 while doing so. The training module 106d is stored as a collection of algorithms, including mathematical formulas, implemented as computer code using a programming language, wherein the code is interpreted and executed by the processor 104. Storage device 106 also includes a transfer module 106e, which stores a collection of algorithms, including mathematical formulas, implemented as computer code using a programming language, wherein the code is interpreted and executed by processor 104. Transfer module 106e can take a demonstration task executed by robot simulator 135 and provided as trajectory 170, and convert it into another trajectory 170 to be executed by real robot 195. Finally, storage device 106 includes robot simulator 135. Robot simulator 135 is stored as a collection of algorithms, including mathematical formulas, implemented as computer code using a programming language, wherein the code is interpreted and executed by processor 104.

[0038] Task description 106a contains a description of the task performed by the robot. One example of a task is an end effector moving from an initial position in 3D space to another position in 3D space; another example is opening a gripper end effector, moving to a position in 3D space, closing the gripper to grasp an object, and moving to a final position in 3D space. Typically, a task has a start condition and an end condition, referred to as the task objective. Once the task objective is achieved, the task is considered complete. A task can be divided into subtasks. For example, if the task is to move the robot to a 3D position in Cartesian space and then pick up an object, the task can be broken down into subtasks: (1) moving to the 3D position; and (2) picking up the object. It should be understood that more complex tasks can be broken down into subtasks by a program before the subtasks are provided as input to the trajectory imitation system 100. If a task is broken down into subtasks, a task description 106a can be provided for each subtask. In one implementation, the task description can be provided by a human operator. In another implementation, a program is executed to obtain the task description 106a for imitation from the trajectory system 100. Task specification 106a may also include constraints for the robot. An example of such a constraint is that the robot cannot move faster than a certain speed specified by the human operator. Another example of a constraint is prohibiting the robot from entering a portion of a 3D Cartesian workspace specified by the human operator. The robot's objective is to complete the task specified in task specification 106a as quickly as possible, given any constraints in task specification 106a.

[0039] The human-machine interface 102 within the trajectory emulation system 100 can connect the system to a keyboard 130 and a pointing device 132, wherein the pointing device 132 may include a mouse, trackball, touchpad, joystick, pointing stick, stylus, or touchscreen, and others. The system 100 can be linked via a bus 110 to a display interface 114 suitable for connecting the system 100 to a display device 124, wherein the display device 124 may include a computer monitor, camera, television, projector, or mobile device, and others.

[0040] The trajectory imitation system 100 includes a data processing module 137 connected via a bus 110 to process received robot states and sensor data received from environmental sensors 139. A robot controller 138 controls the robot's movement to complete the task defined by task specification 106a. In a preferred embodiment, the robot controller 138 receives input from the trajectory imitation system 100 and sends commands to the physical robot system 195. In another embodiment, the robot controller 138 is integrated into the robot system 195. In yet another embodiment, the robot controller 138 is part of the trajectory imitation system 100.

[0041] Network interface controller 112 is adapted to connect trajectory mimicry system 100 to network 134 via bus 110. In a preferred embodiment, data from a robot controller is received via network 134 and provided as input to data processing 137 via bus 110 for further processing, for example, by robot simulator 135. In another embodiment, robot controller 138 is connected to trajectory mimicry system 100 via a dedicated connection (e.g., using a data serial cable). In a preferred embodiment, one or more environmental sensors 139 are connected to network 134. In another embodiment, sensors may be connected via a dedicated connection (e.g., using a data serial cable). Environmental sensors 139 may include one or more color cameras, position sensor arrays, or depth cameras. An example of a position sensor is a camera capable of tracking the position of a set of infrared markers attached to an object. It should be understood that other sensors or combinations of sensors may be used. Sensor data may be processed by data processing 137 in trajectory mimicry system 100. An example of sensor data may be the position of infrared markers in 3D Cartesian space. Another example of sensor data may be a point cloud obtained from a depth camera.

[0042] Figure 2A diagram of a physical robot system is shown. In this specification, the physical robot system 195 is a set of components 211, 213, 215, and 217 linked by joints 210, 212, 214, 216, and 218. In the described embodiment, joints 210, 212, 214, 216, and 218 are rotary joints, but in another embodiment, these joints may be sliding joints or other types of joints. The set of joints determines the degrees of freedom of the robot arm 195. The robot arm 195 has five degrees of freedom, one for each joint 210, 212, 214, 216, and 218. In another embodiment, the robot may contain six joints. An end effector 220 is attached to the robot arm 195. The end effector 220 is attached to one of its components, and when considered in a chain, the end effector 220 is typically attached to the last component 217. The end effector 220 may be a parallel jaw gripper. The parallel jaw gripper has two parallel fingers whose distance is adjustable relative to each other. Alternatively, many other end effectors can be used, such as end effectors containing welding nozzles. Joints 210, 212, 214, 216, and 218 can be adjusted to achieve a desired configuration of the component. The desired configuration may involve a desired position in Euclidean space or a desired value in joint space. Joints can also be commanded in the time domain to achieve desired (angular) velocities and / or (angular) accelerations. The joints have embedded sensors that can report the joint's state. The reported state can be angle, current, velocity, torque, acceleration, or any combination thereof. The set of reported joint states is referred to as robot state 260. Commands for the robot are received from robot controller 138 via connection 250, and robot states are received by robot controller 138 via connection 250. In a preferred embodiment, connection 250 is a dedicated data cable. In another embodiment, connection 250 is an Ethernet cable. The robot can perform a variety of different tasks. For example, the robot can pick up objects to be placed in a bin. Object 240 typically appears on some working surface 230. The position and orientation of the working surface and object 240 may be part of the task description 106a.

[0043] When we refer to a robot without using the categories of "physical," "real," or "real world," this can refer to a physical robot system 195 or a robot simulator 135 designed to faithfully simulate the behavior of a physical robot system 195. The robot simulator 135 is a program consisting of a set of algorithms based on mathematical formulas to simulate the kinematics and dynamics of a real-world robot. In a preferred embodiment, the robot simulator 135 also simulates a robot controller 138. The robot simulator 135 can generate data for 2D or 3D visualization of the robot, which can be output to a display device 124 via a display interface 114.

[0044] A policy represents instructions used to make the robot decide what action it should take. A policy can simply be a sequence of robot actions or states 260. More complex policies are represented by a function that takes robot states 260 as input and provides the robot with the action to take. An action refers to a value that should be applied to robot joints 210, 212, 214, 216, and 218. A policy can be deterministic, meaning it provides a single value for each joint. A policy can also be stochastic, providing a probability distribution over the values ​​of the joints. In a preferred embodiment, the policy takes robot states 260 as input and is represented by a neural network. In another embodiment, the policy can be represented by an analytical function that takes robot states 260 as input. A policy can be executed over multiple consecutive time steps, meaning that for robot state 260 at some initial time step t0, the policy provides the robot with an action, which is then executed by the robot. The robot will then reach a new robot state 260 at time step t1. The policy is then executed using this robot state 260 at time step t1. This is repeated a desired number of time steps, or until the robot receives an indication that the task has been completed. This repeated execution of the strategy results in a trajectory.

[0045] Figure 3A A trajectory with sub-objectives is depicted. Trajectory 170 is associated with a task or sub-task that the robot should perform. Trajectory data points 310 are a sequence of values ​​in a certain time order at discrete time steps. For example, these values ​​could be coordinates of 3D positions in Cartesian space. Another example of values ​​could be a vector of joint angles of each of the robot's joints. Other values ​​such as velocity and acceleration could also be used. Trajectory data points 310 can be stored in robot state 260 at each discrete time step. Depending on the number of discrete time steps, trajectory 170 is said to have a length T. The starting point of trajectory 170 is at time step t1, and the ending point of trajectory 170 is at time step t2. T The trajectory data point 310 is depicted as a continuous line, but this line is actually a set of discrete points. The continuous line is used merely for simplification of the diagram. Each time step along the trajectory 170 is a sub-target 320. A sub-target 320 is a specific data trajectory point 310 at a certain discrete time step t. Sub-targets 320 can be stored as time step values ​​t. Alternatively, trajectory data point 310 can contain sub-target index values. For example, given three sub-targets s1, s2, and s3 stored as integers, trajectory data point 310 can contain integers s1, s2, or s3. In the depiction, the four sub-targets 320 at time step t... i t j t k and tl The location of the sub-target at time step t is stored. The time step location of the sub-target depends on the task associated with trajectory 170. Sub-target 320 essentially divides trajectory 170 into a sequence of trajectory data points 310. This sequence can have different lengths determined by the location of the sub-target 320. For example, at time step t i and t j The length of the sequence between sub-targets 320 is: t j -t i Trajectory 170 may not have any sub-target 320 at all.

[0046] Figure 3B A trajectory 170 is depicted, consisting solely of trajectory data points 310 and devoid of any sub-targets. Trajectory 170 can be generated by a human operator, software, robot controller, or strategy. It should be understood that other mechanisms may also generate trajectory 170.

[0047] Sub-targets 320 can be stored on the trajectory imitation system 100 as part of a source policy 106b with sub-targets and / or a target policy 106c with sub-targets. In a preferred embodiment, the sub-targets 320 are generated by a neural network that takes the trajectory 170 generated from the policy as input. In a preferred embodiment, the number of sub-targets 320 is predetermined by a human operator. In another embodiment, the number of sub-targets 320 can be learned by analyzing the trajectory 170. An example of trajectory analysis is determining whether the change in value in the trajectory between two subsequent time steps is greater than a certain threshold, in which case one of the time steps is considered a sub-target.

[0048] Figure 4 The correspondence between trajectories passing through sub-targets is depicted. Trajectory data point 310, along with sub-target 320, is considered to be generated in the source domain. The source domain could be, for example, the domain of a simulated robot. We can refer to trajectory 310 in the source domain as the source trajectory. Additionally, we refer to the policy in the source domain as the source policy. Another trajectory data point 430, along with sub-target 440, is considered to be generated in the target domain. The target domain could be, for example, the domain of a real robot. We can refer to trajectory 430 in the target domain as the target trajectory. Additionally, we refer to the policy in the target domain as the target policy. Both trajectories are instances of trajectory 170. Sub-targets 320 and 440 in different domains can occur along each trajectory at different time steps. In a preferred embodiment, trajectories 310 and 430 have the same number of sub-targets. Since trajectories 310 and 430 are associated with the same task, sub-targets 320 and 440 provide a correspondence 450 between trajectories. The lengths of trajectory data points 310 and 430 can be different. For example, trajectory data point 310 has a length T. s The trajectory data point 430 has a length Tt .

[0049] Figure 5A A block diagram of the robot controller is shown. In a preferred embodiment, the robot controller 138 takes a task description 106a and a target strategy 106c with sub-objectives as input. Alternatively, instead of the target strategy 106c with sub-objectives, a trajectory 170 consisting of trajectory data points 310 and possible sub-objectives 320 can be given as input. The trajectory generator 510 generates the desired trajectory to be followed by the robot. At time t = T, the robot is in a final configuration as determined by the task objective in the task description 106b. The control command generator 520 takes the desired trajectory generated by the trajectory generator 510 and the robot state 260 as input and generates commands to send to the physical robot 195. An example of a command generated by 520 is the torque applied to the motors of the robot 195. The control command generator generates commands for each time step in the desired trajectory generated by the trajectory generator 510. In another embodiment, the trajectory generator can be separate from the robot controller 138. For example, the trajectory generator can be stored as an executable program in the trajectory imitation system 100.

[0050] Figure 5B A block diagram illustrating an implementation of trajectory generator 510 is shown. Task description 106a and target strategy 106c with sub-objectives are input to planner 530. Planner 530 is responsible for the initial path or trajectory. Planner 530 can be a simple linear interpolation between start and end positions. However, planner 530 can also be more complex and avoid collisions with objects in the environment or with the physical robot 195 itself. Planner 530 generates paths that take into account the robot's physical capabilities. For example, it avoids paths requiring speeds exceeding the maximum joint speed. Command shaping 540 takes the output from planner 530 into account and transforms the output from planner 530 to meet the task objective. The transformed path or trajectory output from command shaping 540 undergoes trajectory time scaling 550. Trajectory time scaling ensures that the time steps and time intervals in the trajectory are valid for the physical robot 195. In one implementation, command shaping 540 and trajectory time scaling 550 are interchanged. It should be understood that trajectory generator 510 can be implemented in different ways. Alternatively, the planner can take task description 106a and trajectory 170 as input, the trajectory 170 consisting of trajectory data points 310 and possible sub-objectives 320. The planner creates its path directly from the input trajectory 170.

[0051] The output from trajectory generator 510 is the desired trajectory, data point 310, representing the value at the discrete time step. The output from trajectory generator 510 can be stored as trajectory 170 in trajectory imitation system 100. Controller 138 will move the robot in a manner that closely follows the desired trajectory. The desired trajectory will be represented as:

[0052] q d (t) (1)

[0053] q d The value at each time step t in (t) can be one or more scalar or vector values. An example of three scalar values ​​at time step t is, for example, (p(t), v(t), a(t)) for an end effector, which includes position, velocity, and acceleration in Cartesian space. An example of a vector value at time step t is... These represent the angle, angular velocity, and angular acceleration vectors of the joint, respectively. It should be understood that different combinations and values ​​can be obtained.

[0054] Figure 6A block diagram of a robot simulator 135 is shown. In a preferred embodiment, the robot simulator 135 includes a simulation controller 610 and a robot dynamics model 620. In a preferred embodiment, the robot simulator 135 takes a task description 106a and a source policy 106b with sub-objectives as input. Alternatively, instead of the source policy 106b with sub-objectives, a trajectory 170 consisting of trajectory data points 310 and possible sub-objectives 320 can be given as input. The simulation controller 610 is a program designed to replicate the behavior of the robot controller 138. The simulation controller 610 generates input for the robot dynamics model. The robot dynamics model 620 is a program based on known formulas and physical laws, designed to replicate the dynamic behavior of the physical robot 195. The robot simulator generates one or more simulated robot states 630. In a preferred embodiment, the simulated robot state 630 contains the same values ​​as robot state 260. For example, if robot state 260 stores vectors of robot joint angles, then simulated robot state 630 stores vectors of simulated robot joint angles. In another embodiment, the values ​​stored in robot 260 and simulated robot state 630 are different. Robot simulator 135 also includes external sensing simulation 625. External sensing simulation 625 uses outputs from robot dynamics model 620 and external sensor model 627 to generate simulated sensor output 640. In a preferred embodiment, external sensor model 627 is an RGB camera model, and external sensing simulation 625 is a computer graphics program that creates images as simulated sensor output 640 for simulating robot configuration, derived from the output of robot dynamics model 620. In another embodiment, external sensor model 627 is a depth camera. It should be understood that other external sensor models and algorithms can be used alternatively to create simulated sensor outputs. In another embodiment, robot simulator 135 does not generate simulated sensor outputs, but only generates simulated robot state 630. The set of simulated robot states 630 can be stored as trajectory 170.

[0055] Figure 7AThe processing flow of a training module 106d for training a source policy 106b with sub-objectives is illustrated. The training module 106d takes a task description 106a and a simulated robot state 630 as input. In a preferred embodiment, the training module 106d also takes a simulated sensor output 640 as input. The training module processes the inputs and generates updated parameters for the source policy 106b with sub-objectives. During the training process, the training module generates one or more tasks to be performed by the robot simulator 135. In a preferred embodiment, the processing of the simulated robot state 630 and the simulated sensor output 640 is performed by a neural network. Furthermore, in a preferred embodiment, the updated parameters for the source policy with sub-objectives are determined by another neural network. In another embodiment, the neural network can be replaced by a parameterized function. In yet another embodiment, the training module uses a lookup table to store information associated with a specific simulated robot state and simulated sensor output. The lookup table is a memory indexed by specific values ​​(e.g., robot). In a preferred embodiment, the training module 106d uses reinforcement learning to determine the weights associated with the neural network. Reinforcement learning is an iterative method designed to learn, given a state, to take the action that provides the highest possible reward value, such as moving the robot 1 cm to the left. The reward value is the numerical value provided to reinforcement learning. An example of reinforcement learning is the policy gradient method. The policy gradient method adjusts the policy parameters based on gradient descent optimization. The training module 106d uses the task description 106a to determine how much progress has been made toward task completion. During training, the training module 106d generates inputs for the robot simulator 135. If a predefined level of accuracy for task completion has been reached, the training module 106d completes its execution. For example, the training module 106d evaluates whether a source policy with sub-objectives results in the robot simulator achieving the task objective with a predefined success rate (measured as a percentage). Alternatively, the execution of the training module 106d completes if a predefined amount of time or a predefined number of execution steps have been performed. The updated parameters of the source policy with sub-objectives are stored during the execution of the training module 106d. After the training module 106d terminates, the parameters of the source policy 106b with sub-objectives are stored for later use.

[0056] Figure 7B The processing flow for training module 106d, used to train source policy 106b with sub-objectives, is shown. However, the input to training module 106d is trajectory 170 generated by robot simulator 135.

[0057] Figure 7CThe processing flow of training module 106d for training a target policy 106c with sub-objectives is shown. Training module 106d is the same as the training module described above for training the source policy 106b with sub-objectives. To train the target policy 106c with sub-objectives, training module 106d takes task description 106a and robot state 260 as inputs. In a preferred embodiment, training module 106d also takes the output from external sensor 139 as input. The training module processes the inputs and generates updated parameters for the target policy 106c with sub-objectives. During the training process, the training module generates one or more tasks to be performed by robot simulator 135. In a preferred embodiment, the processing of robot state 260 and the output from environmental sensor 139 is performed by a neural network. Furthermore, in a preferred embodiment, the updated parameters of the target policy 106c with sub-objectives are determined by another neural network. In another embodiment, the neural network can be replaced by a parameterized function. In yet another embodiment, the training module uses a lookup table to store information associated with a specific simulated robot state and simulated sensor outputs. During training, training module 106d generates inputs for robot controller 138. If a predefined level of accuracy for task completion has been reached, training module 106d completes its execution. For example, training module 106d evaluates whether a target policy with sub-objectives 106c results in the robot simulator achieving the task objective with a predefined success rate (measured as a percentage). Alternatively, if a predefined amount of time or a predefined number of execution steps have been performed, the execution of training module 106d is complete. Updated parameters for the target policy 106c with sub-objectives are stored during the execution of training module 106d. After training module 106d terminates execution, the parameters of the target policy 106c with sub-objectives are stored for later use.

[0058] Figure 7D The processing flow of the training module 106d for training a target policy 106c with sub-objectives is shown. However, the input to the training module 106d is the trajectory generated by the robot controller 138 and the robot 195.

[0059] Training module 106d is executed for both the simulated robot and the real robot using the same tasks as defined in task specification 106a. Training module 106d is executed for n different tasks to enable robot simulator 135 to generate a source policy 106b with sub-objectives, and to enable robot 195 to generate a target policy 106c with sub-objectives. For each task, the policy with sub-objectives is stored. Therefore, for n different tasks (each specified as task specification 106a), a total of n policies with associated sub-objectives will be trained and stored. For the real robot, system 100 stores n realThere are 106c target strategies with sub-objectives, while for a simulated robot, the system stores n sim A source strategy 106b with sub-objectives.

[0060] Figure 8 The training for the transfer module is described. The transfer module 106e aims to learn the correspondence between trajectories generated by a source policy 106b with sub-objectives and trajectories generated by a target policy with sub-objectives. In a preferred embodiment, the source policy 106b with sub-objectives is associated with a robot simulator, while the target policy 106c with sub-objectives is associated with a real-world robot. We define n = n real =n sim For index i, where i is in the set {1,..,n}, a source trajectory 810 with sub-objectives is generated using the i-th source policy with sub-objectives and robot simulator 135. Trajectory 810 has form 170. For the same index i, a target trajectory 820 with sub-objectives is generated using the i-th target policy with sub-objectives, robot controller 138, and robot 195. Target trajectory 820 with sub-objectives has form 170. Transfer module 106e learns the correspondence between simulated robot trajectories and real robot trajectories. In a preferred embodiment, this learning is performed using a neural network (transfer module neural network). The neural network takes as input a sequence of values ​​(i.e., trajectory data points) between two sub-objectives in both simulated robot trajectory 810 and real robot trajectory 820. The neural network then learns the mapping between these sequences. This mapping learning is repeated for all sequences between sub-objectives in simulated robot trajectory 810 and real robot trajectory 820. In another embodiment, the mapping can be learned as a Markov decision process homomorphism, which is a mapping between corresponding structures in two Markov decision processes. It should be understood that other implementations may use different learning methods. The learning of the mapping can be repeated several times using the i-th policy. Furthermore, the training of the transfer module 106e is repeated for all n source and target policies with sub-goals. After a desired termination criterion is met, training of the transfer module is stopped, and parameters associated with the learned mapping between the simulated robot and the real robot for the transfer module 106e are stored. The desired termination criterion may be a desired duration or a desired number of execution steps. Another termination criterion may be determined based on the accuracy of the learned mapping for the transfer module 106e.

[0061] Figure 9The diagram describes the generation of trajectory 920 from trajectory 910 associated with a new task using a transfer module. First, a new task specified by task description 106a is trained using a robot simulator with training module 106d to store a source policy 106b with sub-objectives for the new task. The source policy 106b with sub-objectives for the new task and the robot simulator are used to generate trajectory 910. Trajectory 910 has form 170. Then, transfer module 106e is used to transform trajectory 910 to generate trajectory 920. Trajectory 920 also has form 170. In another embodiment, trajectories 910 and 920 do not contain any sub-objectives.

[0062] Since the transfer module 106e is trained on a trajectory associated with a source policy and a target policy having sub-targets, and the source policy 106b having sub-targets is associated with a robot simulator, while the target policy 106c having sub-targets is associated with a real robot 195, trajectory 920 is a trajectory for the real robot 196. Therefore, trajectory 920 can be used to perform a new task on the real robot. Trajectory 920 can be input to robot controller 138. Alternatively, the learning module 106d can learn the target policy 106c having sub-targets from the trajectory 920 for a new task. The learned target policy 106c having sub-targets is then executed to perform the new task on the real robot.

[0063] For a new task, the trained source policy 106b with sub-objectives can generate more than one trajectory 910. Each such trajectory 910 is transformed to generate a corresponding trajectory 920. The learning module 106d can learn a target policy 106c with sub-objectives from multiple trajectories 920.

[0064] While the invention has been specifically shown and described with reference to exemplary embodiments thereof, the invention is not limited to these embodiments. Those skilled in the art will understand that various changes to the form and detail of this disclosure may be made without departing from the spirit and scope of the invention as defined by the claims.

Claims

1. A system for trajectory mimicry of a robot manipulator, the system comprising: An interface configured to receive multiple task descriptions, wherein the interface is configured to communicate with a real-world robot; The memory is used to store a computer-executable program including a robot simulator, a training module, and a transfer module, wherein the transfer module is used to transform the simulated trajectory of the robot simulator into the real trajectory of the real-world robot. A processor, connected to the memory, is configured to: The robot simulator is trained using the training module in response to the task description to generate multiple source policies with sub-objectives and associated with the robot simulator in response to the task description. The training module is used to train the real-world robot against the task description to generate multiple target policies with sub-objectives and associated with the real-world robot for the task description. Obtain the simulated robot trajectory generated by the source strategy and the real robot trajectory generated by the target strategy, wherein the simulated robot trajectory is defined by a plurality of first trajectory data points and the real robot trajectory is defined by a plurality of second trajectory data points; The correspondence between the simulated robot trajectory and the real robot trajectory is determined as a mapping between a sequence of points representing multiple first trajectory data points between two sub-targets on the simulated robot trajectory and a sequence of points representing multiple second trajectory data points between two sub-targets on the real robot trajectory; and The parameters of the transfer module are updated based on the correspondence between the simulated robot trajectory and the real robot trajectory to train the transfer module, wherein the transfer module is represented by a neural network, and the training of the transfer module updates the parameters of the neural network.

2. The system according to claim 1, wherein, The trajectory is associated with the task description.

3. The system according to claim 1, wherein, The updated transfer module takes the source trajectory as input and generates the target trajectory as output.

4. The system according to claim 1, wherein, The training module takes data from environmental sensors as input.

5. The system according to claim 4, wherein, The environmental sensor is one or more color cameras.

6. The system according to claim 5, wherein, The environmental sensor is an array of position sensors.

7. The system according to claim 1, wherein, The robot simulator generates simulated sensor outputs.

8. The system according to claim 7, wherein, The simulated sensor output is generated using an external sensing simulation that takes the simulated robot state as input.

9. The system according to claim 1, wherein, The real-world robot is a robot manipulator controlled by a robot controller, and generates multiple robot states when performing tasks.

10. The system according to claim 1, wherein, The robot controller is stored in memory as a computer executable program.

11. The system according to claim 1, wherein, The robot simulator generates multiple robot states when performing tasks according to the task description.

12. The system according to claim 11, wherein, The robot simulator includes a simulation controller configured to generate one or more simulated robot states and a robot dynamics model.

13. A computer-executable method for trajectory mimicking of a robot manipulator, the robot manipulator being connected via an interface to a processor and a memory storing a computer-executable program, the computer-executable program including a robot simulator, a training module, and a transfer module, the transfer module being configured to transform the simulated trajectory of the robot simulator into a real-world trajectory of the robot, the computer-executable method comprising the following steps: Receive multiple task descriptions, wherein communication with a real-world robot is conducted via the interface; The robot simulator is trained using the training module in response to the task description to generate multiple source policies with sub-objectives and associated with the robot simulator in response to the task description. The training module is used to train the real-world robot against the task description to generate multiple target policies with sub-objectives and associated with the real-world robot for the task description. Obtain the simulated robot trajectory generated by the source strategy and the real robot trajectory generated by the target strategy, wherein the simulated robot trajectory is defined by a plurality of first trajectory data points and the real robot trajectory is defined by a plurality of second trajectory data points; The correspondence between the simulated robot trajectory and the real robot trajectory is determined as a mapping between a sequence of points representing multiple first trajectory data points between two sub-targets on the simulated robot trajectory and a sequence of points representing multiple second trajectory data points between two sub-targets on the real robot trajectory; and The parameters of the transfer module are updated based on the correspondence between the simulated robot trajectory and the real robot trajectory to train the transfer module, wherein the transfer module is represented by a neural network, and the training of the transfer module updates the parameters of the neural network.

Images