Systems and methods for learning sequences in robotic tasks for generalization to new tasks

Abstract

A robot controller for generating a sequence of movement primitives for sequential tasks of a robot having a manipulator is provided. The controller includes at least one control processor and a memory circuit having instructions stored therein, the memory circuit storing a dictionary including movement primitives, a pre-trained learning module, and a graph search-based planning module. The controller performs the following steps: acquiring a planned task provided by a user-operated interface device, the planned task being represented by an initial state and a goal state for an object; generating a planning graph for the new task by searching for feasible paths for the object using the graph search-based planning module and selecting movement primitives from the dictionary in the pre-trained learning module, the pre-trained learning module having been trained based on a demonstrated task; parameterizing the feasible paths represented by the movement primitives as dynamic movement primitives (DMPs) using the initial states and goal states; and realizing the parameterized feasible paths as trajectories for the planned task by tracking and following the parameterized feasible paths using the robot's manipulator according to the selected movement primitives.

Description

[Technical Field]

[0001] The present invention relates to learning sequences in sequential tasks, and more particularly to methods and apparatus for learning sequences in robotic tasks using these learned sequences and failures observed during demonstrations to perform new robotic tasks. [Background technology]

[0002] The fields of machine learning and artificial intelligence have shown tremendous improvements and successes in the areas of computer vision and natural language processing. However, these algorithms are data-inefficient when used in robotic applications, making them impractical for use in many robotic applications. Learning from Demonstration (LfD) is a data-efficient learning technique in which a robot can learn to perform a task by first recording several demonstrations of the task and then reproducing these demonstrations using an appropriate machine learning model.

[0003] In LfD, a robot is provided with one or several demonstrations of a desired task. Demonstrations of known tasks can be provided by a human or by a programmed controller. When demonstrations are provided by a human, the human can provide the demonstration directly on the robot or by performing the task themselves. In the latter case, the human's movements can be recorded using a motion capture system or a vision system consisting of one or more cameras. On the other hand, when demonstrations are provided directly on the robot, the human can provide the demonstration by moving the robot using kinesthetic instruction or by remotely controlling it using appropriate equipment. In all these cases, the robot's movements and the objects being manipulated by the robot are observed and recorded. This data is then used to learn or represent the robot's movements while performing the tasks indicated to it.

[0004] LfD techniques are widely used to reduce robot programming and allow unskilled workers to demonstrate tasks on robots. Robots can then replicate tasks using standard LfD techniques and perform them autonomously without the need for explicit human programming. The LfD representations learned to perform a task are called skills. However, many useful robot tasks are inherently sequential. For example, consider the task of assembling an electronic item. In such a task, the robot must be able to assemble all the different parts of the electronic item in the desired order to form the complete item. It is also desirable for the robot to be able to use the learned skill to assemble any new electronic item using the same subset of actions in the specified order.

[0005] For learning, LfD techniques require two key elements to autonomously execute these long-horizon tasks. First, the demonstration must be a sequence of different sequences or subtasks while executing the entire task. These individual sequences or subtasks can then be learned using appropriate machine learning models, parameterized by some parameters of the task. These learned subtask models are called skills. Second, the robot must optimize the sequence of these skills based on the new task the robot needs to perform. The new task can be accomplished by using all or a subset of the skills learned in part 1, executing the learned skills in a specific, unknown sequence.

[0006] Therefore, there is a need for a method that can automatically decompose long demonstrations into meaningful sequences and then optimally compose these sequences to perform novel tasks. Summary of the Invention

[0007] Some embodiments of the proposed disclosure recognize that designing controllers for long-horizon or sequential tasks is difficult. The main reason is that optimization-based techniques cannot find a solution because the search space of feasible solutions is too large. Reinforcement learning (RL) can conceivably find a solution, but this would require careful design of rewards and a huge amount of data to guide the RL agent to learn the solution. Such techniques are highly inefficient because they require prohibitive amounts of data. Furthermore, reward engineering for complex tasks is an extremely difficult problem.

[0008] Some embodiments recognize that learning from demonstration (LfD) can help learn efficient controllers for executing long-horizon tasks or sequential tasks. This is because a robot can get ideas on how to perform a task from an expert, either a human or a controller. The robot can use appropriate learning methods (e.g., dynamic locomotion primitives, SEDS, etc.). However, there are challenges that must be overcome when using LfD for long-horizon tasks with a robot. For example, if a task consists of several steps that must be completed for the task to be successful, it is difficult to learn the entire task as a single motor skill. Therefore, it is essential that the robot must identify the sequences in a long-horizon task that have been demonstrated to it.

[0009] It is an object of some embodiments to provide a system and method for identifying sequences in a performance demonstration for performing long-horizon or sequential tasks. Some embodiments of the present invention are based on the recognition that the segmentation of a task trajectory depends on a feature representation of the performance trajectory.

[0010] An objective of certain embodiments is to provide a system and method capable of detecting suitable features that can be used to identify sequences in a demonstrated trajectory. This problem is similar to feature identification or feature selection applicable to demonstrations collected for a robot, which can be used for trajectory segmentation. This method allows for a better segmentation of the demonstration trajectory.

[0011] Additionally or alternatively, an object of certain embodiments is to provide a system and method capable of accurately identifying different sequences and the changes between them by detecting appropriate features from the data. Additionally or alternatively, an object of certain embodiments is to provide a system and method capable of fitting a machine learning model to each of the identified sequences, parameterized by certain parameters of the task. Additionally or alternatively, an object of certain embodiments is to provide a system and method capable of using information from failed demonstration attempts to provide robustness to the detection of sequences.

[0012] Additionally or alternatively, it is an object of certain embodiments to provide a system and method that can generate an optimal sequence of executing a subset of sequences to perform a new task presented to a robot. Additionally or alternatively, it is an object of certain embodiments to provide a system and method that implements learned sequences of tasks in a feedback manner using an object state detection framework.

[0013] According to some embodiments of the present invention, there is provided a robot controller for generating a sequence of movement primitives for sequential tasks of a robot having a manipulator, the robot controller may include at least one control processor and a memory circuit having instructions stored therein. The memory circuit stores a dictionary including movement primitives, a pre-trained learning module, and a graph search-based planning module, and the instructions, when executed by at least the control processor, cause the robot controller to: obtain a planned task provided by a user-operated interface device, the planned task being represented by an initial state and a goal state for an object; for the new task, use the graph search-based planning module to search for feasible paths for the object; and generate a planning graph by selecting movement primitives from the dictionary in the pre-trained learning module, the pre-trained learning module being trained based on a demonstrated task; parameterize the feasible paths represented by the movement primitives as dynamic movement primitives (DMPs) using the initial states and goal states; and for the planned task, use the robot manipulator to track and follow the parameterized feasible paths as trajectories in accordance with the selected movement primitives.

[0014] Further, some embodiments may provide a robot controller for learning a sequence of movement primitives for sequential tasks of a robot having a manipulator, the robot controller including at least one control processor and a memory circuit having stored thereon instructions, the memory circuit storing a dictionary including the movement primitives and a learning module, the instructions, when executed by at least the control processor, causing the robot controller to perform the steps of: collecting demonstration data from trajectories obtained via a motion sensor, the motion sensor being configured to measure trajectories of objects while the objects are manipulated by an interface device operated by a user according to demonstration tasks, each of the trajectories corresponding to a respective demonstration task, each of the demonstration tasks being represented by an initial state and a goal state for each of the objects, the collection continuing until the user stops the demonstration task; and, for each of the demonstration tasks, segmenting the demonstration data into movement primitives by dividing the trajectory into primitive trajectories; and updating the dictionary using the movement primitives based on the collected demonstration data.

[0015] Further, according to some embodiments of the present invention, a robot controller for generating a sequence of movement primitives for a sequential task of a robot having a manipulator is provided. The robot controller may include at least one control processor and a memory circuit having stored thereon instructions. The memory circuit stores a dictionary including movement primitives, a pre-trained learning module, and a graph search-based planning module. The instructions, when executed by at least the control processor, cause the robot controller to perform steps of acquiring, via an interface controller, performance data of one or more performance tasks provided by an interface device operated by a user for a planned task, the planned task being represented by an initial state and a goal state for at least one object to be manipulated, each of the performance data being segmented into a plurality of segments by selecting features from the performance data based on a feature selection method and using a segmentation metric, each of the plurality of segments representing a subtask, and further performing steps of acquiring, via an interface controller, performance data of one or more performance tasks provided by an interface device operated by a user for a planned task, the planned task being represented by an initial state and a goal state for at least one object to be manipulated, each of the performance data being segmented into a plurality of segments by selecting features from the performance data based on a feature selection method and using a segmentation metric, each of the plurality of segments representing a subtask. The method further comprises the steps of: searching for feasible paths for at least one object using a graph search-based planning module; generating a planning graph by selecting movement primitives from a dictionary in a pre-trained learning module, the pre-trained learning module being trained based on collected demonstration data of a training demonstration task; parameterizing the feasible paths represented by the movement primitives as dynamic movement primitives (DMPs) using an initial state and a goal state; and realizing the parameterized feasible paths as trajectories for the planned task by tracking and following the parameterized feasible paths using a manipulator of the robot according to the selected movement primitives.

[0016] The present disclosure will be further described in the following detailed description with reference to the drawings, in which like reference numerals represent like parts in the several views, shown as non-limiting examples of exemplary embodiments of the present disclosure. The drawings shown are not necessarily to scale, rather emphasis is generally placed upon illustrating the principles of embodiments of the present disclosure.

[0017] While the above drawings illustrate embodiments disclosed herein, other embodiments are contemplated as indicated in the description. The present disclosure presents exemplary embodiments by way of representation and not by way of limitation. Numerous other modifications and embodiments can be devised by those skilled in the art which fall within the scope and spirit of the principles of the embodiments disclosed herein. [Brief explanation of the drawings]

[0018] [Figure 1] 1 shows a schematic diagram of a generalized learning paradigm according to an embodiment of the present invention; [Figure 2A] FIG. 1 shows a schematic diagram of a robotic system in which a robotic manipulator is controlled using several different interfaces, according to an embodiment of the present invention. [Figure 2B] FIG. 1 shows a diagram illustrating components of a controller connected to an interface device according to an embodiment of the present invention. [Figure 3] FIG. 1 shows a schematic diagram of how a particular task may be composed of different subtasks, which are different segments that the proposed method extracts during the learning procedure, according to an embodiment of the present invention. [Figure 4] 1 shows a schematic diagram of a block-building task and a state in a world coordinate frame for measuring the state, according to an embodiment of the present invention; [Figure 5] FIG. 1 shows a schematic diagram of metrics used to segment a task into different trajectories and corresponding examples where different features are required to segment the trajectories, according to an embodiment of the present invention. [Figure 6] FIG. 1 shows a schematic diagram of the dynamic motion primitives (DMPs) used in the proposed work for learning different skill representations, according to an embodiment of the present invention. [Figure 7] FIG. 1 is a schematic diagram showing the skills learned from the original demonstrated task that are required to be realized for a desired task not shown in the demonstration. [Figure 8] FIG. 1 illustrates a planning graph used in some embodiments of the present invention, where the initial node is the goal state of the task and additional nodes are added. [Figure 9] 1 shows a flowchart illustrating successive steps implemented in some embodiments of the present invention. [Figure 10] FIG. 1 is a schematic diagram illustrating segmentation of a multi-step peg-in-hole demonstration task for precise motion, according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0019] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other instances, devices and methods are shown in block diagram form solely to avoid obscuring the present disclosure.

[0020] As used in this specification and claims, the terms "for example," "for instance," and "such as," as well as the verbs "comprising," "having," "including," and other forms of these verbs, when used in conjunction with a list of one or more components or other items, should each be construed as open-ended, meaning that the list should not be considered to exclude further components or items. The term "based on" means based at least in part on. Furthermore, it should be understood that the style and terminology used herein are for purposes of description and should not be considered limiting. Any headings used herein are for convenience only and are not to be considered legal or limiting.

[0021] In robotics, designing controllers for long-horizon manipulation tasks remains extremely challenging. There are several reasons that make this task difficult. First, it is extremely difficult to find a solution for very long-horizon control using either model-based techniques or model-free RL-based approaches. Second, the success of the overall task depends on the success of each individual task. Therefore, these problems require careful formulation, in which the overall task can be decomposed into smaller subproblems and each individual subproblem can be reliably completed. To reduce the effort of designing these controllers, it is also desirable to use appropriate learning-based methods that can be trained in a data-efficient manner and generalized to new tasks. This disclosure presents systems and methods that can be used to reduce the programming burden for performing long-horizon tasks.

[0022] Reinforcement learning (RL)-based approaches have been extremely successful in many robotic manipulation tasks, but they suffer from data requirements during training and training for long-horizon tasks. As a result, the use of RL has been limited to short-horizon tasks, where robots can be trained with dense rewards; otherwise, this approach is very data-intensive. Learning from demonstration (LfD) offers an alternative learning-based approach that can utilize expert or human demonstration to learn motor techniques for different tasks. The systems and methods presented in this disclosure are motivated by this requirement, where the proposed methods are data-efficient and require less expert programming.

[0023] FIG. 1 shows a schematic diagram 110 illustrating the block stacking task, a sequential task in which a robot is required to sequentially place individual blocks based on the positions of other blocks in a scene. In this task, the robot is presented with blocks in its workspace so that the robot can observe the positions of the blocks. The basic idea of ​​the proposed learning is shown at 100. An expert user provides multiple demonstrations for performing the task. These demonstrations can then be segmented into individual segments using appropriate feature detection. These individual segments can then be represented as dynamic movement primitives (DMPs). The robot can then use these DMPs on new instances of the problem for autonomous execution.

[0024] Some embodiments recognize that LfD approaches offer a data-efficient alternative to RL-based approaches for designing learning-based controllers for long-horizon, multi-stage manipulation tasks. A robotic system can include a system for providing demonstrations to the robot to perform these tasks. The system can comprise at least one interface for moving the robot by a human expert. Some examples of such interfaces include a three-axis joystick, a spatial mouse, and a virtual or augmented reality system. These interfaces can be used to remotely move the robot. Alternatively, a human expert can demonstrate the task on the robot using a kinesthetic controller for the robot, where the robot can be moved directly by applying forces to the robot arm.

[0025] Alternatively, performance data can be collected in a simulation by creating a simulated environment similar to the physical environment and using a joystick or similar interface, such as a virtual or augmented reality interface, to move the robot within the simulated environment and collect performance data.

[0026] 2A shows a robotic system 200 including a controller (robot controller) 205 and a robotic manipulator 210 configured to stack blocks 220 into a desired shape. The controller 205 of the robotic system 200 is connected to an interface device 230, which is configured to be operated by a user / operator who uses the interface device 230 to perform tasks for the robotic system 200. The robotic system 200 includes the robotic manipulator 210, which has an actuator 2103 and sensors 2101, including a vision sensor (3D sensor) 2102, disposed on the robotic manipulator 210. The motion sensor 2101 may include an acceleration sensor, a position sensor, a torque sensor, and a force sensor. The controller 205 acquires signals measured by the motion sensor 2101 via an interface controller 2110B, which includes an analog-to-digital (A / D) signal converter and a digital-to-analog (D / A) signal converter. In this case, the interface device 230 includes a network interface (not shown) configured to connect to the controller 205 of the robotic system 200 via the communication network 215. In some cases, the communication network 215 may be a wired or wireless network. The interface device 230 is configured to control the robot manipulator 210 via the controller 205, and is provided with demonstration trajectories for a user to operate to learn a sequence of robotic tasks using signals from the sensors 2010 and actuators 2103. For example, the interface device 230 may be any type of joystick 240, 250, or 260 configured to be used / operated by a user / operator. The operator may also use a virtual reality game engine controller to move the robot during these demonstrations. These illustrations are not exhaustive, and humans may use other interaction modes to demonstrate tasks corresponding to the demonstration trajectories.

[0027] The type of task we focus on is a long-horizon task, which is a composition of several subtasks. Assume that a human expert provides several demonstrations of such a long-horizon task. During the demonstration, observations are recorded from different sensors available to the robotic system. Note that the robotic system may include encoders on the robot arm, a vision system for tracking objects in the robot's working environment, and force sensors for observing the forces experienced by the robot end effector during the task demonstration. The robotic system may also include other sensors with other sensing modalities, such as tactile sensors that can provide more detailed information about the contact forces and moments during the manipulation task being demonstrated on the gripper fingers. Thus, the demonstration trajectory is represented by the sequence of sensor trajectories collected by the robotic system during the task demonstration. At any point in time, we represent the state of the robotic system as the collection of the poses of the end effector (or the robot's gripper tip) and all the objects in the robot's workspace.

[0028] 2A illustrates a controller 205 configured to control a manipulator 210 to demonstrate a sequence of steps used by a manipulator arm 2155 of a robotic system 200 to perform a desired manipulation task, according to an embodiment of the present invention. The robotic system 200 may also be referred to as a robot.

[0029] The robot system 200 includes a manipulator 210, a force sensor 2101 disposed on the manipulator 210, and a vision system 2102 (at least one camera). The force sensor 2101, which may also be referred to as at least one force sensor, is configured to detect a force applied to an object by the manipulator 210 at a contact point between the object and the manipulator. The vision system 2102 may be at least one camera or multiple cameras, a depth camera, a range camera, etc. The vision system 2102 is positioned such that the vision system 2102 can observe an object state representing a positional relationship between the object, a tabletop (not shown), and an additional contact surface. The vision system 2102 is configured to estimate the pose of an object on the tabletop with the additional contact surface in the environment of the robot system 200.

[0030] The vision system 2102 is configured to detect and estimate the pose of an object to be manipulated on the tabletop. The controller 205 is configured to determine if parts need to be pre-orientated before they can be used for a desired task (e.g., assembly). The controller 205 is configured to calculate a sequence of control forces to be applied to the object using a bi-level optimization algorithm. The robot 200 applies a sequence of control forces (a sequence of contact forces) to the object, directed at the external contact surface, according to control signals sent from the interface device 230.

[0031] Furthermore, the controller 205 is configured to acquire simulation data and learning data via the communication network 215. The simulation data and learning data generated in the computer (simulation computer system) 2500 is configured to be used in the robot system 200. The collected simulation data and learning data is transmitted to the controller 205 via the communication network 215.

[0032] The controller 205 is configured to generate and send control data including instructions for a sequence of calculated control forces to a low-level robot controller (e.g., a manipulator actuator controller) that causes the manipulator to apply the sequence of calculated control forces (contact forces) to the tabletop. The robot 200 is configured to grasp the reoriented part and use it for a desired task (assembly or packaging) on ​​the tabletop.

[0033] 2B illustrates a robotic system (robot) 200 for manipulating an object on a tabletop (not shown) according to a trajectory generated by the proposed robust trajectory optimization problem, according to an embodiment of the present invention. The robot control system 2100 is configured to control the actuator system 2103 of the robot 2150. The robot control system 200 may also be referred to as the control system of the robot or robot controller.

[0034] The robotic control system 200 may include an interface controller 2110B, a control processor 2120 (or at least one control processor), and a memory circuit 2130B. The memory circuit, sometimes referred to as a memory unit or memory module, may include one or more static random access memories (SRAMs), one or more dynamic random access memories (DRAMs), one or more read-only memories (ROMs), or a combination thereof. The memory circuit 2130B is configured to store a computer-implemented method, including a learning-from-demonstration (LfD) module and a graph search-based planning module capable of generating executable plans for novel tasks by generating executable sequences of LfD skills (using the LfD module). The processor 2120 may be one or more processor units, and the memory circuit 2130B may be a memory device, a data storage device, or the like. The interface controller (robot interface controller) 2110B may be an interface circuit that may include analog-to-digital (A / D) and digital-to-analog (D / A) converters for signal / data communication between sensors 2101, including force sensors and vision sensors 2102, and the motion controller 2150B of the robot 200. Additionally, the interface controller 2110B may include memory for storing data used by the A / D converter or D / A converter. The sensors 2101 are located at the joints of the robot (robot arm (multiple robot arms) or manipulator) or in the object pick mechanism (e.g., fingers, end effectors) to measure contact states with the robot. The vision sensor 2102 may be located at any position that provides a vantage point for observing / measuring object states that represent the positional relationships between the object, the tabletop, and additional contact surfaces.

[0035] The controller 205 includes an actuator controller (device / circuit) 2150B including a policy unit 2151B, which generates action parameters for controlling the robot 200, which controls the manipulator 210, the handling mechanism, or a combination of the arms 2103 including the handling mechanisms 2103-1, 2103-2, 2103-3, and 2103-#N according to the number of joints or handling fingers. For example, the sensor 2101 may include an acceleration sensor, an angle sensor, a force sensor, or a tactile sensor for measuring the position of an object as well as external forces. For example, the interaction between an object and a robot arm of the robot system can be represented using a complementarity constraint to capture the contact state between the object and the robot arm of the robot system. In other words, the interaction is based on the contact state represented by the relationship between the sliding velocity of the object on the tabletop and the friction between the tabletop and the object when the object is moved by the robot arm.

[0036] The interface controller 2110B is also connected to a sensor 2101 mounted on the robot that measures / acquires the state of the robot's motion. The motion sensor 2101 may be configured to measure a sequence of forces applied to the position on the robot where the sensor is placed. This position is represented by the world coordinate frame 1010 in Figure 10.

[0037] In some cases, if the actuators are electric motors, the actuator controller 2150B may control individual electric motors that drive the angle of the robot arm or the handling of an object by the handling mechanism. In some cases, the actuator controller 2150B may control the rotation of individual motors located on the arm to smoothly accelerate or safely decelerate the robot's movement in accordance with policy parameters generated from a computer-implemented method 2000 for learning a sequence of robotic tasks stored in a memory circuit 2130B, the memory circuit including a learning module 2101B for LfD and a graph search-based planning module 2140B for control signals. Furthermore, depending on the design of the object handling mechanism, the actuator controller 2150B may control the length of the actuator in accordance with the policy parameters according to instructions generated by the computer-implemented method 2000 stored in the memory circuit 2130B.

[0038] The controller 205 is connected to an imager or vision sensor 2102 that provides an RGBD image. In another embodiment, the vision sensor 2102 may include a depth camera, a thermal camera, an RGB camera, a computer, a scanner, a mobile device, a webcam, or any combination thereof. In some cases, the vision sensor 2102 is also referred to as a vision system. Signals from the vision sensor 2102 are processed and used to classify, recognize, or measure the state of the object 220.

[0039] Note that there are no labels available for the different segments of the demonstration trajectory. The different segments represent different (sub)tasks that must be executed sequentially to successfully complete the entire long-horizon task, which is composed of these short-horizon tasks. Note that each of these subtasks must be implemented robustly so that the entire long-horizon task can be completed. Figure 3 shows the possible sequences of subtasks that can be implemented to complete the task of stacking blocks into a goal configuration.

[0040] For example, as shown in FIG. 3 , a block-stacking task using a user-operated interface device 230 has five subtasks. In this case, the user operates the robot manipulator using the interface device 230 as follows: In a first step 310, the robot grasps object B. In step 320, the robot manipulator places object B next to object A. In the next step, step 330, the robot manipulator pushes object B next to object A so that objects A and B come into contact. Then, in step 340, the robot manipulator grasps object C. Finally, the stacking is completed by placing object C on top of objects A and B (350). Note that this is one possible sequence of actions that can be demonstrated during the learning phase of a robot task. The user can demonstrate any other possible subtask sequence that can be used to successfully complete the task. However, the user / operator must provide the same demonstration multiple times. The initial states of the blocks and robot may be different during these multiple demonstrations.

[0041] The task can be demonstrated directly on the robot using teleoperation, or by moving the robot using a kinesthetic controller 205 configured to move the robot manipulator 210. For robot teleoperation, a human expert can move the robot 210 during a task using one of several possible joystick interfaces. FIG. 2A shows several different joystick interfaces that can be used to move the robot during a demonstration. For example, a human can use either joystick 240 or 250 to control the robot's movement during a demonstration. A human can also demonstrate a trajectory for moving the robot 210 (physically or by collecting data in virtual reality) using a virtual reality controller 260 in a virtual reality setting. These teleoperation interfaces can be used to move the robot 210 to demonstrate a desired task, such as stacking a set of blocks in a specific manner 220. The desired task is sometimes referred to as a planned task. As each task is being performed, data of the performance trajectory for each task is acquired via a sensing system including sensors 2101 and vision sensors 2102 and entered into memory circuit 2130B. Another option is for a human to attempt to move the robot arm 210 directly using kinesthetic modes that may be available on the robot to move the robot 201.

[0042] 4 shows a schematic diagram of a world coordinate frame for measuring the state and state of a block-building task, according to an embodiment of the present invention. For the block-building task shown in FIG. 4, the state 430 of the system is a concatenation of the states of the manipulator 410 and the block 420 measured in a fixed frame 440. The state of the manipulator is calculated by the state x of the end effector. ee Similarly, the states of blocks A, B, C, and D are represented by x A , x B , x C , and x D, which may represent the pose of the block in fixed frame 440.

[0043] Some embodiments of the present disclosure are based on the recognition that in the absence of labels for the demonstrated trajectory, a metric must be designed that can be used to consistently segment / divide the demonstrated trajectory into different subtasks, represented by segmented trajectories. However, to determine the different segments of the demonstrated trajectory, a metric is designed that can be used to segment / divide the demonstrated trajectory. Note that neither the number of segments nor the metric for segmenting the trajectory are known. Therefore, to enable segmentation of the demonstrated trajectory, feature extraction is first performed, and then a metric that uses these features is used to segment the trajectory into different components.

[0044] For feature extraction in the current work, we simply transform the robot pose data and the object pose data in different object reference frames, which can be achieved by applying the correct transformation and transforming the observations of all the data in different frames and using them as features.

[0045] The frame is used to define a coordinate system that the robot can use to measure its own position and to know the position of objects in the robot's working environment. The features are functions of measurements or observations used to train a machine learning model. Some embodiments of the present disclosure are based on the recognition that different demonstration trajectories can be transformed within various frames that can be attached to different objects in the robot's working environment. Feature selection is performed using a user-defined function or cost function that represents the objective of feature selection. In the case of supervised learning, this can be performed using a metric such as maximum classification accuracy. However, in the present disclosure, there are no labels, and feature selection is performed using an unsupervised learning cost function. This may be a convex sum of the number of segments obtained by the features and maximization of a segmentation metric (as described in FIG. 5).

[0046] Figure 5 shows the metrics proposed in this work for segmenting the demonstrated trajectories. Figure 5 shows the metrics for trajectory segmentation at 520. An example of how to use the metrics 520 for trajectory segmentation is shown at 510. Figure 5 also shows different features that help in segmenting the demonstrated trajectories at 511. As shown in metrics 520, we use the maximum values ​​of different features for segmenting / splitting the demonstrated trajectories.

[0047] 5 shows different features being transformed within the reference frames of different blocks in a scene. For example, the Object A coordinate frame represents data in the reference frame of object (or block) A at 310. Similarly, the Object B coordinate frame and the Object C coordinate frame represent the reference frames for objects B and C. Measurements can be made directly in these frames or can be transformed after collection using a transformation between the global reference frame and the individual object reference frames.

[0048] Once the demonstration trajectory is segmented into different parts (primitive trajectories correspond to dynamic motion primitives) using the metrics shown in 520, a representative motion model is fitted on each of the segmented trajectories.

[0049] In this disclosure, we use dynamic movement primitives or DMPs to represent each of the segmented trajectories. Figure 6 shows a schematic diagram of the dynamic movement primitives (DMPs) used in the proposed work for learning different skill representations, according to an embodiment of the present invention. Again, we describe these here for completeness. DMPs are a set of two dynamical systems described by ordinary differential equations - point attractor dynamics & forcing terms.

[0050]

number

[0051]

number

[0052] Using the segmentation of the trajectory into individual components and fitting each of the individual segments, any expert demonstration of a task can be reproduced. However, if the desired task differs from the demonstrated task, the described method is insufficient for performing this task. Figure 7 shows a scenario in which an interface device 701 is used to demonstrate that a robot performs a task 710. The goal 711 of this task is significantly different from the goal 721 of a desired task 720. In these cases, an algorithm is needed that can help sequence the learned subtasks so that the robot can successfully perform the desired task 720.

[0053] Some embodiments of the present disclosure are based on the recognition that a graph search-based planning algorithm can be used to assist in planning for tasks that were not demonstrated to a robot during training. FIG. 8 illustrates a planning graph used in some embodiments of the present invention, where an initial node 802 is the goal state of the task and other nodes are added. In this case, a graph search-based planning approach 801 is introduced that also considers the feasibility of actions to find feasible paths for performing a new task that was not demonstrated during training. In the graph search-based planning approach, the initial node is the goal node of the task. Next, we continue to add edges and nodes from the set of existing nodes and feasible actions from all such nodes to this graph. For example, at goal node 803, the robot can only take two feasible actions 804 and 805, which lead to states 806 and 807. Similarly, we add feasible actions from all other nodes and add corresponding edges and vertices to the set. This process ends when the initial state of the system is reached or if no solution is found. Note that the actions available to the robot during the graph construction process are the individual DMPs that the robot learned through segmentation of the demonstration trajectory. In graph exploration-based planning, the robot is simply building a feasible graph and can use the learned DMPs in different sequences to perform novel tasks not seen during the demonstration.

[0054] 9 illustrates the overall method for learning and task execution described in this disclosure. A robot system equipped with an interface for providing demonstrations and collecting demonstration data is used to provide demonstrations of different tasks on the robot (901). A detection system including a motion sensor 2101 and a vision sensor 2102 is used to observe and record the demonstration trajectory as demonstration data (902). During the learning process, a training demonstration task is performed, and the demonstration data of the training demonstration task is collected and stored as collected demonstration data in a dictionary located in memory circuit 2130B. The robot controller uses feature selection (feature selection method) and appropriate metric selection for segmentation (segmentation metric) to segment each demonstration into different segments (903).

[0055]

number

[0056] The robot controller uses the segmented demonstration and fits the DMP to each segment to create a dictionary of feasible skills (trajectories) (904). The robot controller generates a planning graph for the new task using the known goal states of the task and adds nodes to the graph based on the feasibility of performing the task from the current state of the task and the dictionary of skills (905). The robot uses the learned DMP to execute the new task using the planning graph, which transitions between the graph nodes (906).

[0057] The method proposed in this disclosure can be used to perform multiple tasks, such as assembly, consisting of multiple steps that must be performed in a specific order. FIG. 10 illustrates a peg insertion task that can be performed using the proposed approach, which provides a demonstration of sequencing different components (segmented trajectories) and then fitting a DMP to each of the components. A possible motion sequence using a robot (robot end effector) is shown in FIG. 10; in this case, the robot would need to align the peg in the XY 1001 plane and then align the peg with a specific axis (e.g., X 1002). The robot can then insert the peg (1003), and the robot retracts the robot end effector at 1004. Note that all of these demonstrations can be recorded at the appropriate frame 1010. The proposed technique may be used to build programming-free systems for performing complex tasks using robotic systems.

[0058] According to one embodiment of the present invention, the method for learning and task execution is performed by a simulation computer system 2500. The simulation computer system 2500 is configured to create a simulated environment corresponding to the physical environment of the robotic system 200, collect demonstration data generated by moving the robot within the simulated environment using an interface device, including a joystick or a virtual reality or augmented reality interface, to accomplish the task / training. Once the simulation computer system 2500 has collected the demonstration data and / or learning data, the data is transferred to the controller 205 of the robotic system 200 via the communication network 215. The robotic system 200 is configured to use the data to perform a desired / planned task or to use the manipulators of the robotic system 200 to improve the performance of the robotic system's 200's manipulations and to perform further training using real parts.

[0059] The above-described embodiments of the present invention may be implemented in any of numerous ways. For example, embodiments may be implemented using hardware, software, or a combination thereof. If implemented in software, the software code may be executed on any suitable processor or collection of processors, whether located on a single computer or distributed across multiple computers. Such a processor may be implemented as an integrated circuit, with one or more processors being components of the integrated circuit. However, a processor may be implemented using circuitry in any suitable format.

[0060] Also, embodiments of the present invention may be implemented as a method, examples of which are provided. The order of operations performed as part of the method may be arranged in any suitable manner. Thus, embodiments may be configured to perform operations in an order different from that illustrated, which may include performing some operations simultaneously, even though in the illustrated embodiment they are shown as a sequence of operations.

[0061] In the claims, ordinal terms such as "first" and "second" modifying a claim element do not in themselves imply any priority, precedence, or order of a claim element relative to another element, or any chronological order in which the actions of a method should be performed, but are merely used as labels to distinguish claim elements (when no ordinal term is used) from other elements of the same name.

[0062] Although the invention has been described with examples of preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the invention.

[0063] Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.

Claims

1. 1. A robot controller for generating a sequence of movement primitives for a sequential task of a robot having a manipulator, the robot controller comprising: at least one control processor; and a memory circuit having stored thereon instructions, the memory circuit storing a dictionary including the movement primitives, a pre-trained learning module, and a graph search based planning module, the instructions, when executed by the at least control processor, causing the robot controller to: executing a step of acquiring, via an interface controller, performance data of one or more performance tasks provided by an interface device operated by a user for a planned task, the planned task being represented by an initial state and a goal state for at least one object to be operated; each of the performance data is segmented into a plurality of segments by selecting features from the performance data based on a feature selection method and using a segmentation metric, each of the plurality of segments representing a subtask; and generating a planning graph for the planned task by searching feasible paths for the at least one object using the graph search-based planning module and selecting movement primitives from the dictionary in the pre-trained learning module, the pre-trained learning module being trained based on collected performance data of a training performance task; and parameterizing the feasible paths represented by the movement primitives as dynamic movement primitives (DMPs) using the initial state and goal state; and for the planned task, realizing the parameterized feasible path as a trajectory by tracking and following the parameterized feasible path using the manipulator of the robot according to the selected movement primitives.

2. The robot controller of claim 1 , wherein the planned task is not included in the dictionary.

3. The robot controller of claim 1 , wherein the dictionary is updated by adding the parameterized feasible paths according to the selected movement primitives.

4. The robotic controller of claim 1 , wherein the features are detected based on a metric that provides maximum separation between the plurality of segments for the one or more performance tasks.

5. The robot controller of claim 1 , wherein the performance data of the one or more performance tasks is segmented using the features detected by the feature selection method and the segmentation metric method.

6. The robot controller of claim 1 , wherein each of the DMPs is trained for each of the segments of the demonstration task and parameterized for the goal state and the initial state of the planned task.

7. 2. The robot controller of claim 1, wherein the planning graph for the planned task is created based on a planned goal state of the planned task, and state transitions from the planned goal state of the planned task are generated based on the feasible paths.

8. The robot controller of claim 1 , wherein the DMP is generated by segmenting a trajectory of the demonstration task and detecting features of the segmented motion.

9. The robot controller of claim 1 , wherein the trajectories of the performance task are segmented using a metric that represents variance between different performances and within the same trajectory.

10. The robot controller of claim 1 , wherein each of the DMPs is stored as a skill representation of a task, and the dictionary is updated by storing each of the DMPs for all of the segments inferred from the demonstration task.

11. The robot controller of claim 1 , further comprising generating a control policy for a new task using the planning graph for the task and fitting DMPs between different nodes of the planning graph from the dictionary of skills.

12. 2. The robot controller of claim 1, wherein the robot controller is connected to a simulation computer system, the simulation computer system configured to generate a simulation environment corresponding to a physical environment of the robot for virtually performing a predetermined task, the robot controller collects demonstration data, training data, or a combination of the demonstration data and the training data from the simulation computer system, and the demonstration data and the training data are generated by the simulation computer system while performing the predetermined task.

13. 1. A computer-implemented method for learning a sequence of movement primitives for sequential tasks of a robot including a manipulator, the method comprising: a robot controller including at least one control processor; and a memory circuit having instructions stored thereon, the memory circuit storing a dictionary including the movement primitives and a learning module, the instructions, when executed by the at least one control processor, causing the at least one control processor to: collecting performance data from trajectories acquired via a motion sensor, the motion sensor being configured to measure trajectories of objects while the objects are manipulated by an interface device operated by a user according to performance tasks, each of the trajectories corresponding to each of the performance tasks, each of the performance tasks being represented by an initial state and a goal state for each of the objects, the collecting continuing until the user stops the performance tasks; and for each of the demonstration tasks, segmenting the demonstration data into movement primitives by dividing the trajectory into primitive trajectories; and updating the dictionary using the movement primitives based on the collected performance data.

Images