Systems and methods for controlling robotic manipulators

Abstract

A controller for controlling a robot manipulator according to a task is provided. The controller collects data regarding object states and object properties and executes a state adapter model to create state corrections for states of objects having object properties that differ from unitary properties of a unitary object. The controller executes a control policy using the state corrections to create actions for the unitary object and executes a behavior adapter model to create action corrections for the actions created by the control policy. The state corrections and action corrections are created based on differences between the object properties and the unitary properties. The control policy maps the state of the unitary object to actions of the robot manipulator for manipulating the unitary object according to the task.

Description

[Technical Field]

[0001] The present disclosure relates generally to controlling robotic manipulators, and more particularly to controlling robotic manipulators to manipulate objects according to tasks. [Background technology]

[0002] A robotic manipulator is an electronically controlled mechanism that performs actions by interacting with its environment. In one example, a robotic manipulator may perform a tabletop manipulation, such as a pivoting motion. In certain cases, this tabletop manipulation may be performed to reorient or manipulate an object to create or fulfill a prerequisite for another manipulation task.

[0003] In one example, a manipulation action may need to be performed on a tilted or lying cup. In this case, a manipulation action may be performed to reorient the cup to a vertically upright position for packing. In such a case, the vertically upright orientation of the cup may be a prerequisite for another manipulation task, such as packing, so that the robot responsible for packing the cup can properly grasp the cup. In another example, a book on a table that is oriented in the wrong direction may be too large for a robot to grasp for packing and must be manipulated to the required orientation. In yet another example, a peg may be placed in the wrong orientation that a robot would not be able to grasp for an insertion task and must be manipulated to the desired orientation. In such cases, manipulating and reorienting the book or peg can ensure a smooth subsequent operation, i.e., a grasping task for packing or insertion, by creating or satisfying a prerequisite for successfully grasping the book or peg.

[0004] However, a significant difficulty in performing manipulation operations is that the robotic manipulator must maintain contact between its gripper and a surface, such as a table, on which the object to be manipulated may rest. Also, different kinematic and inertial characteristics of different objects entail different complexities, such as unstable, sliding, and rolling characteristics.

[0005] Typically, contact-based manipulation can be performed using either model-based or model-free methods. In particular, model-based methods build open-loop and feedback control strategies by modeling contact dynamics. However, the contact dynamics are hybrid due to the presence of different contact modes. In certain cases, hybrid contact dynamics may be modeled as complementarity constraints or directly formulated to obtain optimal control laws for controlling robot manipulators. In this regard, a hierarchical framework may be utilized to solve discrete contact modes and robot motion separately, reducing computational time. However, model-based methods require strong assumptions on initial conditions and assumptions about contact pairs. Furthermore, model-based methods require accurate system identification to obtain the physical attributes of the manipulated object. Therefore, such model-based methods may not be used for generalized manipulation of objects, such as reorienting different types of objects.

[0006] Continuing further, model-free approaches omit contact dynamics modeling and directly learn policies for performing manipulation tasks, such as reorientation, packing, and insertion. In certain cases, reinforcement learning (RL) methods may be used to learn manipulation skills by optimizing a specified reward. However, in model-free approaches, RL policies for performing manipulation tasks on objects are learned on a single object during training. Such RL policies may only be applicable to objects of similar size and / or design to the object on which the RL policy was trained. This limits the applicability of the trained RL policy to a specific manipulation task or to a specific type of object with similar or identical physical attributes.

[0007] Generalization is a major challenge for skill acquisition for robot manipulators, making the robot robust to changes in the environment and task context. Different objects manipulated may have different geometries, physical attributes, and contact dynamics. Each of these differences alters the way a robot manipulator interacts with these different objects, making skill generalization, especially manipulation skills, challenging. To achieve generalization, a robot manipulator must adjust its manipulation skills according to different tasks and different objects. In this regard, it must learn robust manipulation skills that can work for different task settings or that can quickly adapt to new tasks. In certain cases, domain randomization can be applied to force a learned manipulation policy to extract useful information from the state, making it robust to task changes. For example, Model Agnostic Meta Learning (MAML) may learn a set of policy parameters that can adapt to a new task in just a few tries. In certain other cases, task information may be encoded in the manipulation policy. Thus, the robot's behavior generated by the learned operational policy is tailored to different tasks.

[0008] However, for different tasks, the state space and action space may vary according to the corresponding task settings (e.g., robot type, object shape, object design, object size, object weight, goal state, etc.). Therefore, the analysis of the task settings needs to be transferred to the learned operation policy to perform different tasks on different types of objects. However, the mapping between the state space and the action space is either obtained from a manual source that can analyze the differences between the spaces or from a point cloud registration algorithm. This requires utilizing human prior knowledge to find the correct mapping between the state space and the action space. Furthermore, such mapping is prone to errors due to human bias and human fatigue during the generation of such datasets. Summary of the Invention

[0009] Reinforcement learning (RL) is a branch of machine learning. RL considers taking appropriate actions in an environment to maximize the concept of cumulative reward. RL is one of three basic machine learning paradigms, along with supervised learning and unsupervised learning. RL may be used to identify the optimal behavior or path to complete a particular task.

[0010] Some embodiments are based on the recognition that RL does not require labeled input / output pairs and explicitly calibrated actions for training. RL focuses on finding a balance between exploring unknown territory and leveraging current knowledge to generate output actions.

[0011] Some embodiments recognize that RL is advantageous for robotic manipulation of different objects because labeled data for manipulation tasks is rarely available. For example, RL may be used to learn to map object states to robotic manipulator actions for manipulating the object according to a task. Such learning may vary depending on the shape of the object and the details of the task. Tasks may be, for example, tilting, picking, rotating, inserting, etc. Therefore, a learned mapping by RL for an object with a specific shape to be manipulated according to a specific task may not be suitable for manipulating objects of different shapes and / or different tasks.

[0012] An objective of some embodiments is to adapt or extend reinforcement learning to learn generalization policies for robotic manipulation of different objects of different shapes according to the task.

[0013] Some embodiments are based on the understanding that tabletop manipulation operations must be performed to manipulate or reorient an object, creating prerequisites for other manipulation operations. A manipulation setup may include an object, where a robotic manipulator may manipulate the object using interactions with the object and / or surfaces surrounding the object.

[0014] Some embodiments are based on the understanding that a significant challenge in performing a manipulation operation is that the robotic manipulator must maintain contact with the object and with the surface surrounding or on which the object rests.

[0015] Some embodiments are based on the understanding that another difficulty in performing manipulation operations is that different kinematic and inertial properties of different objects require different complexities such as unstable, sliding, and rolling properties.

[0016] Some embodiments are based on the understanding that performing manipulation actions to reorient an object with a robotic manipulator is challenging due to external forces acting on the robotic manipulator due to contact interactions. The complexity of the manipulation actions increases when the same policy needs to be applied to different object shapes, object sizes, and / or object designs.

[0017] Some embodiments are based on the understanding that contact dynamics for performing manipulation operations are hybrid due to different contact modes during contact. In particular, contact-heavy approaches for performing manipulation operations would require strong assumptions on initial conditions or contact pairs of manipulations and identification to capture object properties and / or surfaces of the object. As a result, such contact-heavy approaches may not be generalizable and may not be usable for different objects and / or different tasks.

[0018] Some embodiments are based on the understanding that generalization of policies for performing manipulation actions for different objects is difficult due to different objects having different geometries, different object properties, and different contact dynamics. The differences between different objects make generalization difficult because the robot manipulator may vary the way it interacts with different objects depending on the task. Furthermore, for different tasks, the state and action spaces may change depending on the task setting, requiring manual analysis of the space differences.

[0019] Therefore, it is an object of some embodiments to provide such a method and system for learning RL manipulation policies for performing manipulation actions on real-world objects in a structured environment with zero-shot transfer learning from simulation to the real world. It is an object of some embodiments to generalize the learned manipulation policies to perform manipulation actions according to the same or different tasks so that they can be applied to any unseen object. It is an object of some embodiments to provide a system and method that can automatically discover an underlying mapping over a state space or an action space by optimizing trajectory returns for performing manipulation actions on different objects.

[0020] It is an objective of some embodiments to disclose a framework for learning robust and generalizable manipulation policies for manipulating real-world objects solely from simulation experience.

[0021] Some embodiments are based on the understanding that in the context of robotic manipulation, behavior predominates over physical properties such as the shape or size of an object. Therefore, the physical properties of an object can be used as constraints to create behaviors for manipulating the object. It is an objective of some embodiments to disclose a framework for learning manipulation policies that utilize properties such as the physical properties of an object as constraints.

[0022] Some embodiments are based on the understanding that using physical properties as constraints allows the physical properties of an object to be expressed as physical properties associated with a predefined unitary object corrected with a correction value. This correction depends on the difference between the physical properties associated with the predefined unitary object and the physical properties of the object. Hence, the correction may be applied to different objects based on an RL manipulation policy learned for the predefined unitary object.

[0023] An objective of some embodiments is to disclose a framework for learning generalizable manipulation policies for unitary objects using RL. RL is used to learn a manipulation policy to perform a manipulation, such as reorienting or rotating, one specific object, referred to as a unitary object. The manipulation policy is then applied to different objects, such as objects that are not unitary objects. Supervised learning may be used to extract low-dimensional object information about objects from high-dimensional object depth images. The manipulation policy learns an object feature space by applying supervised learning to a dataset collected in a simulation to predict object size and class by encoding the kinematic properties of any object.

[0024] The framework then adapts the manipulation policy learned on a unitary object to multiple objects by learning data-driven projections based on the object properties of different objects and adjusting the state and action spaces for new manipulation tasks.By adjusting the state and action spaces, object-specific state and action projections are learned and the manipulation policy is adapted to new arbitrary objects.

[0025] Some embodiments are based on the understanding that since RL can learn policies for mapping the states of unitary objects to the actions of a robotic manipulator, embodiments can extend RL by adapting the inputs and outputs, states and actions, respectively, of such manipulation policies to manipulate different objects with different object properties.

[0026] Some embodiments are based on the understanding that compensation based on differences between the physical properties of unitary objects (hereinafter referred to as unitary properties) and the physical properties of real objects (hereinafter referred to as object properties) may fall outside the scope of RL learned for unitary objects. The basic concept of some embodiments is that manipulation policies for manipulating different objects can be computed as transformations of the manipulation policies learned with unitary objects, rather than being learned from scratch each time.

[0027] An objective of some embodiments is to train a framework for learning a generalizable turning policy for manipulating an object in a simulation by turning it against one or more contact surfaces. The framework requires only one depth image of a unitary object and / or multiple different objects to learn the turning policy, and provides zero-shot transfer of this learning to real-world turning tasks for different objects. The trained turning policy is robust to learning transfer from simulation to the real world and generalization to multiple objects.

[0028] Accordingly, one embodiment discloses a controller for controlling a robotic manipulator to manipulate an object according to a task. The controller comprises a memory configured to store a control policy configured to generate behaviors for the robotic manipulator to manipulate a unitary object having a predetermined unitary property. The control policy is configured to map a state of the unitary object to a behavior of the robotic manipulator to manipulate the unitary object according to the task. The memory is further configured to store a configured state adapter model for generating state corrections for a state of an object having object properties different from the unitary property based on the difference between the object property and the unitary property. The memory is further configured to store a behavior adapter model for generating behavior corrections for the behavior generated by the control policy based on the difference between the unitary property and the object property. The controller includes a processor configured to collect data regarding object states and object properties, execute a state adapter model, create state corrections to the object states based on the collected data to generate corrected states for the object, execute a control policy using the corrected states of the object, create behaviors for the unitary object, execute a behavior adapter model, create behavior corrections to the behaviors for the unitary object to generate corrected behaviors, and control a robotic manipulator with the corrected behaviors for the unitary object.

[0029] The behavioral adapter model is configured to create behavioral corrections to behaviors for unitary objects based on the correction state of the object and property encodings of object properties for the object.

[0030] The processor is further configured to process object properties of the object to create a property encoding by executing the encoder, send the property encoding to the state adapter model and the behavior adapter model, and create a corrective behavior based on the property encoding by executing the control policy, the state adapter model, and the behavior adapter model.

[0031] The object properties relate to at least one of shape, size, density, weight, or material.

[0032] If the object property and the unitary property are associated with the shape of the object and the unitary object, respectively, the processor is further configured to use the state adapter model to create state corrections for the state of an object having an object shape different from the unitary shape based on the difference between the object shape and the unitary shape, and to use the behavior adapter model to create behavior corrections for the behavior created by the control policy based on the difference between the unitary shape and the object shape.

[0033] To train the control policy, the processor is further configured to generate a dataset consisting of a set of random objects, the set of random objects having corresponding random shapes that differ from a unitary object shape of the unitary object, the processor is further configured to learn a feature space for the random objects based on the difference between each of the random object shapes and the unitary shape, and determine a predicted object shape for each set of random objects based on the learned feature space.

[0034] Object properties are captured using one or more sensors. The corrective behavior consists of behavioral parameters for controlling the interaction between the robot manipulator and the object. The control policy consists of a neural network trained with reinforcement learning. The state adapter model and the action adapter model each consist of a neural network, and the state adapter model and the action adapter model are trained with machine learning.

[0035] To train the controller, the processor is further configured to: cause the state adapter model to learn one or more linear transformations to create state corrections to the states of the unitary object for the composite object; cause the behavior adapter model to learn one or more linear transformations to create behavior corrections to the behaviors for the unitary object based on the corrected states; and adapt the control policy to the composite object based on the learning.

[0036] The processor is further configured to generate a simulated environment for the task based on the unitary property and to create a behavioral correction based on the simulated environment.

[0037] Another embodiment discloses a method for controlling a robotic manipulator to manipulate an object according to a task using a controller having a memory configured to store a control policy, a state adapter model, and a behavior adapter model. The control policy is configured to generate behaviors for the robotic manipulator to manipulate a unitary object having a predetermined unitary property, such that the control policy is configured to map states of the unitary object to behaviors of the robotic manipulator to manipulate the unitary object according to the task. The state adapter model is configured to generate state corrections for states of objects having object properties different from the unitary property based on the difference between the object property and the unitary property. The behavior adapter model is configured to generate behavior corrections for actions generated by the control policy based on the difference between the object property and the unitary property. The method includes collecting data regarding a state and object properties of the object; using a state adapter model to create a state correction for the state of the object based on a difference between the object property and a predetermined unitary property of the unitary object to generate a corrected state of the object; using a control policy to create a behavior for the unitary object using the corrected state of the object; using a behavior adapter model to create a behavior correction for the behavior for the unitary object to generate a corrected behavior; and controlling a robotic manipulator with the corrected behavior for the unitary object.

[0038] Yet another embodiment discloses a non-transitory computer-readable storage medium embodied with a program executable by a processor for executing a method for generating a robotic manipulator to manipulate an object according to a task using a controller having a memory configured to store a control policy, a state adapter model, and a behavior adapter model. The control policy is configured to generate behaviors for the robotic manipulator to manipulate a unitary object having a predetermined unitary property, and the control policy is configured to map states of the unitary object to behaviors of the robotic manipulator to manipulate the unitary object according to the task. The state adapter model is configured to generate state corrections for states of objects having object properties different from the unitary property based on differences between the object properties and the unitary property. The behavior adapter model is configured to generate behavior corrections for actions generated by the control policy based on differences between the unitary property and the object property. The method includes collecting data regarding a state and object properties of the object; using a state adapter model to make state corrections to the state of the object based on the collected data to generate a corrected state; using a control policy to make a behavior for the unitary object using the corrected state of the object; using a behavior adapter model to make behavior corrections to the behavior for the unitary object to generate a corrected behavior; and controlling a robotic manipulator with the corrected behavior for the unitary object.

[0039] The presently disclosed embodiments will be further described with reference to the accompanying drawings, in which the drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the presently disclosed embodiments. [Brief explanation of the drawings]

[0040] [Figure 1A]FIG. 1 illustrates a robotic manipulator for manipulating an object, according to some embodiments of the present disclosure. [Figure 1B] 1A-1C illustrate exemplary manipulation actions according to an embodiment of the present disclosure. [Figure 1C] FIG. 1 is a block diagram of a controller for controlling a robotic manipulator according to an embodiment of the present disclosure. [Figure 2] FIG. 1 illustrates an exemplary method for training a control policy for manipulating a unitary object, according to an embodiment of the present disclosure. [Figure 3A] FIG. 1 illustrates an exemplary method for training a control policy to adapt to new objects, according to an embodiment of the present disclosure. [Figure 3B] FIG. 1 illustrates an exemplary method for training a control policy to adapt to different synthetic objects, according to an embodiment of the present disclosure. [Figure 3C] FIG. 1 illustrates an exemplary method for training state and behavioral adapter models to generalize control policies, according to an embodiment of the present disclosure. [Figure 4] 1A-1C are exemplary diagrams of different states of a unitary object, according to various embodiments of the present disclosure. [Figure 5] FIG. 1 is a block diagram for training a control policy according to an embodiment of the present disclosure. [Figure 6] FIG. 1 illustrates an exemplary method for performing a desired manipulation task on a new object, according to an embodiment of the present disclosure. [Figure 7A] 1A-1C illustrate example control trajectories for manipulating an object during a manipulation action, according to various embodiments of the present disclosure. [Figure 7B] 1A-1C illustrate example control trajectories for manipulating an object during a manipulation action, according to various embodiments of the present disclosure. [Figure 7C] 1A-1C illustrate example control trajectories for manipulating an object during a manipulation action, according to various embodiments of the present disclosure. [Figure 7D] 1A-1C illustrate example control trajectories for manipulating an object during a manipulation action, according to various embodiments of the present disclosure. [Figure 8] FIG. 1 illustrates an exemplary method for performing a desired manipulation task on a new real-world object, according to an embodiment of the present disclosure. [Figure 9A] FIG. 1 illustrates an exemplary method for generating corrective behavior for an object, according to an embodiment of the present disclosure. [Figure 9B] FIG. 1 illustrates an exemplary method for generating corrective behavior for an object, according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0041] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other instances, devices and methods are shown only in block diagram form in order to avoid obscuring the disclosure. Various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the subject matter as set forth in the appended claims.

[0042] As used in this specification and the appended claims, the terms "for example," "for instance," and "such as," and the verbs "comprising," "having," and "including" and other verb forms thereof, when used in conjunction with a list of one or more components or other items, should each be construed as open-ended, meaning that the list should not be considered to exclude other, additional components or other items. The term "based on" means based, at least in part, on. Furthermore, it should be understood that the phraseology and terminology used herein are for descriptive purposes and should not be regarded as limiting. The headings used herein are for convenience only and do not have any legal or limiting effect.

[0043] Specific details have been set forth in the above description to provide a thorough understanding of the embodiments. However, those skilled in the art will understand that the embodiments may be practiced without these specific details. For example, systems, processes, and other elements of the disclosed subject matter may be illustrated in block diagrams as components to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known processes, structures, and techniques may be illustrated without unnecessary detail to avoid obscuring the embodiments. Furthermore, like reference numerals and symbols in the various drawings indicate like elements.

[0044] An objective of some embodiments is to disclose a controller for controlling a robotic manipulator to manipulate an object according to a task. An objective of some embodiments is to disclose a RL-trained control policy for generating behaviors for a robotic manipulator to manipulate a unitary object of a given unitary shape. An objective of some embodiments is to disclose a technique for generalizing a control policy to manipulate unseen objects without training in a real-world environment. An objective of some embodiments is to disclose a technique for domain randomization. Domain randomization leads to robustness to measurement errors in the state of an object, e.g., object position, force measurements, initial pose distribution of the environment, and failure recovery. An objective of some embodiments is to disclose a control policy that recovers from complete failures, such as losing contact with an object during manipulation. According to some embodiments, control policy generalization allows a control policy to be trained only in simulation and generalizes to novel objects of various shapes in the real world, reducing real-world training time to zero. An objective of some embodiments is to disclose techniques for various tabletop manipulation actions, such as, for example, changing the orientation of an object to a different orientation, pushing an object to a different location, grasping an object in a different position or manner, sliding, tilting, rolling, throwing, spinning, or other types of manipulation of the state of an object.

[0045] FIG. 1A illustrates an environment 100 in which a robotic manipulator 102 is implemented, according to some embodiments. The robotic manipulator is configured to manipulate an object. The robotic manipulator 102 includes a movement device 104 for providing movement to the robotic manipulator 102. In one example, the movement device 104 may include one or more motors for providing the movement. For example, the one or more motors may include, but are not limited to, an electric motor, a hydraulic motor, or a pneumatic motor. The movement device 104 may provide a force for movement or movement of a robotic arm 104a of the robotic manipulator 102. For example, the robotic arm 104a may have one or more degrees of freedom for moving an object or the robotic manipulator 102. In one example, the robotic arm 104a includes an end tool 106 for providing multiple degrees of freedom for moving an object. In some implementations, the end tool 106 includes a gripper 108 for manipulating or holding an object.

[0046] 1B, an exemplary manipulation action 120 is shown, according to an embodiment. For example, the purpose of the manipulation action 120 is to change the state of the object 122, such as changing the orientation of the object 122 by changing its pose.

[0047] Robotic manipulation tasks are performed by the robotic arm 104a. The robotic arm 104a may be configured to perform manipulation operations, for example, to reorient the object 122 with the environment. Other examples of robotic manipulation tasks may include, but are not limited to, inserting an object into a connector known as a slotted peg, pick-and-place operations, product assembly, etc.

[0048] 1B , an object 122 is at rest on a flat surface, such as a tabletop 126, and is proximate to a second surface, i.e., a contact surface 124, that is orthogonal to the tabletop 126. The object 122 may be manipulated by, for example, an end tool 106 of a robotic arm 104 a, which may establish patch contact with the object 122 between the fingers of a gripper 108 and the object 122. The gripper 108 may reorient or manipulate the object 122 by pressing the object 122 against the contact surface 124 on the tabletop 126 in a predetermined environment and using a control policy according to an embodiment of the present disclosure.

[0049] In one example, the control policy can be expressed as π(·|s), where s is a system state. The control policy can generate control commands to control the movement of the gripper. In this regard, to perform a manipulation operation, the control policy may cause the gripper 108 to contact the object 122 and subsequently establish contact between the object 122 and the contact surface 124. The control policy may utilize environmental contacts, such as the gripper 108's contact with the object 122, the object 122's contact with the tabletop 126, and the object 122's contact with the contact surface 124, to manipulate and reorient the object 122 to a standing or upright position. In one example, the control policy may cause the gripper 108 to rotate the object 122 while maintaining contact between the object 122 and the contact surface 124, the object 122's contact with the tabletop 126, and the gripper 108's contact with the object 122. In one example, the control policy may include a neural network trained using reinforcement learning techniques. In particular, the control policy is trained to generalize across different types of objects using state and action adapter models, such that the control policy is capable of zero-shot transfer of simulated learning learned on unitary objects with predefined unitary properties to unseen real-world objects. The term "control policy" is used interchangeably with "reinforcement learning (RL) policy" and "manipulation policy."

[0050] In particular, a unitary object refers to any object, such as a random object, that has defined properties. In one example, a unitary object may have a defined shape, a defined size or dimensions, defined material properties, a defined weight, and a defined density. That is, the physical parameters of a unitary object may be known and defined. In one example, a unitary object may be a cube with each side measuring 10 centimeters. Furthermore, the cube may weigh 10 grams and have a density of 0.5 g / cm. 3The unitary object may be made of a hard plastic such as ABS. These examples of unitary objects are illustrative only and should not be construed as limiting.

[0051] To this end, the illustration of the robotic manipulator 102 and the manipulation task of orienting and rotating the object 122 using the contact surface 124 and the tabletop 126 should not be construed as limiting. In other embodiments of the present disclosure, the robotic manipulator 102 may include other components, such as a body, one or more sensors, one or more end effectors, one or more feedback devices, one or more manipulators, etc. In other embodiments of the present disclosure, the manipulation task may be another task, such as an insertion task, a grasping task, etc.

[0052] 1A , the robotic manipulator 102 includes a force sensor 110 operatively connected to the end tool 106 and / or gripper 108 of the robotic arm 104 a. The robotic manipulator 102 also includes a vision sensor 112. For example, the vision sensor 112 may be a camera, an RGB camera, an RGBD camera, a camcorder, a depth camera, a range camera, a thermal camera, a computer, a scanner, a mobile device, a webcam, etc. For example, measurements or signals from the force sensor 110 and / or the vision sensor 112 may be processed and utilized to classify, recognize, or measure the state and / or object properties of the object 122.

[0053] The force sensor 110 may be configured to measure force tensors experienced by the end tool 106 and / or gripper 108 of the robotic manipulator 102 during a manipulation operation. In one example, the force sensor 110 may be mounted on the gripper 108 of the robotic arm 104a. For example, the force tensors measured by the force sensor 110 include force measurements and moment measurements along the axes of the robotic manipulator 102. For example, the force measurements and moment measurements along the axes of the robotic manipulator 102 may be referred to as Fx, Fy, Fz, Mx, My, and Mz.

[0054] 1B, the force sensor 110 measures the force and torque of the object 122 striking the contact surface 124 and the tabletop 126 during object manipulation. Additionally, the force sensor 110 may include one or more encoders to measure the position and orientation of the object 122 along the three axes of the robot's end tool 106 and the three axes of the robot's joints. In certain cases, the robot manipulator 102 may also include acceleration sensors, angle sensors, and tactile sensors.

[0055] The visual sensor 112 is positioned to observe the state of the object 122 being manipulated. In some embodiments, the object state is a 6D pose, which includes a Cartesian pose and an orientation of the object 122. In some embodiments, the visual sensor 112 is configured to estimate the pose of the object 122 on the tabletop 126 with respect to a contact surface 124 in the environment. To do this, the visual sensor 112 detects and estimates the pose of the object 122 being manipulated. Measurements from the force sensor 110 and data from the visual sensor 112 may be stored as measurements 111. The measurements 111 may be used to manipulate the object 122 using the controller 114.

[0056] In one example, the force sensors 110 are positioned at joints of the robotic manipulator 102, such as the robotic arm(s) 104a, the end tool 106, and / or the gripper 108, to measure forces external to the robotic manipulator 102. The vision sensors 112 may be positioned in any location that provides a vantage point for observing or determining the state of the object 122, which is indicative of the positional relationship between the object 122, the tabletop 126, and the contact surface 124.

[0057] Referring to FIG. 1C , an example block diagram of a controller 114 for controlling the robotic manipulator 102 is shown, according to some embodiments. The controller 114 is coupled to the robotic manipulator 102 for controlling the movement of the robotic manipulator 102, according to embodiments of the present disclosure. In one example, the controller 114 is configured to determine whether the object 122 needs to be manipulated before it can be used for another desired task. For example, the controller 114 is configured to determine whether the pose of the object 122 needs to be manipulated as a prerequisite for performing another task. The other tasks may include, but are not limited to, assembly, packaging, insertion, and the like. The controller 114 is configured to compute and send commands or control actions to be applied to the robotic manipulator 104 a to manipulate the object 122. For example, the controller 114 may be configured to compute the commands or control actions using a control policy, a state adapter model, and a behavior adapter model proposed in embodiments of the present disclosure. The robotic manipulator 102 applies the commands to manipulate the object 122.

[0058] The controller 114 may include an interface controller 116, a processor 117, and a storage unit 118. The storage unit 118 is configured to store a control policy 118a, a unitary property of a unitary object 118b, a state adapter model 118c, and a behavior adapter model 118d. In one example, the processor 117 may include one or more processing units. The storage unit 118 may include, but is not limited to, a storage device, a data storage device, etc.

[0059] Furthermore, the interface controller 116 may be an interface circuit including, for example, an A / D (analog-to-digital) converter and a D / A (digital-to-analog) converter. The interface controller 116 may generate signals or data for communication with the force sensor 110 and / or the vision sensor 112 to obtain the measurements 111 and for communication with the motion controller 119 to control the robotic manipulator 102. The interface controller 116 may include a memory for storing data used by the A / D converter or the D / A converter. In addition, the motion controller 119 includes a policy unit 119a for generating behavior parameters based on behaviors produced by the control policy, and controls the robotic manipulator 102 to control the robotic arm 104a. In one example, the interaction between the object 122 and the robotic arm 104a may be controlled and adjusted by the trained control policy 118a using a state adapter model 118c and a behavior adapter model 118d.

[0060] The controller 114 may be connected to the force sensors 110 and the vision sensors 112 to obtain the measurements 111. In particular, the motion controller 119 may control individual electric motors that drive the angle of the robot arm 104a or the handling of the object 122 by the gripper 108. In one example, the motion controller 119 may control the rotation of individual motors located on the robot arm 104a to smoothly accelerate or safely decelerate the movement of the robot manipulator 102 in response to control signals. In another example, the motion controller 119 may control the actuators of the robot arm 104a in response to control signals generated by the control policy 118a. For example, the control signals may be generated by the control policy 118a based on the unitary property 118b, the state adapter model 118c, the behavior adapter model 118dc, and the measurements 111.

[0061] To this end, the controller 114 may adapt the control policy 118a, trained using reinforcement learning techniques, to robustly manipulate different objects of different shapes according to the task. For example, the controller 114 and the control policy 118a may be a neural network-based reinforcement learning policy. The control policy may be used to change the orientation of the object 122 or parts of the object 122 during the robot's assembly process, packaging process, etc. The state adapter model 118c and the behavior adapter model 118d enable the control policy 118a to adapt or generalize to manipulate different objects with different physical properties according to the task. The manner in which the control policy 118a is trained to perform manipulation actions for manipulating different objects is described in connection with the following figures.

[0062] An objective of some embodiments of the present disclosure is to adapt or extend reinforcement learning to robotics for manipulating different unseen objects with different physical properties based on a control policy 118a trained on a unitary object with predefined unitary properties 118b. Some embodiments are based on the understanding that, at least in the context of robotics, behavior predominates over the physical properties of the object. Therefore, the physical properties of the object can be viewed as constraints for creating behaviors for manipulating the object.

[0063] To this end, a control policy 118a having an RL-based neural network may be used to generate behaviors for the robot manipulator 102 to manipulate unitary objects having predetermined unitary properties 118b. In such cases, the control policy 118a maps the state of the unitary object to the behaviors of the robot manipulator to manipulate the unitary object according to a task. In this manner, the control policy 118a is trained to map the state of a new or novel object to the behaviors of the robot manipulator 102 to manipulate the new object, as will be described in more detail in conjunction with FIG. 2. Furthermore, the control policy 118a must be trained to adapt according to the object. The manner in which the control policy 118a is trained to adapt to a new object and generate manipulation control behaviors for the new object will be described in more detail in conjunction with FIGS. 3A, 3B, 3C, and 5.

[0064] 2 illustrates an example method 200 for training a control policy 118a for manipulating a unitary object, according to some embodiments. In one example, the control policy 118a may be trained with reinforcement learning to generate behaviors for the robotic manipulator 102. Based on the generated behaviors, the robotic manipulator 102 may manipulate the unitary object according to a task.

[0065] As mentioned above, RL is advantageous for robotic manipulation of heterogeneous objects because labeled data for such tasks is rarely available. RL focuses on how an agent should act in the environment to maximize the concept of cumulative reward. For example, RL can be used to learn to map object states to the actions of a robot manipulator to manipulate the object according to a task. Such learning relies on both the object properties of the unseen object and the specifics of the task itself, such as tilting, turning, picking, or inserting. The method 200 is based on using RL to train a control policy 118a for manipulating a unitary object.

[0066] At 202, unitary properties 118b of the unitary object are obtained. For example, the unitary object may have a defined physical structure. To this end, the unitary properties 118b may correspond to the defined physical structure of the unitary object. Furthermore, the unitary properties 118b may indicate, for example, at least one of the shape, size, dimensions, density, material, and weight of the unitary object.

[0067] In one example, the unitary properties 118b may be determined based on the use of one or more sensors. For example, if the sensor is a camera, a depth image of the unitary object may be captured. For example, such depth image may be captured by a vision sensor 112, where the vision sensor 112 is a depth camera. Based on the depth image of the unitary object, the unitary properties 118b corresponding to the physical structure of the unitary object may be determined. Also, in certain cases, the depth image of the unitary object may be used to identify the state of the unitary object. For example, the state of the unitary object may indicate, but is not limited to, the pose of the unitary object, the position of the unitary object, the orientation of the unitary object, etc. In another example, the unitary properties 118b and the state of the unitary object may be provided as inputs for training the control policy 118a.

[0068] At 204, a control policy 118a for manipulating the unitary object is trained. In particular, reinforcement learning is used to train the neural network comprising the control policy 118a. In this regard, the RL control policy 118a may identify and execute actions for manipulating the unitary object. For example, the trained control policy 118a may include a state of the unitary object mapped to a robot manipulator's actions for manipulating the unitary object according to a task. In one example, the task may be performed on the unitary object. Tasks may include, but are not limited to, assembly, manufacturing, packaging, quality checks, and the like. For example, a task may have a set of predefined steps, each of which may have a corresponding predefined input and a predefined output. To this end, the predefined input and the predefined output of a step of a task may indicate a desired state of the unitary object. For example, the input of a step may indicate a desired state of the unitary object for performing the step, and the output may indicate a desired state of the unitary object after an action associated with the task has been performed. Based on the desired state of the unitary object for performing the step and the current state of the unitary object, a manipulation action may need to be performed. For example, a manipulation action may be performed to manipulate the unitary object such that the current state of the unitary object changes to a desired state for performing the step of the task. In this regard, the current state of the unitary object may be mapped to an action to be performed on the unitary object based on the desired state of the unitary object. In one example, the mapping of states to actions may be performed in an embedded latent space.

[0069] For example, the control policy 118a may be trained using RL to generate behavior parameters that cause the robotic manipulator 102 to cause the robotic arm 104a or gripper 108 to manipulate the unitary object according to a desired behavior. For example, the control policy 118a may learn to evaluate the unitary properties 118b, identify a desired behavior, and generate behavior parameters based on the unitary properties 118b and the desired behavior to manipulate the unitary object from a current state to a desired state.

[0070] In one example, the task related to a unitary object is a packing task. For example, for a packing task, the defined steps may include, but are not limited to, picking the unitary object, placing the unitary object on a conveyor belt, checking the parameters of the unitary object according to a quality check, picking the unitary object, and placing the unitary object in a packing box. Furthermore, the unitary object may be positioned on the conveyor belt in a specific manner so that a robotic arm can grasp or pick the unitary object reliably without damaging it. For example, if the unitary object is a cup, the cup may be placed in a vertically upward orientation so that a robotic arm can lift the cup by its handle without breaking or damaging it. In such a case, if the cup's current state is tilted sideways, such as lying on its side on the conveyor belt, a manipulation action may be performed to lift the cup to a vertically upward position and bring the cup to the desired state. To this end, the manipulation action may be performed by applying a turn behavior to the cup to change the cup's orientation to a vertically upward position.

[0071] In this manner, control policy 118a may learn to generate behaviors for manipulating unitary objects according to a task. For example, the generated behaviors may be further used to generate new behaviors for manipulating different objects in a real-world scenario. The manner in which control policy 118a is trained to adapt or transform to generate new behaviors for different objects is described in detail in conjunction with Figures 3A, 3B, and 3C.

[0072] To this end, the control policy 118a includes all the advantages of RL algorithms. Advantages include domain randomization, which leads to robustness against measurement errors on the object's state, such as the object's position, force measurements, and the initial pose distribution of the environment. Advantages also include failure recovery, which means that the control policy 118a will be able to recover from a complete failure, such as losing contact with an object during collaboration or losing the interaction to manipulate the object.

[0073] 3A illustrates an example method 300 for training a control policy 118a to adapt to new objects, according to some embodiments of the present disclosure. For example, the controller 114 may cause the robotic manipulator 102 and / or the robotic arm 104a to perform a desired manipulation task based on control inputs from the control policy 118a. As described above, the control policy 118a may be trained to generate behaviors for a unitary object according to the task.

[0074] Note that manipulation tasks for different objects can be computed as transformations of the control policy 118a learned with unitary objects, rather than learning them from scratch each time. The steps of method 300 describe training the control policy 118a to adapt or change to different objects according to the task.

[0075] In this regard, at 302, a simulated environment for a desired task is created. The simulated environment is created based on unitary objects and corresponding unitary properties 118b. Furthermore, the simulated environment is created to train the control policy 118a to adapt to new objects. For example, the task may be a packing task or an assembly task.

[0076]

number

[0077]

number

[0078] The parameters of the RL framework for training the control policy 118a are mathematically based on a state space, an action space, a reward function, and a transition function. Note that embodiments of the present disclosure reliably describe how to use the control policy 118a to control the robotic manipulator 102 and / or gripper 108 to manipulate unitary objects and learn how to manipulate the robotic manipulator 102 for different objects.

[0079]

number

[0080]

number

[0081]

number

[0082]

number

[0083]

number

[0084]

number

[0085]

number

[0086] At 306, a latent space of unitary properties 118b is learned to represent the unitary properties 118b of the unitary object. For example, the unitary object is represented in a simulated environment based on an embedding of the unitary properties 118b into the latent space. In one example, representation learning may be utilized to encode the unitary properties 118b of the unitary object in the latent space. Some embodiments of the present disclosure rely on representation learning to learn a low-dimensional feature space of the unitary properties 118b based on a depth image of the unitary object. For example, the RGB texture of the depth image may be used to identify the physical characteristics or unitary properties 118b of the unitary object, which may then be represented in the latent space. For example, supervised learning may be used to generate the encoding of the unitary properties 118b.

[0087]

number

[0088] At 308, the state adapter model 118c and the action adapter model 118d are trained to adapt the control policy 118a to the new object by adjusting the state space and action space. In particular, the state adapter model 118c and the action adapter model 118d learn linear transformations to adjust both the state space and the action space of the unitary object and generalize the control policy 118a to the new object. Such linear transformations of the state space and the action space are referred to as state and action projection networks.

[0089] The state adapter model 118c and the behavior adapter model 118d enable the control policy 118a to generalize to multiple different objects for robotic manipulation tasks. The state adapter model 118c and the behavior adapter model 118d also enable the control policy 118a to generalize to represent the physical properties of different objects (hereinafter referred to as object parameters) in a latent feature space. In one example, the state adapter model 118c and the behavior adapter model 118d are each trained using machine learning, particularly supervised machine learning, to adapt the RL control policy 118a. In this way, the state adapter model 118c and the behavior adapter model 118d enable the RL control policy 118a to create new behaviors for new objects based on a simulated environment.

[0090] In one example, an encoder may be utilized to process object parameters of a random new object. The encoder may create a feature encoding of the object parameters in a latent space for the new object. The encoder may then send the encoded object parameters to state adapter model 118c and behavior adapter model 118d. State adapter model 118c and behavior adapter model 118d may use the encoded object parameters to adapt or transform control policy 118a to the new object. Thus, the transformed or adapted control policy 118a may generate a state-to-behavior mapping for the new object according to an associated task. In this manner, the transformed control policy 118a may determine an action for the new object and change the current state of the new object to a desired state according to the task.

[0091] In one example, the overall generalizable RL framework consists of a control policy 118a, an object feature extraction network (i.e., an encoder for encoding unitary properties 118b in latent space), a state adapter model 118c, and an action adapter model 118d. To achieve this, once the parameters of the RL framework are defined and the encoding of unitary properties 118b is generated, the control policy 118a and the object feature extraction network are trained and their corresponding weights are frozen. During training, the object feature extraction network learns to extract physical parameters of new objects based on corresponding depth images or other sensor values ​​and encode the physical parameters of the new objects in latent space. Based on the encoded object parameters, the parameters of the RL framework, the encoding of unitary properties 118b, and the state-action mapping for manipulating the unitary objects, the control policy 118a learns to generate a state-action mapping and determine the actions to be taken on the new objects to manipulate them.

[0092] Thereafter, state adapter model 118c and behavior adapter model 118d simply need to be trained to adapt control policy 118a to different objects. The manner in which state adapter model 118c and behavior adapter model 118d generalize control policy 118a for different objects will be described in more detail in conjunction with FIG. 3C.

[0093] 3B, an example method 310 for training a control policy 118a to adapt to different composite objects is shown, according to some example embodiments. In particular, the control policy 118a is trained to generate a single latent space for different composite object parameters and a single state-to-action mapping for different composite objects having different sizes, shapes, etc. For example, the unitary property 118b of a unitary object may correspond to a cubic box with sides of 9 cm. The control policy 118a is then trained to adapt to various object shapes and sizes that differ from the unitary shape (i.e., cube) and unitary size (i.e., 9 cm x 9 cm x 9 cm) of the unitary object.

[0094] At 312, a data set is generated that is comprised of a collection of random synthetic objects. The collection of random synthetic objects may include synthetic objects having random object shapes. The random object shapes may be different from the unitary shape of the unitary object.

[0095] In one example, a collection of random synthetic objects may be synthetically generated, for example, by generating a synthetic depth image. Based on the synthetic depth image, object shapes different from unitary shapes may be identified. To this end, the synthetic objects may have random object shapes different from unitary shapes, i.e., unitary object cubes. For example, the random object shapes of the synthetic objects may be, but are not limited to, rectangles, cylinders, spheres, cones, truncated cones, prisms, pyramids, pegs, and other real-world shapes that blend several different shapes. To this end, the synthetic depth image may show the different shapes of the synthetic objects.

[0096]

number

[0097]

number

[0098]

number

[0099]

number

[0100]

number

[0101] In one example, supervised learning may be used to predict synthetic object size and shape for a collection of random synthetic objects. The first two terms of the loss adjust the supervised learning for prediction, and the KL divergence loss adjusts the learned feature space to mitigate overfitting.

[0102] 3C illustrates an example method 320 for generalizing control policy 118a with state adapter model 118c and behavior adapter model 118d, according to some embodiments of the present disclosure. In particular, state adapter model 118c and behavior adapter model 118d are trained to learn to generalize control policy 118a to generate behaviors for different objects.

[0103] Some embodiments of the present disclosure recognize that if a new object is represented by the same Markov decision process (MDP) as the MDP defined for the unitary object, then the control policy 118a will receive inputs that indicate the current state of the new object. However, the output behavior generated by the control policy 118a for the new object may not be appropriate for the new object because it may be selected based on the unitary properties 118b of the unitary object. Some embodiments of the present disclosure also recognize that, in certain cases, the object properties and behavior trajectories of the new object during a manipulation operation may be similar to the unitary properties and behaviors generated for the unitary object. Subsequently, new behaviors for the new object may be described by trivial transformations in the latent space of encodings of the unitary properties and / or state-to-action mappings.

[0104] In one example, control policy 118a may include a state space consisting of encodings of different states of a unitary object, a behavior space consisting of encodings of different actions that can be performed on the unitary object to manipulate the unitary object according to a task, and a mapping between the state space and the behavior space that indicates one or more actions for a particular state of the unitary object.

[0105] At 322, one or more linear transformations are learned by the state adapter model 118c to create state corrections to the states of the unitary object for the composite object. The state adapter model 118c enables adaptation of the control policy 118a. The state adapter model 118c is configured to create state corrections to the states of the composite object having composite object parameters that differ from the unitary properties 118b.

[0106]

number

[0107] In particular, projecting the latent space of the composite object onto the latent space of the state of the unitary object may indicate differences in the composite object parameters of the composite object relative to the unitary properties 118b of the unitary object. In one example, the differences between the composite object parameters and the unitary properties 118b may be based on shape, size, weight, texture, material, density, etc. For example, the current state of the composite object may be mapped to the initial state of the unitary object, i.e., the state before the manipulation action. A state correction for the composite object may be generated based on the differences between the composite object parameters of the composite object in the projected latent space and the unitary properties 118b of the unitary object. For example, the state correction may indicate a change in the current state of the composite object required to correct the state of the composite object to a desired state according to the task. The state correction may indicate, for example, a required angle or position or pose corresponding to one or more faces of the composite object.

[0108]

number

[0109] Furthermore, the state corrections, when applied to the composite object, can become the corrected state of the composite object. A behavior for the composite object based on the composite object parameters is determined based on the trained behavior adapter model 118d.

[0110] At 324, one or more linear transformations are learned by behavior adapter model 118d to create behavior corrections for actions for the unitary object. The behavior adapter model 118d enables adapting control policy 118a to create corrective behaviors for the composite object based on the corrective states and the actions for the unitary object. The behavior adapter model 118d is configured to create behavior corrections for actions for the composite object having composite object parameters that differ from unitary property 118b. In this regard, control policy 118a may be configured to create corrective behaviors for the robot manipulator 102 to manipulate the composite object having composite object parameters based on the state corrections generated by state adapter model 118c and the behavior corrections generated by behavior adapter model 118d.

[0111] Note that during training, the control policy 118a may learn to map the state of the unitary object to the robot manipulator's behaviors 102 for manipulating the unitary object according to a task. For example, the task may be an assembly task, a packing task, etc. The behavior may also be a turning behavior or a reorientation behavior. In particular, the control policy 118a may learn to map the state space of the unitary object to an action space associated with the task. For example, based on this mapping, the control policy 118a may learn to identify the actions required to be performed on the unitary object to change the unitary object's current state to its desired state. For example, such actions generated by the control policy 118a may not be dependent on composite object parameters, i.e., may be compatible for use with different composite objects having different parameters, e.g., in terms of size, shape, material, density, weight, texture, etc.

[0112]

number

[0113]

number

[0114] To transform the behavior generated by the control policy 118a, the behavior adapter model 118d is executed to generate a behavior correction for the behavior for the unitary object. Based on the behavior correction, a corrective behavior for the composite object may be identified. In particular, the behavior adapter model 118d is configured to generate a behavior correction for the behavior for the unitary object based on the correction state for the composite object, the composite object parameters during training, and the simulated environment.

[0115]

number

[0116]

number

[0117]

number

[0118]

number

[0119] In this way, the state adapter model 118c or the action adapter model 118d maximizes the likelihood of a higher reward-to-go outcome.

[0120] At 326, the control policy 118a is adapted to the composite object based on the learning. In this regard, transformations for the control policy 118a are learned based on the state adapter model 118c and the behavior adapter model 118d. Based on the state adapter model 118c and the behavior adapter model 118d, the RL control policy and the control policy 118a may be trained to adapt to new and / or unseen objects in the real world to manipulate such objects according to a task.

[0121] The generalization introduced by the state adapter model 118c and the behavior adapter model 118d allows training to be performed only in simulation, while the control policy 118a generalizes to new, unseen objects with different shapes in the real world. This reduces real-world training time to zero. Embodiments of the present disclosure are effective for a variety of tabletop manipulation actions, including reorienting an object to a different orientation, pushing an object to a different location, grasping an object in different ways at different positions, sliding, tilting, rolling, and throwing.

[0122] This disclosure describes techniques for learning generalizable robotic skills for manipulating real-world objects from simulated data alone. Specifically, a control policy 118a, consisting of a neural network trained with reinforcement learning, is trained only on unitary objects with predetermined or predefined unitary properties 118b, such as unitary shape and unitary size. A state adapter model 118c and a behavior adapter model 118d are then trained to adapt the control policy 118a to determine actions for manipulating different random objects based on synthetic depth images of the random synthetic objects.

[0123] The proposed framework for training the control policy 118a involves learning to manipulate unitary objects based on reinforcement learning. Furthermore, an object feature extraction network, e.g., a neural network trained with supervised learning, is used to extract low-dimensional object information from high-dimensional synthetic object depth images of a collection of random synthetic objects. Supervised learning of the object feature extraction network may be performed on a generated dataset collected through simulation to learn a latent space (or a projection of the latent space) by predicting object shapes and sizes for a collection of synthetically generated random synthetic objects.

[0124] Additionally, object-specific state adapter models 118c and behavior adapter models 118d are trained to adapt the control policy 118a to new objects by adjusting the state and action spaces. The state adapter model 118c and behavior adapter model 118d may determine one or more linear transformations for the composite object parameters of the composite object from the unitary properties. The state adapter model 118c and behavior adapter model 118d are trained using a policy gradient-based approach. The state adapter model 118c then adjusts the state of the unitary object based on the state of the new object. Furthermore, the behavior adapter model 118d changes the output of the control policy 118a according to the new object, improving the performance of the controller 118. The control policy 118a, state adapter model 118c, and behavior adapter model 118d all comprise corresponding neural networks that can be trained in a simulated environment and zero-shot transferred to a series of real-world operational tasks.

[0125]

number

[0126] Diagram 400 illustrates different states of a unitary object 402. At 404, the state of the unitary object 402 indicates that the narrow longitudinal side 402a of the unitary object 402 faces downward, and the wide longitudinal side 402b of the unitary object 402 extends in the vertical direction. Also at 404, the end face 402c of the unitary object 402 is inclined toward the Y-axis. Furthermore, at 406, the state of the unitary object 402 indicates that the narrow longitudinal side 402a of the unitary object 402 faces downward, the wide longitudinal side 402b of the unitary object 402 extends in the vertical direction, and the end face 402c of the unitary object 402 is inclined toward the Z-axis. Furthermore, at 408, the state of the unitary object 402 indicates that the wide longitudinal side 402b faces downward, the narrow longitudinal side 402a extends in the vertical direction, and the end face 402c faces toward the Z-axis.

[0127] According to this example, the desired state of unitary object 402 is state 410. In 410, the state of unitary object 402 indicates that both narrow long side 402a and wide long side 402b extend vertically upward, and end face 402c of unitary object 402 faces downward.

[0128] For this reason, states 404, 406, and 408 may not be the desired states of the unitary object 402 for a task or step of an operation, such as an assembly task, a packing task, or the like. Subsequently, any of the current states, e.g., states 404, 406, and 408 of the unitary object 402, may have to be changed before performing the task. For example, the desired state 410 of the unitary object 402 may be predefined according to the task, i.e., by a task setting. Furthermore, the control policy 118a may learn to map the current state 402 of the unitary object to actions required to realize the desired state 410 of the unitary object 402. Based on the identified actions, the controller 114 may generate specific behavior parameters for controlling the gripper 108 of the robotic arm 104a. In this manner, the robotic manipulator 102 may manipulate the unitary object 402, i.e., perform actions to change the current state 402 of the unitary object to its desired state 410. Furthermore, the control policy 118a is generalized using the state adapter model 118c and the behavior adapter model 118d to determine the actions required to manipulate different unseen objects. Once trained, the generalized control policy 118a will apply what it has learned to manipulate new, unseen real-world objects.

[0129] The manner in which control policy 118a is trained to manipulate unitary object 402 according to a task, and the manner in which control policy 118a is adapted or adapted to new, unseen objects that differ from unitary object 402, has been described in detail in conjunction with Figures 2, 3A, 3B, and 3C. Therefore, the steps for training control policy 118a and the steps for training state adapter model 118c and behavior adapter model 118d are not described here to avoid cluttering the description.

[0130] 5 is a block diagram 500 for training a control policy 118a, according to some example embodiments. The control policy 118a may be trained to perform manipulation operations on different objects. The manipulation operations may include, for example, rotating an object, changing an object's orientation, inserting, etc.

[0131] In this regard, a simulated environment is created. For example, the simulated environment may include a simulation of the gripper 108, a simulation of the unitary object 402, and a simulation of external surfaces such as the contact surface 124 and the tabletop 126. In one example, the simulated environment may be created on a physics engine. Examples of physics engines include, but are not limited to, Mujoco, Bullet, and Drake.

[0132] In the simulated environment, synthetic depth images may be generated that consist of a collection of random synthetic objects. The synthetic depth images may exhibit different random object shapes and / or random object sizes for training the control policy 118a. Such random object shapes and / or random object sizes may differ from the unitary shapes and unitary sizes of the unitary objects 402. The random object shapes of the collection of random synthetic objects may include, for example, cubes, spheres, prisms, pyramids, cones, peg shapes, etc. Furthermore, synthetic depth images of the collection of random synthetic objects may be generated corresponding to different training object classes, resulting in random synthetic objects having different sizes, material properties, textures, densities, etc.

[0133]

number

[0134] To train and generalize the control policy 118a, a latent space of composite object parameters for a set of random composite objects is learned. In this regard, a latent space is first created to represent the unitary properties 118b of the unitary objects 402. For example, a deep neural network for the control policy 118a may be used to generate the latent space for the unitary properties 118b. For example, a supervised machine learning model for the object feature extraction network 502 may be used to generate an encoding of the composite object parameters for the set of random composite objects.

[0135]

number

[0136] Additionally, one or more linear transformations are learned to tune the state adapter model 118c and the behavior adapter model 118d, which enable the control policy 118a to generalize. In particular, the state adapter model 118c and the behavior adapter model 118d enable the control policy 118a to generate behaviors for new, unseen objects to be manipulated, such as random synthetic objects or new real-world objects.

[0137]

number

[0138]

number

[0139] 6 illustrates an example method 600 for performing a desired manipulation task on a new object, according to some embodiments of the present disclosure. The manipulation task is performed by the robot manipulator 102 based on manipulation behaviors generated by a control policy 118a having an RL-based neural network. As described above, the control policy 118a may be trained on a unitary object 402 having a predefined unitary property 118b. The control policy 118a learns to map the state of the unitary object 402 to actions for the unitary object 402. The state adapter model 118c and the behavior adapter model 118d may be used to adapt or transform the control policy 118a to determine actions for manipulating a new, unseen object.

[0140] At 602, data is collected regarding the state of the object and the object properties of the object. In one example, the object may be a part of an assembly and may be transported on a conveyor line for an assembly process. In this regard, the desired state of the object may be a prerequisite for an assembly step of the assembly process. Subsequently, if the state of the object does not meet the prerequisite for the assembly process, the object may have to be manipulated, such as being reoriented. The object may be, for example, a peg, a book, a bottle, a soap dispenser, a bowl, a cup, a glass, or a part of any object.

[0141] In one example, one or more sensors may be used to collect data about a real-world object. Various sensors, such as a force sensor 110, a vision sensor 112, and a pressure sensor, may be coupled to the robotic manipulator 102 to obtain different object properties of the object. According to this example, the data may be collected by the vision sensor 112 of the robotic manipulator 102. In such a case, the vision sensor 112 may capture a depth image of the object. Based on the depth image, the state and object properties of the object may be identified. In one example, the state may indicate the pose or orientation of the object. The object properties of the object may indicate at least one of the shape, size, density, weight, material, and / or texture of the object.

[0142] At 604, a determination is made to ascertain whether the object needs to be manipulated. For example, this determination may be made based on a comparison of the identified state, i.e., the current state of the object, with a desired state of the object as a prerequisite for the assembly process.

[0143] In one example, if the state of the identified object is the same as the desired state of the object, no manipulation of the object may be necessary. In such a case, the object may be further transported or moved to perform the assembly process, and the method 600 for manipulating an object may end. However, if the current state of the identified object is not the same as the desired state, the method 600 may proceed to 606.

[0144] At 606, an action for the object is determined. In this regard, the controller 114 may receive measurements 111 indicating the state and object properties of the object from the vision sensor 112, the force sensor 110, and / or other sensors coupled to the robotic manipulator 102. The controller 114 may trigger the trained control policy 118a to generate the action needed to manipulate the object, for example, to reorient or pivot the object from a current state to a desired state. Based on the object properties, the control policy 118a may be adapted or transformed for the object. In this regard, the state adapter model 118c and the behavior adapter model 118d may transform the control policy 118a to create a corrective action to perform the manipulation operation of the object. The transformed control policy 118a will generate a corrective action for the object.

[0145] At 608, commands are generated for the robot manipulator 102. In one example, the commands may include behavior parameters for controlling the robot arm 104a of the end tool 106 and / or gripper 108 of the robot manipulator 102, such that the end tool 106 and gripper 108 perform the determined behavior on the object. For example, the controller 114 is configured to compute and send commands or control behaviors to manipulate the object based on the corrective behavior determined by the control policy 118a. The controller 114 may send the commands to a subordinate robot controller of the robot manipulator 102. The robot manipulator 102 then applies the commands to the object to manipulate the object.

[0146] At 610, a determination is made to determine whether the new state of the object is the same as the desired state. For example, one or more sensors associated with the robotic manipulator 102 may be used to identify the new state of the object.

[0147] Furthermore, if the new state of the object is the same as the desired state, method 600 will end. However, if the new state of the object is not the same as the desired state, method 600 will return to 606. In such a case, another action for manipulating the object may be determined.

[0148] 7A, 7B, 7C, and 7D are diagrams illustrating example control trajectories for manipulating an object during a manipulation operation, according to various exemplary embodiments. For example, a series of behavioral steps may be performed to complete a single behavior of manipulating an object. For example, if the desired state of the object is vertically up state 410, the object will be reoriented to vertically up state 410 from other states such as states 404, 406, and 408.

[0149] 7A , a series of behavioral steps learned by the control policy 118a while manipulating a unitary object 704 are shown. For example, as shown at 702, the unitary object 704 is lying down. Further, as shown at 706, an action to change the orientation of the unitary object 704 is triggered, and control commands are provided to the gripper 108 and end tool 106 of the robotic manipulator 102. As shown at 708 and 710, the gripper 108 establishes contact with the unitary object 704, and the end tool 106 moves to change the orientation of the unitary object 704 by fixing one face of the unitary object 704 against a contact surface 712 and moving or pivoting another face of the unitary object 704 to bring the unitary object 704 into a vertically upright state. At 714, the vertically upright state, i.e., the desired state of the object 704, is achieved. For example, during its training, control policy 118a may learn a state-action mapping between a state space and an action space for manipulating a unitary object into a desired state.

[0150] In one example, the behavioral steps performed by the gripper 108 may include the following steps: The gripper 108 first pushes the unitary object 704 toward a contact surface 712, such as a wall, establishing contact between the unitary object 704 and the gripper 108 and between the unitary object 704 and the wall 712. The robot arm 104a or end tool 106 then moves upward while pushing the unitary object 704 toward the wall 712, beginning to rotate the unitary object 704. After the unitary object 704 has rotated to a certain angle, the robot arm 104a or end tool 106 applies a downward force to maintain contact with the unitary object 704. This is a robust method of stabilizing the unitary object 704 and preventing it from falling. Finally, the unitary object 704 is flipped up and placed on a table in a vertically upward position.

[0151] 7B and 7C, during training, control policy 118a determines the manipulation trajectory of the composite object. Having determined the actions required to manipulate unitary object 704, i.e., how to reorient unitary object 704 to a desired state, control policy 118a may learn to apply those actions to different composite objects, depicted as composite object 716 and composite object 718 (collectively referred to hereinafter as composite objects 716 and 718). For example, control policy 118a may learn to establish contact between robot manipulator 102 and different types of composite objects 716 and 718 and move the different types of composite objects 716 and 718, which have different shapes and sizes, to a desired state, i.e., a vertically upward orientation. The manner in which composite objects 716 and 718 are manipulated or reoriented to a desired vertically upward orientation is shown at 720 and 722, respectively. The behavior steps and movement patterns of the robot arm 104 a, end tool 106 and gripper 108 applied to the composite objects 716 and 718 are similar to the behavior steps learned while rotating the unitary object 704 .

[0152] FIG. 7D shows an example diagram 724 illustrating a sequence of behavioral steps, poses, and contact configurations between the gripper 108 and / or end tool 106 of the robotic manipulator 102 and a real-world object 726. For example, the control policy 118a trained on the unitary object 704 and the composite objects 716 and 718 is transformed based on the object properties of the object 726 and the state adapter model 118c and behavior adapter model 118d. The control policy 118a then generates a sequence of behavioral steps 728, 730, 732, and 734 for manipulating the object 726. As shown in the figure, in step 728, the object 726 is lying down, and its state is not the same as the desired state, i.e., a vertically-upward state. Furthermore, in steps 730 and 732, the gripper 108 may establish contact with the object 726, fix one face of the object 726 against a wall 736, and move another face of the object 726 to a vertically-upward state. In 734, a vertically upward state is realized.

[0153] Note that the desired state of all objects 704, 716, 718, and 720 being a vertically upward state is merely exemplary and should not be construed as limiting. Control policy 118a may learn different state-to-action mappings, where unitary object 704 may have different desired states according to the task and different initial or current states. In particular, the desired state may be determined based on preconditions or operational instructions for the next behavioral step of the task or computation. Furthermore, control policy 118a may also learn different possible states of unitary object 704 and corresponding actions to manipulate unitary object 704 into the desired state.

[0154] Additionally, the behavior for achieving a vertically-upward state, i.e., a pivoting behavior in which one face of the object is fixed against a surface and another face of the surface is moved to achieve a vertically-upward state, is merely exemplary and should not be construed as limiting. For example, such a change in the object's orientation may be necessary when the object being manipulated is not easily grasped in its current state and the object is not in a desired state. However, in other examples, different behaviors may be identified for changing the object's orientation to achieve a desired state.

[0155] FIG. 8 illustrates an exemplary method 800 for performing a desired manipulation task on a new real-world object, according to an embodiment. Method 800 may be performed on real-world object 726 after training control policy 118a, state adapter model 118c, and behavior adapter model 118d. Note that control policy 118a learns to manipulate unitary objects, such as unitary object 704 or 402. Control policy 118a is generalized by trained state adapter model 118c and trained behavior adapter model 118d, allowing control policy 118a learned on unitary object 402 or 704 to be adapted or transformed to suit different objects. This transformation of control policy 118a is achieved based on linear transformation of state adapter model 118c and behavior adapter model 118d according to the object properties of the new object.

[0156] In particular, the control policy 118a may generate corrective actions for the robot manipulator 102, causing the robot manipulator 102 to manipulate the object 726 by executing the actions. For example, the robot manipulator 102 includes a robot arm 104a having an end tool 106 and a gripper 108. In one example, the robot arm 104a may be controlled in an impedance control mode with a predefined stiffness in all directions. For example, the predefined stiffness may be in the range of 5 N / mm to 20 N / mm. According to this example, the predefined stiffness of the robot arm 104a may be 12 N / mm.

[0157] In this regard, at 802, data is collected regarding the state and object properties of the object 726. The data may be collected by one or more sensors coupled to the robotic manipulator 102, such as a vision sensor 112, a force sensor 110, etc. In one example, the robotic manipulator 102 may include multiple sensors coupled to the robotic arm 104a. For example, a force sensor 110 and a vision sensor 112 may be coupled to the robotic arm 104a. The force sensor 110 may be mounted on the end tool 106 above the gripper 108 and may measure external forces experienced by the robotic arm 104a.

[0158] Additionally, the vision sensor 112 may include, for example, an RGB-D (RGB-depth) camera. The RGB-D camera may capture a depth image of the object 726. Based on the depth image, object properties of the object 726 may be identified to estimate the state of the object 726. The object properties may include, for example, shape, size, weight, density, material properties, texture, pose, orientation, position, etc. In one example, a tracking tag may be attached to the object 726 to obtain accurate state information of the object 726. In such a case, the state information may be considered as the ground truth object pose or orientation of the object 726. In another example, a vision-based 6D pose prediction technique may be used to extract masks and deep features of the object 726 and track the pose of the object 726. To this end, the object properties of the object 726 may be extracted by applying processing techniques to the acquired depth images, measured forces, or measurements from other sensors associated with the robotic arm 104a.

[0159] Thereafter, at 804, the state adapter model 118c is executed to create state corrections for the state of the object 726 based on the collected data, generating a corrected state for the object 726. The state adapter model 118c enables adaptation of the control policy 118a. The state adapter model 118c is configured to create state corrections for the state of the object 726 having object properties that differ from unitary properties.

[0160] In one example, the state adapter model 118c takes as input the object properties of the object 726 and outputs a diagonal matrix indicating the state properties of the object 726. For example, the state properties may indicate the extent or space occupied by the object 726 in its current state in different directions, i.e., the X direction, the Y direction, and the Z direction. The state properties of the object 726 are used to project the latent space of the current state of the object 726 onto the latent space of the states of the unitary object 402 or 704.

[0161] In particular, the projection of the state of the unitary object 402 or 704 onto the latent space of the object 726 may indicate differences between the object properties of the object 726 and the unitary properties of the unitary object 402 or 704. In one example, the differences between the object properties and the unitary properties may be based on, for example, shape, size, weight, texture, material, density, etc. For example, the current state of the object 726 may be mapped to the initial state of the unitary object 402 or 704, i.e., the state before the manipulation action. Based on the differences between the object properties of the object 726 and the unitary properties of the unitary object 402 or 704 in the projected latent space, a state correction may be generated to correct the state space according to the object 726. For example, the state correction may indicate a change in the initial state of the unitary object 402 or 704 required to reach the current state of the object 726. Furthermore, the state correction may indicate a change in the current state of the object 726 required to correct the state of the object 726 to a desired state according to the task. The state correction may indicate, for example, a required angle or position or pose corresponding to one or more faces of the object 726 .

[0162]

number

[0163] At 806, the control policy 118a is executed to create an action for the unitary object 402 or 704 based on the corrected state of the object 726. In particular, the control policy 118a may be configured to create an action for the robotic manipulator 102 to manipulate the unitary object 402 or 704 having a predetermined unitary property such that the unitary object 402 or 704 is in a state similar to the current state of the object 726.

[0164] During training, the control policy 118a may learn to map the states of the unitary object 402 or 704 to actions 102 of the robot manipulator for manipulating the unitary object 402 or 704 in accordance with the task. In particular, the control policy 118a may learn to map the state space of the unitary object 402 or 704 to an action space associated with the task. For example, based on the mapping, the control policy 118a may learn to identify actions that need to be performed on the unitary object 402 or 704 to change the state of the unitary object 402 or 704 to its desired state.

[0165] Thereafter, when control policy 118a is implemented on real-world object 726, control policy 118a may generate a state space of corrective states for object 726 and map the state space to an action space created for manipulating unitary object 402 or 704. In particular, the actions generated by control policy 118a based on the mapping between the corrective state space for object 726 and the action space for unitary object 402 or 704 will have to be translated to work with object 726 due to differences between object properties and unitary properties.

[0166] To transform the behavior generated by the control policy 118a, at 808, the behavior adapter model 118d is executed to generate behavioral corrections to the behavior for the unitary object 402 or 704 based on the difference between the unitary properties and the object properties. Based on the behavior corrections, a corrective behavior will be generated for the object 726. In particular, the behavior adapter model 118d is configured to generate behavioral corrections to the behavior for the unitary object 402 or 704 based on the corrective state of the object 726 and the simulated environment during which the behavior adapter model 118d is being trained.

[0167] In one example, the behavior adapter model 118d may output behavior dimensions, i.e., a diagonal matrix of behaviors in different directions, i.e., X-direction, Y-direction, and Z-direction. The behavior dimensions may be used to determine the behavior to be performed on the object 726 to perform the manipulation operation. For example, if a surface of the object 726 is small in size and easy to grasp, the behavior correction may indicate gripping that surface with the gripper 108 and moving the end tool 106 while applying downward pressure to pivot or turn the object 726.

[0168] For example, a behavior performed by the robotic manipulator 102 to manipulate a small cubic unitary object may be to rotate the unitary object using an external contact surface for an assembly task. In this example, the object 726 may be pyramidal in shape, the task configuration for the task performed on the object 726 may be packing, which requires its apex to be oriented vertically upward, and a state correction indicates that the object 726 is lying on its surface. In this case, the control policy 118a may determine a reorientation or rotation behavior to be performed on the object 726 to manipulate it. However, because the contact dynamics, class, etc. of the object 726 differ from the unitary properties 118b, the behavior would not be directly applicable to the object 726. To address this, the behavior adapter model 118d may determine a behavior correction that indicates reorienting the object 726 by grasping the apex with the gripper 108 and moving the gripper robot arm 104a along the horizontal axis to reorient the object 726. A corrective behavior may then be generated based on the behavior correction that can be used to manipulate the object 726.

[0169] At 810, the robot manipulator 102 is controlled to manipulate the object 726 based on the corrective behavior for the unitary object 402 or 704. In one example, the corrective behavior includes behavior parameters for controlling the interaction of the robot manipulator 102 with the object 726. For example, behavior parameters may be generated for different parts of the robot manipulator 102, such as the rotation angle of the end tool 106 and / or the robot arm 104a, the gripping force of the gripper 108, the lifting force, drag force, or tensors of the end tool 106 and / or the robot arm 104a. Based on the behavior parameters, the robot manipulator 102 may be controlled to manipulate or reorient the object 726.

[0170] Note that the example of learning a turn action for unitary object 402 or 704 during training of control policy 118a and generating an action for object 726 as a turn action is illustrative and should not be construed as limiting. In other embodiments of the present disclosure, control policy 118a may learn to perform different actions, such as turn, insert, or reorient, on unitary object 402 or 704 to achieve its desired state according to the next action step in a task or process. Furthermore, trained control policy 118a may generate any action, such as turn, insert, or reorient, for object 726 based on the current state of object 726, the desired state of object 726, the task associated with object 726, and / or differences between object properties and unitary properties.

[0171] Referring to FIG. 9A, an exemplary method 900 for generating corrective actions for an object 726 is shown, according to an embodiment.

[0172] At 902, an encoder is executed to process object properties of the object 726 to create a property encoding or feature encoding in a latent space. For example, the encoder may correspond to the object feature extraction network 502. Further, the property encoding may indicate the shape, size, material, density, etc. of the object 726. In one example, the property encoding of the object properties may be performed in a latent space having a unitary property encoding. For example, the object properties may be associated with the object shape and object size of the object 726, and the unitary property 118b may be associated with the unitary shape or unitary size of the unitary object 402 or 704. In one example, the property encoding in the latent space may represent a cluster of object property encoding points, where the cluster corresponds to the object shape and object size of the object 726, such as a nonlinear cylinder. The encoding of unitary properties 118b in latent space may indicate clusters of unitary property encodings, where the clusters correspond to unitary shapes, such as cubes, and unitary sizes.

[0173] At 904, the encoder sends the property encoding to the state adapter model 118c and the behavior adapter model 118d. In particular, the encoder of the object feature extraction network 502 may send property encodings of object properties to the state adapter model 118c and the behavior adapter model 118d. For example, the property encoding may indicate the difference between an object property and a unitary property. The encoder sends the encoded object property in the latent space with the encoded unitary property 118b to the state adapter model 118c and the behavior adapter model 118d.

[0174] At 906, the state adapter model 118c and the behavior adapter model 118d are executed to create corrective states and corrective actions for the object 726. In particular, the state adapter model 118c may generate state corrections for the object 726 based on the property encodings and the encodings of the states of the unitary object 402 or 704. The behavior adapter model 118d may also generate behavior corrections for the object 726 based on the property encodings and the corrective states. In one example, the corrective actions generated by the behavior adapter model 118d may be applied to the object 726 to manipulate the object 726 into a desired orientation or state.

[0175] 9B, an exemplary method 910 for generating corrective actions for an object 726 is shown, according to an embodiment. According to this example, the object properties relate to the object shape of the object 726, and the unitary properties 118b relate to the unitary shape of the unitary object 402 or 704. For example, the unitary shape may be a box or a cube. Alternatively, the object shape may be a nonlinear cylinder, such as a soap dispenser or a ketchup bottle.

[0176] At 912, data is collected regarding the state and object shape of the object 726. For example, the state or pose and object shape may be identified based on a depth image of the object 726 generated by an RGB-D camera.

[0177] At 914, the state adapter model 118c is executed to create state corrections for the state of the object having an object shape different from the unitary shape based on the difference between the object shape and the unitary shape. In one example, the state adapter model 118c takes the object shape of the object 726 as input and projects the latent space of the state of the object 726 and / or the object shape of the object 726 onto the latent space of the state of the unitary object 402 or 704 and / or the unitary shape of the unitary object 402 or 704. A state correction for the state of the object 726 may be determined based on the difference in the latent spaces to generate the corrected state.

[0178] Further, at 916, control policy 118a executes to create an action for unitary object 402 or 704 based on the corrective state of object 726. In one example, control policy 118a generates an action for the unitary object based on a state-action mapping between the corrective state of object 726 and the action space of unitary object 402 or 704.

[0179] At 918, behavioral adapter model 118d may be executed to generate behavioral modifications to the behaviors generated by control policy 118a based on differences between the object shape and the unitary shape. In one example, the behaviors generated by control policy 118a may not depend on the object shape. Behavior modifications may be generated to transform the generated behaviors to conform to, for example, the type of task performed on the unitary object, the unitary size, the external environment of the unitary object, the unitary shape, the robotic manipulator used, and other properties associated with unitary object 402 or 704. Based on the behavior modifications, the behaviors to be performed on object 726 may be transformed based on features associated with object 726.

[0180] At 920, the corrective behavior is used to control the robot manipulator 102 to manipulate the object 726.

[0181] The behavioral adapter model 118d allows the control policy 118a to be adjusted according to object properties or object shape. In particular, despite the wide variety of original behavioral trajectories for manipulating the unitary object 402 or 704, the state adapter model 118c and the behavioral adapter model 118d summarize these behavioral trajectories, allowing the control policy 118a to adapt to new objects.

[0182] It should be understood that each block of the flow diagrams of methods 200, 300, 310, 320, 600, 800, 900, and 910 may be implemented by various means, such as hardware, firmware, processors, circuits, and / or other communication devices executing software that includes one or more computer program instructions. For example, one or more of the procedures described above may be embodied by computer program instructions. In this regard, the computer program instructions incorporating the procedures described above may be stored by memory 118 of controller 114 and executed by processor 117, employing embodiments of the present invention. It should be understood that any such computer program instructions may be loaded into a computer or other programmable device (e.g., hardware) to create a machine, such that the resulting computer or other programmable device implements the functions specified in the flow diagram blocks. These computer program instructions may also be stored in a computer-readable memory that directs the computer or other programmable device to function in a particular manner, such that the instructions stored in the computer-readable memory, when executed, generate a product that implements the functions specified in the flow diagram blocks. The computer program instructions may also be loaded into a computer or other programmable device to cause the computer or other programmable device to perform a series of operations, creating a computer-executed process such that the instructions operating on the computer or other programmable device provide operations to implement the functions specified in the flow diagram blocks.

[0183] Thus, the blocks of methods 200, 300, 310, 320, 600, 800, 900, and 910 support a combination of means for performing the specified functions and a combination of acts for performing the specified functions. It should also be understood that one or more blocks of methods 200, 300, 310, 320, 600, 800, 900, and 910, and combinations of blocks of methods 200, 300, 310, 320, 600, 800, 900, and 910, may be implemented by a dedicated hardware-based computer system that performs the specified functions, or by a combination of dedicated hardware and computer instructions.

[0184] Furthermore, by using the steps described in the accompanying embodiments of methods 200, 300, 310, 320, 600, 800, 900, and 910 implementing a controller for controlling the robotic manipulator 102, the accuracy of manipulation actions on new, unseen objects may be significantly improved.

[0185] The embodiments of the present disclosure described above may be implemented in any of numerous ways. For example, embodiments may be implemented using hardware, software, or a combination thereof. If implemented in software, the software code may be executed on any suitable processor or collection of processors, whether located on a single computer or distributed among multiple computers. Such a processor may be implemented as an integrated circuit including one or more processors on a single integrated circuit component. However, a processor may be implemented using circuitry in any suitable format.

[0186] Additionally, embodiments of the present disclosure may be implemented as described methods. Operations performed as part of a method may be ordered as appropriate. Thus, embodiments may be configured to perform operations in an order different from the illustrated order, and may include performing some operations simultaneously even though the illustrated embodiments show sequential operations.

[0187] The use of ordinal numbers such as "first," "second," etc. in the appended claims does not, by itself, imply a preference, precedence, or order in which one claim element precedes another, nor does it imply a chronological order in which the actions involved in a method should be performed; rather, it merely serves to distinguish claim elements (however ordinal numbers are used) as labels to distinguish one named claim element from another named claim element.

[0188] Although the present disclosure has been described by way of examples of preferred embodiments, it will be understood that various other adaptations and modifications may be made within the spirit and scope of the present disclosure.

[0189] Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the spirit and scope of this disclosure.

Claims

1. 1. A controller for controlling a robotic manipulator to manipulate an object according to a task, the controller comprising: a memory, the memory comprising: and storing a control policy configured to generate behaviors for the robot manipulator to manipulate a unitary object having a predetermined unitary property, the control policy configured to map states of the unitary object to the behaviors of the robot manipulator to manipulate the unitary object according to the task, the memory further comprising: a state adapter model configured to generate state corrections for states of the object having object properties that differ from the unitary properties based on the differences between the object properties and the unitary properties; a behavioral adapter model configured to generate a behavioral correction for the behavior generated by the control policy based on the difference between the unitary property and the object property, and the controller further stores: a processor, the processor comprising: collecting data regarding the state of the object and the object properties; executing the state adapter model to generate the state corrections to the state of the object based on the collected data to generate a corrected state of the object; Executing the control policy using the corrective state of the object to create the behavior for the unitary object; executing the behavioral adapter model to make the behavioral corrections to the behavior for the unitary object to generate corrective behavior; A controller configured to control the robotic manipulator with the corrective behavior for the unitary object.

2. The controller of claim 1 , wherein the behavior adapter model is configured to create the behavior correction to the behavior for the unitary object based on the correction state of the object and a property encoding of the object property for the object.

3. The processor further comprises: executing an encoder to process the object properties of the object to create the property encoding; sending the property encoding to the state adapter model and the behavior adapter model; The controller of claim 2 , configured to create the corrective behavior based on the property encoding by executing the control policy, the state adapter model, and the behavior adapter model.

4. The controller of claim 1 , wherein the object properties relate to at least one of shape, size, density, weight, or material.

5. If the object property and the unitary property are associated with the shape of the object and the unitary object, respectively, the processor further: using the state adapter model to generate the state correction for the state of the object having an object shape that differs from a unitary shape based on the difference between the object shape and the unitary shape; The controller of claim 4 , configured to use the behavioral adapter model to generate the behavioral corrections to the behavior generated by the control policy based on the difference between the unitary shape and the object shape.

6. To train the control policy, the processor further and generating a dataset comprised of a set of random objects, the set of random objects having a corresponding random shape that differs from a unitary shape of the unitary object, the processor further comprising: learning the feature space for the random objects based on differences between each of the random object shapes and the unitary shape; The controller of claim 1 , further comprising: determining a predicted object shape for each set of random objects based on the learned feature space.

7. The controller of claim 1 , wherein the object properties are captured using one or more sensors.

8. The controller of claim 1 , wherein the corrective behavior is comprised of behavior parameters for controlling an interaction between the robotic manipulator and the object.

9. The controller of claim 1 , wherein the control policy is comprised of a neural network trained with reinforcement learning.

10. The controller of claim 1 , wherein the state adapter model and the behavioral adapter model each comprise a neural network, and the state adapter model and the behavioral adapter model are trained using machine learning.

11. To train the controller, the processor further comprises: training the state adapter model with one or more linear transformations to generate state corrections for a composite object to the states of the unitary object; training the behavior adapter model with one or more linear transformations to generate behavior corrections to the behavior for the unitary object based on the correction state; The controller of claim 1 , configured to adapt the control policy to the composite object based on the learning.

12. The processor further comprises: generating a pseudo-environment for the task based on the unitary property; The controller of claim 1 , configured to generate the behavioral correction based on the simulated environment.

13. 1. A method for controlling a robotic manipulator to manipulate an object according to a task, the method comprising: collecting data regarding the object's state and object properties; using a state adaptor model to create a state correction to the state of the object based on a difference between the object property and a predetermined unitary property of a unitary object to generate a corrected state of the object; and generating actions for the unitary object using the corrective states of the object using a control policy, the actions being actions for a robot manipulator to manipulate the unitary object, the control policy being configured to map states of the unitary object to the actions of the robot manipulator to manipulate the unitary object according to the task, the method further comprising: using a behavioral adapter model to generate a behavioral correction to the behavior for the unitary object based on the behavior for the unitary object created by the control policy and the difference between the unitary property and the object property to generate a corrective behavior; and controlling the robotic manipulator with the corrective action for the unitary object.

14. processing the object properties of the object by executing an encoder to create a property encoding; sending the property encoding to the state adapter model and the behavior adapter model; 14. The method of claim 13, further comprising: creating the corrective behavior based on the property encoding by executing the state adapter model and the behavior adapter model.

15. The method of claim 13 , wherein the object properties relate to at least one of shape, size, density, weight, or material.

16. where the object property and the unitary property are associated with the shape of the object and the unitary object, respectively; using the state adapter model to generate the state correction for the state of the object having an object shape that differs from a unitary shape based on the difference between the object shape and the unitary shape; 16. The method of claim 15, further comprising: using the behavioral adapter model to generate the behavioral corrections to the behavior generated by the control policy based on the difference between the unitary shape and the object shape.

17. To train the control policy, the method further comprises: generating a data set comprised of a set of random objects, the set of random objects having corresponding random shapes that differ from the unitary shape of the unitary object; learning the feature space for the random objects based on a difference between each of the random object shapes and the unitary shape; 14. The method of claim 13, further comprising: determining a predicted object shape for each of the set of random objects based on the learned feature space.

18. The method of claim 13 , wherein the control policy comprises a neural network trained with reinforcement learning.

19. The method of claim 13 , wherein the state adapter model and the behavioral adapter model each comprise a control policy, and the state adapter model and the behavioral adapter model are trained using machine learning.

20. A non-transitory computer-readable storage medium embodied with a program executable by a processor for performing a method, the method comprising: collecting data about object states and object properties; using a state adaptor model to create a state correction to the state of the object based on a difference between the object property and a predetermined unitary property of a unitary object to generate a corrected state of the object; and generating actions for the unitary object using the corrective states of the object using a control policy, the actions being actions for a robot manipulator to manipulate the unitary object, the control policy being configured to map states of the unitary object to the actions of the robot manipulator to manipulate the unitary object according to the task, the method further comprising: using a behavioral adapter model to generate a behavioral correction to the behavior for the unitary object based on the behavior for the unitary object created by the control policy and the difference between the unitary property and the object property to generate a corrective behavior; and controlling the robotic manipulator with the corrective action for the unitary object.

Images