Motion abstraction controller for fully actuated robotic manipulator

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for controlling a robotic manipulator having a plurality of joints. One of the methods includes obtaining a control input including one or more speed values specifying a target speed for a reference point in a given coordinate system; determining a respective joint velocity for each of the plurality of joints by generating a solution to an optimization problem formulated from the control input; and controlling the robotic manipulator, including moving a plurality of joints of the robotic manipulator according to respective joint speeds to approximate the control input.

Description

Technical Field

[0001] The present description relates to robotics and, more particularly, to controlling the movement of a robotic manipulator. Background Art

[0002] In robotics, robot manipulator kinematics involves the movement of the robot's end effector and how the robot's joints need to move to achieve the desired movement of the end effector. The end effector usually refers to the device at the end of the robot's arm, but in some cases, such as in serial robot manipulators, the end effector may also refer to the last link of the robot. Summary of the invention

[0003] This specification describes systems and techniques that allow a user to more easily generate control instructions for operating a robot. For example, the control instructions can include instructions for operating a robot to manipulate, such as grasping, an object.

[0004] According to one aspect, a method for controlling a robotic manipulator having multiple joints is provided, wherein the method includes: obtaining a control input, the control input including one or more velocity values ​​specifying a target velocity of a reference point in a given coordinate system; determining a corresponding joint velocity of each of a plurality of joints by generating a solution to an optimization problem formulated from the control input; and controlling the robotic manipulator, including causing a plurality of joints of the robotic manipulator to move according to the corresponding joint velocities to approximate the control input.

[0005] The reference point may be located at a specified position relative to a plurality of contact points on a robotic manipulator used to manipulate the object.

[0006] The reference point may be the instantaneous centroid of multiple contact points on the robotic manipulator used to manipulate the object.

[0007] The target velocity of the reference point may include a target angular velocity defined with reference to a given coordinate system.

[0008] The target velocity of the reference point may include a target linear velocity defined with reference to a given coordinate system.

[0009] The control input may further specify a corresponding target velocity for each contact point relative to a reference point.

[0010] Generating a solution to an optimization problem may include: formulating the optimization problem as a least squares optimization problem, the problem including an objective function, the objective function including terms that depend on a set of decision variables, the set of decision variables including (i) a first decision variable corresponding to respective joint velocities of a plurality of joints, (ii) a second decision variable corresponding to respective velocities of a contact point relative to a reference point, (iii) a third decision variable corresponding to a target linear velocity of the reference point, and (iv) a fourth decision variable corresponding to a target angular velocity of the reference point, and a constraint set defining lower or upper limits for possible values ​​of some or all of the decision variables; and generating a solution to the optimization problem by calculating, subject to the constraint set, an optimized value of the first decision variable that optimizes the objective function.

[0011] The least squares optimization problem can be of the following form: by constraint where x is the decision variable, , and are the weight matrix, coefficient matrix, and bias matrix that define the objective function, respectively, and , and are the lower bound, upper bound, and coefficient matrix defining the constraint set, respectively.

[0012] The robotic manipulator may be a dexterous robotic hand.

[0013] Obtaining data specifying a target speed of the object may include obtaining teleoperation data transmitted from a teleoperation device of the robot manipulator.

[0014] A teleoperated device may have motion sensors with lower degrees of freedom (DoF) than a robotic manipulator.

[0015] The method may further include controlling the robotic manipulator using a remote operation device to generate training data for training a machine learning model, the machine learning model being configured to generate control signals for controlling the robotic manipulator to perform one or more different tasks.

[0016] Machine learning models can be trained on training data using reinforcement learning techniques.

[0017] According to another aspect, one or more computer-readable storage media encoded with instructions are provided, which, when executed by one or more computers, cause the one or more computers to perform the operations of the above-described method aspects.

[0018] According to another aspect, a system is provided, which includes one or more computers and one or more storage devices storing instructions, which, when executed by the one or more computers, cause the one or more computers to perform corresponding operations of the above-mentioned method aspects.

[0019] It will be appreciated that features described in the context of one aspect may be combined with features described in the context of another aspect.

[0020] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0021] The robotic motion control techniques described in this specification allow users to more easily generate control instructions for robotic manipulators with any number of degrees of freedom, including fully actuated robotic hands, to facilitate customized and dexterous object manipulation by the robotic manipulator that meets their needs. Specifically, the use of the described techniques can allow users to generate precise control instructions for the robotic manipulator by using a controller with many fewer degrees of freedom than the robot (or, by otherwise submitting inputs specifying many fewer degrees of freedom than the degrees of freedom required to actually control the robot). In contrast, many existing methods for operating (or remotely operating) robots with a large number of degrees of freedom, performing dexterous robotic tasks, or both require highly customized hardware through which users can appropriately generate control instructions. Such hardware is unlikely to achieve scalable adoption because it is expensive to build, maintain, and operate, and is sometimes not interactive enough or not sensitive enough.

[0022] The robotic motion control techniques described in this specification enable various technical improvements that were previously impossible. For example, custom and expensive hardware, such as tactile gloves, required to remotely operate a dexterous robotic manipulator are no longer required. Instead, any suitable remote control device equipped with a motion sensor having a lower, sometimes much lower, degree of freedom than the actual robotic manipulator can effectively control the robotic manipulator. This not only expands the usability of the robotic manipulator in a wide range of technical use cases, but also expands the accessibility of the robotic manipulator by different individual users. For example, the described techniques facilitate scalable generation of demonstration data by many expert demonstrators for training a machine learning model that is configured to generate model outputs that can be used to control a robotic manipulator.

[0023] As another example, the described robot motion control technology can help robot control based on reinforcement learning. Many existing RL-based robot tasks require selecting an action from a large discrete action space, a continuous action space, or a mixed action space, that is, some of the sub-actions are discrete and other sub-actions are continuous. Whether during training or during operation after training, iteratively searching for possible action sets is computationally intensive, and consumes significant computing resources when the action space is large or continuous or both. By combining the described robot motion control technology, the robot manipulator can be effectively controlled using control inputs that define a relatively small number of actions. In addition, during training, the described technology allows more efficient exploration of the action space, and therefore it is easier to learn more effective action selection strategies. Therefore, the described robot motion control technology can control the robot manipulator in the RL-based robot task with a large discrete action space, a continuous action space, or a mixed action space while maintaining effective performance with reduced latency and reduced computing resource consumption.

[0024] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a diagram illustrating an example system.

[0026] Figure 2 is a flow chart of an example process for controlling a robotic manipulator.

[0027] Figure 3 is a diagram illustrating an example reinforcement learning system.

[0028] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0029] Figure 1 is a diagram illustrating an example system 100. System 100 is an example of a system capable of implementing the robotic motion control techniques described in this specification.

[0030] The system 100 includes a plurality of functional components, including a robotic manipulator 110, a robotic controller 120, which is implemented as a computer program on one or more computers in one or more locations that controls the movement of the robot 110, and an input device 130 through which a user can generate control inputs 132 for operating the robot 110, e.g., by inputting or otherwise specifying a target movement of the robotic manipulator 110, e.g., a target pose, a target velocity, or both.

[0031] In this specification, a robotic manipulator (or simply robot) is a machine having one or more movable components and one or more actuators capable of achieving physical movement of the one or more movable components.

[0032] exist Figure 1 In the example of , the robot 110 has been illustrated as a robotic hand, which includes a plurality of links, for example, a link 112 (which corresponds to a distal finger portion) and a link 114 (which corresponds to an intermediate finger portion), and joints connecting the links, for example, a joint 116 (which corresponds to a distal finger joint) coupled between the links 112 and 114. Each joint, which can be, for example, a revolute joint, a translational joint, or a prismatic joint, can include an actuator that moves the link to a target pose and at a target velocity based on a command 128 received from a robot controller 120. For example, the actuator can include a motor or other energy source that moves the link to a target pose and at a target velocity.

[0033] The robot controller 120 can generate the command 128 based on a control input 132 that specifies a target velocity for the robot 110 .

[0034] The input device 130 can be equipped with one or more motion sensors (e.g., infrared or optical motion sensors) configured to detect movement of the input device and output one or more velocity values ​​derived from the detected movement. For example, where the input device 130 is a three-degree-of-freedom (3 DoF) input device, such as an input device equipped with a 3 DoF motion sensor, the control input 132 can include a corresponding linear velocity value along each axis (e.g., x, y, or z axis) of a given three-dimensional coordinate system. As another example, where the input device 130 is a six-degree-of-freedom (6 DoF) input device, the control input 132 can additionally include a corresponding angular velocity value along each axis of the three-dimensional coordinate system.

[0035] Additionally or alternatively, the robot controller 120 can generate the command 128 based on a control input 132 that specifies a target pose for the robot 110. For example, the control input 132 can include spatial coordinates (e.g., x, y, and z values) in a three-dimensional coordinate system, and can also include orientation coordinates (e.g., pitch, yaw, or roll values) in the three-dimensional coordinate system.

[0036] The target velocity, target pose, or both for the robot 110 can be defined relative to any reference point 142 in space defined by a relative three-dimensional coordinate system. The reference point 142 of the robot 110 can generally indicate a "point of interest" for object manipulation. In some embodiments, the reference point can be the center of mass or gravity of an actual (or virtual) object to be manipulated by the robot - e.g., grasped. In some other embodiments, the reference point can be located at a specified position relative to a plurality of contact points on the robot 110 used to manipulate the object. For example, the reference point can be the instantaneous center of mass of a plurality of contact points on the robot that can be used to manipulate the object, such as Figure 1 The instantaneous center of mass of the fingertips of the robot 110 shown in .

[0037] As another example, where the input device 130 includes one or more depressible keys or buttons, the control input 132 can further specify a respective target speed for each of the one or more components of the robot 110, such as a respective target speed for a plurality of contact points of the robot to move away from a reference point (i.e., Figure 1 ), which can be similarly defined relative to a three-dimensional coordinate system. In this example, the value of the corresponding target speed can be derived from the state or movement information reported by the touch sensor, transducer or switch of the input device 130 coupled to the corresponding depressible key or button.

[0038] Specifically, the robot controller 120 uses a least squares solver 122 to determine the value of the joint velocity 124 of one or more joints of the plurality of joints of the robot 110 based on the target velocity of the robot 110 specified by the control input 132. The least squares solver 122, examples of which are described in http: / / ceres-solver.org, is typically implemented as software configured to generate a solution to a given optimization problem specific to the robot model by calculating the values ​​of the joint velocities that provide the target velocity using, for example, an iterative numerical or analytical algorithm. The least squares solver 122 allows for greater versatility than unconstrained differential kinematics solvers, such as those based on Jacobian inverses, by allowing for the specification of affine constraints that can be customized for the decision variables. In order to control the robot using the velocity values ​​of the joints, the least squares solver 122 can provide the determined joint velocity values ​​as input to a motion controller 126 of the robot controller 120.

[0039] The motion controller 126 can then generate and provide commands 128 to the actuators of each joint, which cause the joints to move at their respective target velocities, thereby controlling the robot 110 to approach the control input 132 .

[0040] Determining the joint velocities 124 from the control inputs 132 using the least squares solver 122 will be described further below, but in short, this configuration allows the robot controller 120 to provide an abstraction level higher than the hardware interface level of the motion controller to allow the user to more easily specify a target velocity for the robot 110, for example, when controlling the robot 110 to perform a robotic task. With the robot controller 110 described in this specification, the user will be able to effectively control the robot 110 by using an input device 130 equipped with a motion sensor having fewer degrees of freedom (DoF) than the robot 110. In some examples, the user will control multiple fingertips of a fully actuated robotic hand to move in a coordinated manner, despite only having to provide a control input 132 that includes velocity values ​​for fewer DoFs than the actual number of joints of the robotic hand.

[0041] Additionally or alternatively, the robot controller 120 can use a different solver (e.g., an inverse kinematics solver) to determine position and angle parameters for each of one or more of the plurality of joints of the robot 110 based on the target pose of the robot 110 specified by the control input 132. The angle of a joint is the angle between the two links that the joint connects. After being provided with the determined joint position and angle parameters, the motion controller 126 can similarly generate and provide commands 128 to the actuators of each joint that cause the joint to move to its corresponding target position and angle.

[0042] Although shown as being logically adjacent, the robot 110, the robot controller 120, and the input device 130, as well as the software or hardware modules that perform the operations, may actually be located across different computing devices or different memory devices. In some embodiments, these components can be implemented in multiple locations coupled to each other through any appropriate communication network, such as an intranet or the Internet, or a combination of networks. For example, the input device 130 can be a remotely operated device that is physically separated from the robot 110, the robot controller 120, or both. As another example, although shown as being included as part of the robot controller 120, the least squares solver 122 can actually be implemented on a remote server that communicates with the system 100 through a data communication network, and the system 100 can use the solver, such as by using an application program interface (API) provided by the solver.

[0043] Figure 2 is a flow chart of an example process 200 for controlling a robotic manipulator. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed system—e.g. Figure 1 The system 100 of can perform the process 200. The robotic manipulator can have multiple joints.

[0044] The system obtains a control input including one or more velocity values ​​(step 202). The control input can be generated by a user of the system using an input device of the system. The input device can be equipped with one or more motion sensors configured to detect movement of the input device and output one or more velocity values ​​derived from the detected movement. For example, the control input can include a corresponding linear velocity value along each axis of a three-dimensional coordinate system, and can also include a corresponding angular velocity value along each axis of the three-dimensional coordinate system.

[0045] In some embodiments, a three-dimensional coordinate system is a coordinate system having three mutually perpendicular axes (i.e., x-axis, y-axis, and z-axis), which is centered on a specific location on or near the robot, for example, at a reference position of the robot or at different fixed locations in the robot's operating environment.

[0046] The one or more velocity values ​​collectively specify a target velocity of a reference point in space defined by a three-dimensional coordinate system. For example, the target velocity of the reference point can include a target 3D angular velocity and a target 3D linear velocity of the reference point. For example, the reference point can be located at a specified position, such as an instantaneous center of mass, relative to a plurality of contact points on a robotic manipulator used to manipulate an object.

[0047] The control input can further specify a respective target speed for each of the one or more components of the robot 110, which can be derived from state or movement information reported by a touch sensor, transducer, or switch coupled to each of the one or more depressible keys or buttons of the input device. For example, the control input can specify a respective target speed for a plurality of contact points of the robot to move away from a reference point.

[0048] The system determines the corresponding joint velocity of each joint in the plurality of joints by generating a solution to an optimization problem formulated from the control input (step 204). Specifically, the optimization problem can be a linear least squares quadratic optimization (LSQP) problem of the form: by constraint in is the set of decision variables, , and They are The weight scalars, coefficient matrices, and bias vectors for the tracking tasks define the objective function of the optimization problem, and , and are the lower bound, upper bound, and coefficient matrix, respectively, which define the constraint set of the optimization problem. The constraint set defines the lower or upper bounds on the possible values ​​of the decision variables.

[0049] Specifically, the set of decision variables can be defined as follows: , Where n is the number of joints (the number of degrees of freedom associated with a joint), is a first decision variable corresponding to respective joint velocities of the plurality of joints; is the second decision variable corresponding to the corresponding velocity of the contact point relative to the reference point; is a third decision variable corresponding to the target linear velocity of the reference point; and is the fourth decision variable corresponding to the target angular velocity of the reference point.

[0050] here, can be represented as an n-dimensional vector, ie a vector with n values ​​defining the n corresponding joint velocities of the joints. Can be expressed as a scalar value that defines the speed at which each contact point moves away from the reference point. can be represented as a three-dimensional vector that defines the 3D target linear velocity with reference to a reference point defined in a given coordinate system. And can be represented as a three-dimensional vector that defines the three-dimensional target angular velocity of a reference point defined with reference to a given coordinate system. Therefore, the above decision variable set can be represented as a vector with n+7 dimensions (hereinafter referred to as a decision variable vector).

[0051] The system generates a solution to the optimization problem by calculating optimized values ​​of the first decision variable that optimize—e.g., minimize—the objective function, subject to the constraint set. In other words, in response to the user specifying a decision variable using an input device, , and The system is able to calculate the joint speed that the robot should track based on the target value of The system can achieve this by, for example, inputting the objective value specified by the user and data describing the constraint set into a least squares solver, for example, by using an application program interface (API) provided by the solver, and in response receiving values ​​of the joint velocities of the robot from the solver as output. In some cases, the solution can be an exact solution that defines a globally optimized value of the first decision variable such that no other feasible values ​​can further optimize the objective function, while in other cases, the solution can be an approximate solution, for example, one of a plurality of suboptimal local solutions.

[0052] In more detail, the target term is of the form: , It is possible to define separate tracking tasks for different elements of the decision variable vector. In the above equation, is the selection matrix from which elements are extracted from the entire decision variable vector; is a weight scalar that defines a measure of the importance of the target term relative to the other target terms; and is the target (that is, user-specified) value for the selected element.

[0053] Specifically, four separate tracking tasks can be defined through the objective function, each task is aimed at , , and Each of them is defined to track the expected value of each quantity. In addition, the angular velocity of each contact point can be defined by the objective function Define an additional task. Once defined, this additional task will allow the user to set the appropriate angular velocity to maintain the contact point pointing towards the reference point: , in It is The target angular velocity of each contact point, and It is for The angular velocity of a contact point can be described by attaching a coordinate system to the contact point and then defining the angular velocity of its coordinate axes relative to the given coordinate system.

[0054] Therefore, the objective function can be defined as: , in , , and They are , , and target value.

[0055] To ensure that the decision variables are physically consistent, a set of constraints is defined as follows. These equality constraints are usually defined as equality constraints, although in some cases they can be relaxed to inequality constraints or tasks to improve numerical stability. Joint velocity and radial velocity relative to a reference point

[0056] In order to and Keep consistent, by The velocity of each contact point relative to the reference point must be the same as the velocity in the direction connecting the reference point and the contact point. This is a 1 DoF constraint that can be added for each contact point of the robot. The relationship is given by the following equation: , in From the reference point to the The instantaneous vector of the contact point, is its associated unit vector, It is The linear velocity of each contact point, and is the scalar product between two vectors. By noting is the Jacobian matrix by Rewriting the equation as related to the joint velocity, the constraint can then be defined as: , It can be achieved by taking and Written in the form of least squares quadratic optimization (LSQP), represents a column vector of length 3 that is zero everywhere. Joint velocity and 3D linear velocity of reference point

[0057] In order to and Keep consistent, by The average of all the resulting contact point velocities must be equal in each direction to matches. This constraint is a 3 DoF constraint. The relationship is given by the following equation: .

[0058] Writing the left-hand side in terms of joint velocities gives: , It can be achieved by taking and Written in LSQP form, where represents a 3x3 matrix with zeros everywhere, and is the identity matrix of size 3x3. Joint velocity and 3D angular velocity at the reference point

[0059] In order to and To be consistent, the contact point velocity must produce an angular velocity relative to the reference point This constraint is a 2 DoF constraint that can be added for each contact point of the robot.

[0060] The relationship between the angular velocity of a rigid body and the tangential linear velocity at a point is given by the following equation: , This equation can be solved by choosing Any two axes that form an orthonormal basis and These axes can be obtained by following the Gram-Schmidt procedure. The expanded equation can take the following form: .

[0061] In order to make the left-hand side vector equal to the right-hand side vector, it is in the orthogonal basis , , Each independent component of must match: , in is a skew-symmetric matrix such that for any vector , .

[0062] This is a 2 DoF constraint because the radial component of the contact point velocity relative to the reference point is not constrained, since is a property of the triple product of vectors, and for , , , Any option, The equality constraints on the components of are always satisfied. In LSQP form, this can be rewritten as two separate 1 DoF constraints: and ; and .

[0063] Thus, the matrix consisting of can be generated by concatenating the values ​​in these matrices along the column dimension (ie, stacking the matrices vertically). The matrix and each contact point , and A coefficient matrix of a component in the matrix The lower bound vector can be generated in a similar manner and the upper limit vector . Collectively, the lower bound, upper bound, and coefficient matrix define the constraint set for the optimization problem.

[0064] The system controls the robot to approximate the obtained control input, which specifies the target speed of the reference point (step 206). Specifically, the system can use the motion controller to generate and send commands to move multiple joints of the robot at the corresponding joint speeds determined from step 204.

[0065] In some embodiments, Figure 1 The robot 110 in the example can be controlled by a machine learning system, which typically includes a machine learning model that has been configured through training, such as reinforcement learning (RL) training, to generate model outputs that can be used to control the robot to perform one or more different tasks.

[0066] Figure 3 An example reinforcement learning system 300 is shown. The reinforcement learning system 300 is an example of a system implemented as a computer program on one or more computers at one or more locations that controls, for example, Figure 1 The robot 110 in the robot is used to enable the robot to perform a given task.

[0067] For example, the tasks can include causing the robot to navigate to different locations in the environment 304, causing the robot to locate different objects, causing the robot to pick up different objects or move different objects to one or more specified locations, etc.

[0068] Reinforcement learning system 300 controls a robot by selecting actions 302 to be performed by the robot while the robot is interacting with environment 304 in response to observations 306 characterizing a state of the environment. The robot typically moves within environment 304 (eg, navigates and / or changes its configuration).

[0069] Observations 306 may include, for example, one or more of the following: images (such as images captured by a camera and / or a Lidar sensor), object position data, and other sensor data from sensors that capture observations as the agent interacts with the environment, such as sensor data from an image, distance or position sensor, or from an actuator. For example, in the case of a robot, the observation may include data characterizing the current state of the robot, such as one or more of the following: joint positions, joint velocities, joint forces, torques or accelerations, such as gravity-compensated torque feedback, and global or relative poses of items held by the robot. In other words, the observation may similarly include one or more of the position, linear or angular velocity, force, torque or acceleration, and global or relative pose of one or more components of the agent. Observations may be defined in 1, 2, or 3 dimensions, and may be absolute and / or relative observations. Observations may also include, for example, sensed electronic signals, such as motor currents or temperature signals; and / or image or video data, such as from a camera or Lidar sensor, such as data from a sensor of the agent or data from a sensor located separately from the agent in the environment.

[0070] Action 302 may be a control input for controlling a robot, such as a torque or higher-level control command of a robot joint, or a control input for controlling an autonomous or semi-autonomous land, air, or sea vehicle, such as a torque or higher-level control command of a control surface or other control element of a vehicle.

[0071] In other words, an action can include, for example, position, velocity, or force / torque / acceleration data of one or more joints of a robot or a component of another mechanical agent. The action data may additionally or alternatively include electronic control data, such as motor control data, or more generally data for controlling one or more electronic devices within the environment, the control of which has an effect on the state of the observed environment.

[0072] Specifically, reinforcement learning system 300 uses control neural network 310 and training engine 340 to select actions 302 to be performed by the robot.

[0073] The control neural network 310 is a neural network that receives as input observations 306 about the state of the environment 304 and generates outputs 312 that can be used to control the robot.

[0074] The control neural network 310 can be implemented using any suitable neural network architecture that enables it to perform the functions described herein. As a specific example, the control neural network 310 can include an "embedding" subnetwork, a "core" subnetwork, and a "selection" subnetwork. A subnetwork of a neural network refers to a group of one or more neural network layers in a neural network. When the observations include images, the embedding subnetwork can be a convolutional subnetwork, i.e., the convolutional subnetwork includes one or more convolutional neural network layers configured to process observations of time steps. When the observations include lower dimensional data, the embedding subnetwork can be a fully connected subnetwork. The core subnetwork can be a recurrent subnetwork, for example, including one or more long short-term memory (LSTM) neural network layers, configured to process the output of the embedding subnetwork and, in some cases, process action data (or data derived from the action data, or both) defining each action in a set of possible actions. The selection subnetwork can be configured to process the output of the core subnetwork to generate an action selection output, which can be used to determine the action to be performed by the robot at each time step in a plurality of time steps. For example, the action selection output can include corresponding Q-value outputs for possible actions, and the system 300 can select actions to be performed by the robot, for example, by sampling actions according to their Q-values ​​(or probability values ​​derived from the Q-values), or by selecting the action with the highest Q-value. For each action, the corresponding Q-value is a prediction of the expected reward that would result from the robot performing the action in response to the observation.

[0075] To allow the robot 110 to effectively interact with the environment 304, the reinforcement learning system 300 includes a training engine 340 that trains the control neural network 310 to determine training values ​​for the parameters 320 of the control neural network 310. To train the control neural network 310, the training engine 340 can use any of the example training techniques described in Ibarz, Julia, et al., “How to train your robot with deep reinforcement learning: lessons we have learned,” International Journal of Robotics Research 40.4-5 (2021): 698-721, in addition to or in lieu of other known reinforcement learning training techniques.

[0076] Because the use of the above motion control techniques can allow a robot to be effectively controlled by any suitable teleoperation device equipped with motion sensors, having fewer degrees of freedom than the actual robot, and without the need for custom and expensive hardware such as tactile gloves, it is now possible to generate and collect a large amount of demonstration data of sufficient quality in a scalable manner. For example, the teleoperation device can be a 3 DoF input device, or a 6 DoF input device, while the actual robot is a dexterous robotic hand with 10, 20 or more DoF provided by the joints of the dexterous robotic hand.

[0077] Demonstration data can be generated by multiple expert demonstrators. For example, expert demonstrators can be learned robot control machine learning models, fixed robot control policies, random robot control policies, human experts, etc. Figure 1 For example, the expert demonstrator can be a remote user of the system 100 that controls the robot, such as by using an application program interface (API) or other interface that may be provided by the system 100. The demonstration data can be used to assist in the training of the control neural network 310 in any of a variety of ways.

[0078] In some embodiments, the training engine 340 maintains a replay memory 330 that stores pieces of experience data generated as a result of the robot 120 (or another robot) interacting with the environment 304 (or with another instance of the environment) for use in training the control neural network 310. The training engine 340 trains the control neural network 310 by repeatedly selecting pieces of experience data from the replay memory 330 and training the control neural network 310 on the selected pieces of experience data and by using an offline reinforcement learning training technique. In these embodiments, some of the experience data can include or otherwise be derived from demonstration data generated by an expert demonstrator controlling the robot (or another robot) to perform one or more specified tasks, which demonstration data need not be the same as the given task for which the control neural network 310 is being trained.

[0079] In some implementations, the demonstration data includes a plurality of demonstration sequences, wherein each demonstration sequence includes a plurality of demonstration observations that characterize a state of an environment while a demonstration robot is interacting with the environment, and the training engine 340 trains the control neural network 310 on the demonstration data by imitation learning. In these implementations, the control neural network 310 can be configured as a "goal-conditioned" control neural network that, at any given time step during training, generates an output 312 by processing not only a current observation 306 that characterizes a current state of the environment 304 (or data derived from the current observation, or both), but also a target observation 308 (or data derived from the target observation, or both) of a demonstration sequence that characterizes a target state of the environment 304.

[0080] This specification uses the term "configured" in relation to systems and computer program components. For a system of one or more computers configured to perform specific operations or actions, it means that the system has installed thereon software, firmware, hardware, or a combination thereof, which, when operated, causes the system to perform those operations or actions. For one or more computer programs configured to perform specific operations or actions, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform those operations or actions.

[0081] The embodiments and functional operations of the subject matter described in this specification can be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more computer program instruction modules, which are encoded on a tangible non-transitory storage medium for execution by a data processing device or for controlling the operation of the data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0082] The term "data processing apparatus" refers to data processing hardware and covers all kinds of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus can also be or further include a dedicated logic circuit system, for example, an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to the hardware, the apparatus can optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0083] A computer program, which may also be referred to or described as a program, software, software application, application, module, software module, script or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program involved, or in multiple coordinated files, such as files storing one or more modules, subroutines or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0084] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured in any particular way, or at all, and can be stored on a storage device in one or more locations. Thus, for example, an index database can include multiple data sets, each of which can be organized and accessed in a different way.

[0085] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.

[0086] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit system such as an FPGA or ASIC, or a combination of a dedicated logic circuit system and one or more programmed computers.

[0087] Computers suitable for executing computer programs can be based on general or special microprocessors or both, or any other kind of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The elements of a computer are a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by or incorporated into a special logic circuit system. Typically, the computer will also include or be operably coupled to receive data from or transmit data to or both of one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk. However, the computer does not need to have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0088] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks.

[0089] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other kinds of devices can also be used to interact with the user; for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. In addition, the computer can interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving a response message from the user in return.

[0090] The data processing apparatus for implementing the machine learning model can also include, for example, dedicated hardware accelerator units for processing the common and computationally intensive part of machine learning training or production, namely the inference workload.

[0091] Ability to implement and deploy machine learning models using a machine learning framework such as the TensorFlow framework, Microsoft Cognitive Toolkit framework, Apache Singa framework, or Apache MXNet framework.

[0092] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer with a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0093] The computing system can include a client and a server. The client and the server are usually remote from each other and typically interact through a communication network. The relationship between the client and the server is generated by means of computer programs running on the respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data such as HTML pages to the user device, for example, for displaying data to a user interacting with the device acting as a client and receiving user input from the user. Data generated at the user device, such as the result of the user interaction, can be received from the device at the server.

[0094] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or the scope that may be claimed, but should be interpreted as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment can also be implemented separately or in any suitable sub-combination in multiple embodiments. In addition, although features may be described as working in certain combinations and even initially claimed as such, in some cases, one or more features can be deleted from the claimed combination, and the claimed combination can be directed to a sub-combination or a variant of a sub-combination.

[0095] Similarly, although operations are depicted in a particular order in the figures and recited in the claims, it should not be understood as requiring that the operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0096] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for controlling a robot manipulator having multiple joints, wherein: The method comprises: obtaining a control input, the control input comprising one or more velocity values ​​specifying a target velocity of a reference point in a given coordinate system; determining a respective joint velocity for each of the plurality of joints by generating a solution to an optimization problem formulated by the control input; and Controlling the robotic manipulator includes causing the plurality of joints of the robotic manipulator to move according to the respective joint velocities to approximate the control input.

2. The method according to claim 1, wherein: The reference point is located at a specified position relative to a plurality of contact points on the robotic manipulator used to manipulate an object.

3. The method according to claim 2, wherein: The reference point is the instantaneous centroid of the plurality of contact points on the robotic manipulator used to manipulate an object.

4. The method according to any one of claims 1 to 3, wherein: The target velocity of the reference point includes a target angular velocity defined with reference to the given coordinate system.

5. The method according to any one of claims 1 to 3, wherein: The target velocity of the reference point includes a target linear velocity defined with reference to the given coordinate system.

6. A method according to any preceding claim when dependent on claim 2 or 3, wherein: The control input further specifies a respective target velocity for each contact point relative to the reference point.

7. The method according to any one of claims 1 to 6, wherein: Generating the solution to the optimization problem comprises: The optimization problem is formulated as a least squares optimization problem, which includes an objective function including terms depending on a set of decision variables, the set of decision variables including (i) a first decision variable corresponding to the respective joint velocities of the plurality of joints, (ii) a second decision variable corresponding to the respective velocities of the contact points relative to the reference point, (iii) a third decision variable corresponding to the target linear velocity of the reference point, and (iv) a fourth decision variable corresponding to the target angular velocity of the reference point, and a set of constraints defining lower or upper bounds on possible values ​​for some or all of the decision variables; and The solution to the optimization problem is generated by calculating an optimized value of the first decision variable that optimizes the objective function, subject to the constraint set.

8. The method according to claim 7, wherein: The least squares optimization problem is of the form: by constraint, where x is the decision variable, , and are the weight matrix, coefficient matrix and bias matrix defining the objective function, respectively, and , and are the lower bound, upper bound and coefficient matrix defining the constraint set, respectively.

9. The method according to any one of claims 1 to 8, wherein: The robotic manipulator is a dexterous robotic hand.

10. The method according to any one of claims 1 to 9, wherein: Obtaining the data specifying the target speed of the object comprises: Remote operation data transmitted from a remote operation device of the robot manipulator is obtained.

11. The method according to claim 10, wherein: The teleoperation device has a motion sensor with a lower degree of freedom (DoF) than the robotic manipulator.

12. The method according to claim 11 further includes using the remote operation device to control the robotic operator to generate training data, wherein the training data is used to train a machine learning model configured to generate control signals, and the control signals are used to control the robotic operator to perform one or more different tasks.

13. The method according to claim 12, wherein: The machine learning model is trained on the training data using reinforcement learning techniques.

14. One or more computer-readable storage media storing instructions which, when executed by one or more computers, cause the one or more computers to perform corresponding operations of the method according to any one of the preceding claims.

15. A system comprising one or more computers and one or more storage devices, the one or more storage devices storing instructions which, when executed by the one or more computers, cause the one or more computers to perform corresponding operations of the method according to any one of the preceding claims.

Images