A control method, device, equipment and medium of a robot manipulator

Abstract

The application provides a robot manipulator control method, device, equipment and medium, and relates to the technical field of manipulator control. The method comprises the following steps: determining the center of mass position of a carried object by using a trained neural network model according to obtained force data of robot hands; determining the grasping point position of the robot hands for grasping the carried object according to the center of mass position based on a hand force-torque balance condition, and controlling the robot hands to grasp the carried object; generating a collision-free translation motion trajectory of the carried object and a collision-free translation motion trajectory of the robot hands based on model predictive control technology and control barrier function technology according to obtained obstacle information; and determining the expected posture of the robot hands when carrying the carried object according to obtained translation acceleration of the robot hands when carrying the carried object. The application comprehensively makes up for the deficiencies of the prior art in the aspects of object grasping, hand posture adjustment and obstacle avoidance through logical connection between steps.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, in particular to a robot manipulator control method, device, equipment and medium. BACKGROUND

[0002] Robots are intelligent equipment that can autonomously or semi-autonomously perform various tasks through programming, integrating perception, decision-making, execution, and other core functions, and can replace humans to complete repetitive, complex, or dangerous operations in various fields such as industry, service, and scientific research. For example, as an important branch of robots, humanoid upper body robots are designed to simulate the structure and function of the human upper body, with a focus on the coordinated operation of the torso, arms, and hands, and are designed for complex upper limb tasks such as grasping, carrying, and assembly. It usually carries a multi-degree-of-freedom manipulator, covering flexible joints such as the shoulder, elbow, and wrist, and a dexterous hand with an embedded force-torque sensor, and some also integrate a waist rotation joint to expand the operation space. In terms of function, it can obtain real-time environmental information such as force, position, and obstacles through sensors, autonomously plan grasping points and motion trajectories with the help of algorithms, and realize double-arm or waist-arm coordinated motion, which can handle complex scenarios such as centroid-unbalanced object carrying and dynamic obstacle avoidance, and balances operation stability, safety, and dexterity, effectively overcoming the limitations of traditional robots in single task execution, and is commonly used in industrial assembly, service assistance, and scientific research.

[0003] In related technologies, in terms of robot manipulator grasping control, existing methods focus on object stability or manipulator operation ability, without considering the influence of object gravity on the manipulator, which can easily lead to damage to the manipulator or dropping of the object; in terms of robot obstacle avoidance control, existing methods use reinforcement learning, imitation learning, etc., but still have problems such as poor universality, unstable motion, or the need to pause the carrying task; in terms of robot posture control, existing fixed posture control methods reduce the dexterity of the robot, and existing variable posture control methods lack autonomy or have insufficient degrees of freedom, making it difficult to adapt to complex carrying scenarios. SUMMARY

[0004] The present application aims to solve at least one of the above problems.

[0005] To solve the above problems, the present application provides a robot manipulator control method, device, equipment and medium.

[0006] In a first aspect, the present application provides a robot manipulator control method, comprising:

[0007] According to the obtained force data of the robot's two hands, a trained neural network model is used to determine the centroid position of the carried object;

[0008] determine a grasp point position of the robot double hands to grasp the carried object according to the center of mass position based on a double hands force-torque balance condition, and control the robot double hands to grasp the carried object;

[0009] generate a collision-free translation motion trajectory of the carried object and a collision-free translation motion trajectory of the robot double hands according to the obtained obstacle information based on a model predictive control technique and a control barrier function technique;

[0010] determine an expected pose of the robot double hands when carrying the carried object according to the obtained translation acceleration of the robot double hands when carrying the carried object.

[0011] Optionally, the determining of the expected pose of the robot double hands when carrying the carried object according to the obtained translation acceleration of the robot double hands when carrying the carried object comprises:

[0012] determine a synthetic acceleration according to the translation acceleration and a preset modified auxiliary acceleration, wherein the translation acceleration is determined according to the collision-free translation motion trajectory of the carried object;

[0013] determine the expected pose of the robot double hands when carrying the carried object according to an expected pose rotation matrix constructed according to the synthetic acceleration.

[0014] Optionally, the generating of the collision-free translation motion trajectory of the carried object and the collision-free translation motion trajectory of the robot double hands according to the obtained obstacle information based on the model predictive control technique and the control barrier function technique comprises:

[0015] construct a discrete point mass kinematics model of the carried object by using a five-level integrator and a second-order forward difference;

[0016] determine a constraint condition according to the obstacle information and the discrete point mass kinematics model based on the model predictive control technique and the control barrier function technique, add a relaxation variable and a regularization term, and construct an MPC-CBF optimization problem;

[0017] solve the MPC-CBF optimization problem to generate the collision-free translation motion trajectory of the carried object, and determine the collision-free translation motion trajectory of the robot double hands according to the collision-free translation motion trajectory of the carried object.

[0018] Optionally, the determining of the grasp point position of the robot double hands to grasp the carried object according to the center of mass position based on the double hands force-torque balance condition comprises:

[0019] establish a force-torque balance relationship of the robot double hands according to the center of mass position.

[0020] According to the force-torque balance relationship, a grasping point optimization problem satisfying the double-hand force-torque balance condition is constructed, and the grasping point optimization problem is solved to obtain the grasping point position.

[0021] Optionally, the control of the robot double hands to grasp the carried object comprises:

[0022] According to the double-hand grasping point, the double-arm reachability data of the robot is determined based on a double-arm reachability condition;

[0023] When the double-arm reachability data is less than a reachability threshold, the robot double hands grasp the carried object through waist-arm cooperative motion control;

[0024] When the double-arm reachability data is greater than or equal to the reachability threshold, the robot double hands grasp the carried object through double-arm cooperative motion control.

[0025] Optionally, after determining the expected pose of the robot double hands when carrying the carried object, it further comprises:

[0026] According to the collision-free translational motion trajectory of the robot double hands and the expected pose of the robot double hands when carrying the carried object, joint speed instructions are generated.

[0027] Optionally, before the determination of the center of mass position of the carried object according to the acquired force data of the robot double hands by using the trained neural network model, it further comprises:

[0028] The robot double hands are controlled to grasp the carried object at an initial position, and a six-dimensional force-torque sensor is used to acquire the force data;

[0029] The robot is controlled to put down the carried object.

[0030] In a second aspect, the present application provides a control device of a robot manipulator, comprising:

[0031] A center of mass module is configured to determine the center of mass position of the carried object according to the acquired force data of the robot double hands by using the trained neural network model;

[0032] A grasping module is configured to determine the grasping point position of the robot double hands to grasp the carried object according to the center of mass position based on a double-hand force-torque balance condition, and control the robot double hands to grasp the carried object;

[0033] The obstacle avoidance module is configured to generate a collision-free translation movement track of the object to be carried and a collision-free translation movement track of the robot hands based on a model prediction control technology and a control obstacle function technology according to the acquired obstacle information.

[0034] The carrying module is configured to determine the expected pose of the robot hands when carrying the object to be carried according to the acquired translation acceleration of the robot hands when carrying the object to be carried.

[0035] In a third aspect, the present application provides an electronic device, comprising a memory and a processor;

[0036] The memory is configured to store a computer program.

[0037] The processor is configured to implement the control method of the robot manipulator as described in the first aspect when executing the computer program.

[0038] In a fourth aspect, the present application provides a computer readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the control method of the robot manipulator as described in the first aspect is implemented.

[0039] The control method of the robot manipulator, the device, the equipment and the medium of the present application have the following beneficial effects:

[0040] By acquiring the force data of the robot's two hands, using the trained neural network model that can inversely deduce the center of mass distribution according to the force data, the center of mass position of the object being carried can be accurately determined, and then based on the double-hand force-torque balance condition for targetedly offsetting the additional force of the object gravity on the mechanical hand, the optimal double-hand grabbing point is calculated and the robot is controlled to re-grab, solving the problem that the prior art does not consider the gravity influence and is prone to cause damage to the mechanical hand or object falling; combined with the obstacle information, the model predictive control technology and the control barrier function technology are used to generate the collision-free translation motion trajectory of the robot's two hands, the multi-step prediction capability of the model predictive control technology can guarantee the continuity of the trajectory, the control barrier function technology can construct the obstacle avoidance constraint in real time, and the combination of the two can cope with the dynamic obstacle speed change without relying on specific task space integration and without suspending the carrying task, effectively overcoming the defects of poor universality, unstable motion or task interruption of the existing obstacle avoidance methods; then the desired pose of the mechanical hand is determined according to the translation acceleration of the robot's two hands when carrying the object, because the translation acceleration directly reflects the motion state change of the object and the two hands, and the adaptive adjustment of the pose of the two hands based on this parameter can maximize the motion freedom of the object and the robot's two hands, breaking through the limitations of the existing fixed robot's two hands pose, which restricts the dexterity of the robot's two arms, and the variable robot's two hands pose lacks autonomy or freedom, so that the robot is more suitable for complex carrying scenarios, and through the logical connection between the steps, the deficiencies of the prior art in the three core links of object grabbing, double-hand pose adjustment and obstacle avoidance are comprehensively made up. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 A flowchart of a control method of a robot's mechanical hand provided by an embodiment of the present application is shown in the figure.

[0042] Figure 2 A structural schematic diagram of a simulation model of a robot grabbing a tray and an object provided by an embodiment of the present application is shown in the figure.

[0043] Figure 3 One of the simplified schematic diagrams of the robot's two hands grabbing the tray plane provided by an embodiment of the present application is shown in the figure.

[0044] Figure 4 The second simplified schematic diagram of the robot's two hands grabbing the tray plane provided by an embodiment of the present application is shown in the figure.

[0045] Figure 5 A structural schematic diagram of a control device of a robot's mechanical hand provided by an embodiment of the present application is shown in the figure.

[0046] Figure 6 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0047] In order to make the above objectives, characteristics and advantages of the present application more apparent, specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms, and should not be interpreted as being limited to the embodiments set forth herein, but rather these embodiments are provided to more thoroughly and completely understand the present application. It should be understood that the drawings and embodiments of the present application are only for illustrative purposes, and are not intended to limit the scope of protection of the present application.

[0048] It should be understood that each of the steps described in the method embodiments of the present application can be performed in different orders, and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0049] The term "comprising" and variations thereof as used herein are open-ended, that is "including, but not limited to"; the term "based on" is "based, at least in part, on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; the term "optional" means "optional in at least some embodiments". Related definitions are given throughout the description. It should be noted that the concepts mentioned in the present application are merely used to distinguish different devices, modules or units, and are not intended to limit the functions performed by these devices, modules or units, or the order or interdependence of these functions.

[0050] It should be noted that the modification of "one" or "multiple" mentioned in the present application is illustrative rather than limiting, and those skilled in the art should understand that, unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0051] The names of the messages or information exchanged between the devices in the embodiments of the present application are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0052] As shown in Figure 1 The control method of the robot manipulator provided by the embodiments of the present application comprises:

[0053] According to the force data of the robot hands obtained, the trained neural network model is used to determine the center of mass position of the object being carried.

[0054] Specifically, during transport, the object to be transported is placed in a tray. The tray includes a tray plane and a handle. The handle can be cylindrical or other shapes that facilitate gripping by the robot's manipulator. The robot grasps the tray handle to transport the object. Before transport, the object is pre-grabbed to obtain force data, and then the object is placed down. In this embodiment, the object to be transported refers to the combination of the object and the tray, i.e., the object-tray whole. The center of mass position refers to the projection of the actual center of mass of the object-tray whole onto the tray plane. A manipulator is connected to the robot's robotic arm, which can grasp the tray handle. Sensors can be used to obtain force data from the robot's hands, for example, two six-dimensional force-torque sensors mounted on the robot's arms. The force data includes the force and torque acting on the robot. Then, a trained neural network model, such as a backpropagation neural network, a multilayer perceptron (MLP), or a radial basis function network (RBFNetwork), is used to analyze the force data to obtain the center of mass position of the object to be transported.

[0055] For example, the training process of a neural network model includes: using a physics simulation engine, such as Vrep, to build a... Figure 2 The robot grasping simulation model shown here refers to the tray handle, the force-torque sensor, the tray flat platform, and the tray coordinate system, denoted as ∑. T Object refers to the physical object, and Force-torque sensor coordinate refers to the force-torque sensor coordinate system, denoted as ∑. F Dexteroushand is the robotic arm of a robot. W Using the world coordinate system, l t,r The straight line on the right edge of the tray plane. l t,l The straight line on the left edge of the tray plane. l h,r The right handle of the tray is in a straight line. l h,l For the straight line of the left handle of the tray, p com Let H and L be the location of the center of mass. α t For the dimensions of the pallet, p1 * and p2 * p represents the virtual gripping points of the robot's two robotic arms on the tray plane. h,1 * and p h,2 *The actual gripping points of the robot's two manipulators are then used. Objects of different masses are randomly placed at different positions on the tray to obtain corresponding force and torque data, i.e., force data, and a training set is constructed. Using the training set, an algorithm for determining the parameters of a neural network is designed using a set of empirical formulas and a centroid prediction error algorithm. The pre-trained neural network model is then trained to determine the number of hidden layers and the number of neurons in each layer. The set of empirical formulas includes:

[0056] ,

[0057] ,

[0058] ,

[0059] ,

[0060] ;

[0061] Where, n j (j∈[1,h]) represents the number of hidden layer nodes, h represents the number of hidden layers, k represents the iteration parameter, and N represents the number of hidden layer nodes. in N represents the number of nodes in the input layer of the neural network. out α represents the number of nodes in the output layer of the neural network, and α, β, and γ are all node count decay coefficients.

[0062] The specific algorithm for determining neural network parameters is as follows:

[0063] Input parameters: Number of neurons in the input layer N in Number of output layer neurons N out Maximum number of hidden layers N max Filtering parameter N avr Training dataset A, validation dataset B, and validation data size N val The training dataset is A, the validation dataset is B, and the amount of validation data is N. val The simulation model is obtained by the robot grasping it; the number of hidden layers is traversed: let the number of hidden layers be... l Cycle from 1 to N max ; Generate combinations of hidden layer node numbers: for each l Let k cycle from 1 to 10, and generate the number of nodes in each layer according to the empirical formula; train the neural network: train the neural network model with the current combination of node numbers using the training dataset A; calculate the validation error: initialize the error and E. sum =0, let k cycle from 1 to N avr Each time, N is selected from the validation dataset B. val Data points are used to obtain predicted values ​​through a trained network, and the mean squared error V between the predicted and actual values ​​is calculated. mse And accumulate to Esum ; record parameters and errors: record current hidden layer node combination and corresponding total error E sum ; screen optimal parameters: sort all combinations of E sum , select the node combination with the smallest total error; output results: determine the optimal number of hidden layers and the number of nodes in each layer.

[0064] Exemplarily, the final number of hidden layers of the neural network is 3, and the number of neurons in each layer is 11, 9, and 7, respectively.

[0065] Based on the double-hand force-torque balance condition, the position of the double-hand grabbing point of the robot for grabbing the object to be carried is determined according to the centroid position, and the robot double hands are controlled to grab the object to be carried.

[0066] Specifically, the double-hand force-torque balance condition means that the force on the robot double hands is balanced, and the undesired overturning torque at the double-hand grabbing point is 0, i.e., the double-hand torque is balanced, so that the double-hand grabbing point of the robot for grabbing the object to be carried is determined according to the centroid position, so that the force and torque of the robot double hands are balanced, so that the robot can more stably grab the object to be carried for carrying.

[0067] Based on the model predictive control technology and the control barrier function technology, the collision-free translation motion trajectory of the object to be carried and the collision-free translation motion trajectory of the robot double hands are generated according to the obtained obstacle information.

[0068] Specifically, model predictive control (MPC) is an advanced control strategy based on rolling optimization, and the core is to predict the state evolution in the future period of time by using the system dynamic model, and to obtain the optimal control sequence by solving the optimization problem online. The control barrier function technology (CBF) is a constraint design method for ensuring system safety, which converts safety requirements such as "no collision" into inequality constraints that can be included in the optimization problem by constructing a differentiable function about the system state and obstacles. The multi-step prediction capability of the model predictive control technology can guarantee the continuity of the trajectory, and the control barrier function technology can construct the obstacle avoidance constraint in real time, and the combination of the two can obtain the collision-free translation motion trajectory of the object to be carried and the collision-free translation motion trajectory of the robot double hands, without relying on specific task space integration, being able to cope with dynamic obstacle speed changes, and without the need to pause the carrying task, effectively overcoming the defects of poor universality, unstable motion, or the need to interrupt the task of existing obstacle avoidance methods.

[0069] According to the obtained translation acceleration of the robot double hands when carrying the object to be carried, the desired pose of the robot double hands when carrying the object to be carried is determined.

[0070] Specifically, during the movement of the robot's double arms carrying the object, it is assumed that the double hands rigidly grasp the tray, and the relative pose between the hands and the tray remains unchanged, then the change of the pose of the tray is consistent with the change of the pose of the double hands. In order to prevent the generation of an unexpected internal force between the double hands and the tray, the relative pose between the double hands remains unchanged. At the same time, if the absolute pose of the robot's double hands in the global coordinate system also remains unchanged, that is, only the translational motion of the double hands is allowed during the carrying process, and the rotational motion is prohibited, which will greatly reduce the dexterity of the robot's double arms. In order to solve this problem, the embodiment determines the expected pose of the robot's double hands carrying the carried object according to the obtained translational acceleration of the object when the robot carries the carried object, that is, the pose of the tray is adaptively changed according to the acceleration of the translational motion of the tray, thereby increasing the dexterity of the robot's double arms.

[0071] In the embodiment, by obtaining the force data of the robot's double hands, using the trained neural network model capable of inversely deducing the center of mass distribution according to the force data, the center of mass position of the carried object can be accurately determined, and then based on the double-hand force-torque balance condition for targetedly offsetting the additional force of the object gravity on the manipulator, the optimal double-hand grasping point is calculated and the robot is controlled to re-grasp, thereby solving the problem that the prior art does not consider the influence of the gravity and is prone to damage the manipulator or cause the object to fall; combined with the obstacle information, the model predictive control technology and the control barrier function technology are used to generate the collision-free translational motion trajectory of the robot's double hands, the multi-step prediction capability of the model predictive control technology can guarantee the continuity of the trajectory, and the control barrier function technology can construct the obstacle avoidance constraint in real time, the combination of the two can cope with dynamic obstacle speed changes without relying on specific task space integration, and the carrying task does not need to be paused, thereby effectively overcoming the defects of the prior art, such as poor universality, unstable motion, or the need to interrupt the task; then the expected pose of the manipulator is determined according to the translational acceleration of the robot's double hands carrying the object, since the translational acceleration directly reflects the change of the motion state of the object and the double hands, and the adaptive adjustment of the pose of the double hands based on the parameter can maximize the motion freedom of the object and the robot's double hands, thereby breaking through the limitations of the prior art, such as the fixed pose of the robot's double hands limiting the dexterity of the robot's double arms, the variable pose of the robot's double hands lacking autonomy or insufficient freedom, and the like, and making the robot more suitable for complex carrying scenarios. Through the logical connection between the steps, the deficiencies of the prior art in the three core links of object grasping, double-hand pose adjustment, and obstacle avoidance are comprehensively made up.

[0072] Optionally, the determining the expected pose of the robot's double hands carrying the carried object according to the obtained translational acceleration of the robot's double hands carrying the carried object comprises:

[0073] determining a synthetic acceleration according to the translational acceleration and a preset modified auxiliary acceleration, wherein the translational acceleration is determined according to the collision-free translational motion trajectory of the carried object;

[0074] Based on the desired posture rotation matrix constructed from the synthetic acceleration, the desired posture of the robot when carrying the object is determined by both hands.

[0075] Specifically, the translational acceleration and the preset corrective auxiliary acceleration are vector-added together to obtain the composite acceleration, i.e., , The composite acceleration of the pallet control point. This refers to the translational acceleration of the object being transported by the robot, i.e., the translational acceleration of the pallet control point. Figure 2 As shown, the pallet control point is located in the pallet center coordinate system ∑ T The origin can also be represented as the acceleration trajectory, i.e. This represents the acceleration trajectory of the tray. Let t be the acceleration of the trajectory of the object being transported, which is the acceleration of the collision-free translational trajectory of the object being transported. The collision-free translational trajectory includes position, velocity, and acceleration. t is time. The preset correction assist acceleration, and = Used to reduce unwanted tray tilt angles θ s This tilt angle is caused by excessive pallet acceleration, where, For design parameters, Let be a three-dimensional real vector space. During the movement of the tray... It is set to a constant value, and its value depends on the maximum acceleration of the pallet's movement. and the maximum permissible pallet tilt angle ,and , , , The maximum value of the composite acceleration. This represents the maximum value of the resultant acceleration in the horizontal direction. This represents the maximum value of the resultant acceleration in the vertical direction. Then, based on the desired attitude rotation matrix constructed from the resultant acceleration, the desired attitude of the robot when carrying the object with both hands is determined. The specific steps are as follows:

[0076] Make the third column component of the rotation matrix Align with the resultant acceleration acting at the tray control point:

[0077] ;

[0078] Next, define a two-dimensional direction vector:

[0079] ;in, is a two-dimensional direction vector, and σ is an angle parameter that takes values satisfying is aligned with the x-axis of the tray center coordinate system ∑ T .

[0080] Then, the first column component of the rotation matrix and the second column component can be calculated:

[0081] ;

[0082] ;

[0083] Finally, we can store the above three components to obtain the desired pose of the tray , and the desired pose of the tray can be obtained by coordinate transformation when the robot hands carry the carried object, wherein the desired pose of the tray is:

[0084] .

[0085] Optionally, the model predictive control technology and the control barrier function technology generate the collision-free translational motion trajectory of the carried object and the collision-free translational motion trajectory of the robot hands according to the obtained obstacle information, including:

[0086] A discrete point mass kinematics model of the carried object is constructed by using a five-level integrator and a second-order forward difference;

[0087] Based on the model predictive control technology and the control barrier function technology, constraint conditions are determined according to the obstacle information and the discrete point mass kinematics model, and a relaxation variable and a regularization term are added to construct an MPC-CBF optimization problem;

[0088] The MPC-CBF optimization problem is solved to generate the collision-free translational motion trajectory of the carried object, and the collision-free translational motion trajectory of the robot hands is determined according to the collision-free translational motion trajectory of the carried object.

[0089] Specifically, in the robot handling scenario, C4 continuity is the fourth-order derivative continuity of the trajectory or function, which means that the position, velocity, acceleration, jerk (third-order derivative) and jounce (fourth-order derivative) of the motion trajectory are all continuous, and it is a high-order smooth continuous motion characteristic. Because the double-hand posture is dynamically bound to the tray acceleration, the joint angular velocity and angular acceleration are related to the jounce and jounce of the tray, respectively. Only the collision-free translational motion trajectory satisfies C4 continuity, which can ensure that the joint motion has no jitter and the force / torque changes smoothly, which not only protects the mechanical arm structure, but also improves the handling accuracy and safety. Therefore, on the one hand, in order to meet the C4 continuity, a five-level integrator and a second-order forward difference are used to construct the discrete point kinematics model of the object being handled:

[0090] ;

[0091] wherein, is the state vector at t+1, is the state vector, and, , and and Similarly, replace x in the matrix with y and z, and x, y, and z are the three-dimensional coordinates of the tray control points in the global coordinate system, is a fifteen-dimensional real number vector space, is a control vector, and the fifth-order derivative of the tray control point is A and B, which are defined as:

[0092] ;

[0093] ;

[0094] wherein, diag is a diagonal extension adapted to three-dimensional space (x, y, z axis) motion, is the control period.

[0095] On the other hand, in order to avoid potential collisions between the robot and the object being handled and the dynamic environment during the movement of the robot double arms handling the object, the model predictive control technology is combined with the control barrier function technology, the constraint conditions are determined according to the barrier information and the discrete point kinematics model, and the relaxation variables and the regularization terms are added to construct the MPC-CBF optimization problem, and then the MPC-CBF optimization problem is solved to generate the collision-free translational motion trajectory, wherein the constraint conditions include:

[0096] ;

[0097] wherein, , and the formula is further abbreviated as:

[0098] ;

[0099] wherein, is the collision detection barrier function, the larger the value is, the farther the distance from the obstacle is, is the system state at time t, is the system control at time t, the symbol means “is defined as”, is the parameter of CBF constraint, (1- ) is the original delay rate, then the relaxable variable is added , is the real number vector space, which allows to relax the original delay rate (1-γ) introduced by the CBF constraint, thus enhancing the feasibility of solving the MPC-CBF optimization problem, at the same time, in order to not affect the obstacle avoidance effect while relaxing the CBF constraint, the regularization term is added to the cost function of the optimization problem, is the weight parameter, so as to minimize the deviation between the relaxed delay rate and the original delay rate, wherein , finally, the MPC-CBF optimization problem is:

[0100] ;

[0101] ;

[0102] ; ; ;

[0103] wherein, the subscript t : t + N -1| t means the time period from time t to t+N-1, is the system control in the time period from t to t+N-1, the subscript t + N | t means the time t+N after the N times from time t, is the system state at time t+N, the subscript t + k | t means the time t+k after the k times from time t, is the system state at time t+k, is the system control at time t+k, n o is the number of obstacles, n cThe number of robot collision detection points, subscript t | t is the t-th time after the t-th time, i.e., the t-th time, is the system state at the t-th time, is the system control amount at the k-th time, is the system state at the k-th time, is the initial condition.

[0104] ;

[0105] ;

[0106] where the terminal cost is

[0107] ; where, is the expected value of the system state, is the actual value of the system state.

[0108] The stage cost is

[0109] ;

[0110] is a regularization term, is a discrete point mass kinematics model, denotes the initial condition, i.e., the actual state of the system at the current time step , , and are the feasible sets of system input, system state and terminal state, respectively, P, Q, R and S are the corresponding positive definite weight matrices, reflects the distance information between the obstacles and the collision detection points on the robot system, which can be constructed in the form of a differentiable quadratic obstacle function as follows:

[0111] ;

[0112] where, and represent the center of the i-th obstacle and the center of the j-th collision detection point, respectively. The position of the robot collision detection point can be obtained by coordinate transformation of the position of the pallet control point p c(t) , r o,i is the obstacle radius, r c,j is the radius of the collision detection point enclosing sphere, and δ sis a safety threshold of collision detection. Finally, the MPC-CBF optimization problem is solved to generate a collision-free translational motion trajectory of the carried object, and a collision-free translational motion trajectory of the robot dual hands is obtained through coordinate transformation according to the collision-free translational motion trajectory of the carried object.

[0113] Optionally, the determination of the grasp point position of the robot dual hands for grasping the carried object according to the center of mass position based on the dual hand force-torque balance condition comprises:

[0114] establishing a force-torque balance relationship of the robot dual hands according to the center of mass position;

[0115] constructing a grasp point optimization problem satisfying the dual hand force-torque balance condition according to the force-torque balance relationship, and solving the grasp point optimization problem to obtain the grasp point position.

[0116] Specifically, a schematic diagram of the robot dual hands grasping a tray plane is as shown in Figure 3 , a grasp point of one of the robot hands is p 1, and a grasp point of the other hand is p 2. p 1and the center of mass position p com The line connecting d 1, P 2and the center of mass position p com is defined as a virtual link d 2, then d 1= p com - p 1, d 2= p 2- p com , the relative position vector between the grasp points can be defined as p r = p 2- p 1, then the force at each grasp point can be represented as:

[0117] ;

[0118] wherein, f 1is the force of the left robot hand, is the projection amplitude of the virtual link d 1on the relative position vector p r , f com is the force acting on the center of mass, and fcom = f 1+ f 2, f 2 represents the force of the robot's right robotic arm. For virtual links d 2. In relative position vectors p r The projection amplitude on, where the projection amplitude The calculation formula is:

[0119] ;

[0120] in, For virtual links d i The transpose of the corresponding vector.

[0121] Torque at the first gripping point and the torque at the second gripping point They are represented as follows:

[0122] ;

[0123] ;

[0124] The effect of the torque exerted by the two hands on each gripping point is considered. Assuming the robot's two-handed gripping of the pallet is a rigid grip, in the pallet coordinate system ∑ T Below, the torques at the two gripping points are equal and related to the position of the center of mass. p com and relative position vector p r Distance between related, It can be defined as:

[0125] ;

[0126] Therefore, the torque at the gripping point with both hands It can be rewritten as This refers to the force-torque balance relationship. Because... f com The component along the normal direction of the tray plane will produce a relationship at the gripping points of both hands. p r Undesirable flipping torques could cause the pallet to fall from the hand or damage the robotic arm; minimizing these torques is crucial. If the above risks can be reduced, then the force-torque balance condition for both hands can be obtained, that is, the forces on the robot's hands are balanced, and the unwanted flipping torque at the gripping points of both hands is 0, that is, the torques of both hands are balanced.

[0127] ;

[0128] ;

[0129] wherein, f e is the dual-hand force error, is the dual-hand torque error, is the vector related to the centroid position and the grasp point position, which determines the torque balance key parameters.

[0130] Although the dual-hand force-torque balance condition can limit the robot to make a stable grasp, in actual operation, the dual-hand grasp position of the robot can only change within a certain range due to actual conditions, for example, limited by the size of the tray, and it is difficult to satisfy the dual-hand force-torque balance condition. Therefore, in order to satisfy the dual-hand force-torque balance condition as much as possible, that is, to minimize the dual-hand force error and the dual-hand torque error, a grasp point optimization problem that satisfies the dual-hand force-torque balance condition can be constructed:

[0131] ;

[0132] ;

[0133] ;

[0134] wherein, α, β, λ are weight coefficients, and are inequality constraint functions, which limit the feasible grasp points of the robot hands to the straight lines l 1and l 2along the edge of the tray, is an additional cost, that is, an objective function, which can make the distance between the dual-hand grasp points as small as possible, so as to select an optimal grasp point position from a plurality of groups of grasp points that satisfy the dual-hand force-torque balance condition, is represented as:

[0135] ;

[0136] wherein, p t，c is the tray center position, d th is a preset distance threshold for judging whether the centroid deviates from the tray center, d min is the minimum allowable distance between the dual-hand grasp points, which is used to ensure that the distance between the dual-hand grasp points is minimized and improve the dexterity of the robot operation. Then, according to the solution of the grasp point optimization problem, the virtual dual-hand grasp points Figure 4 and ​Since the virtual two-handed gripping point is located on a simplified diagram, it needs to be converted into the actual gripping point on the tray handle, i.e., the two-handed gripping point, using a conversion formula. The conversion formula is as follows:

[0137] ;

[0138] in, For one of the virtual crawl points, For another virtual crawl point, As one of the actual crawling points, For another actual crawl point, GraspTransform is the transformation function. H, L and α t These are all pallet size parameters, such as Figure 2 As shown, H The distance from the center point of the central axis of the cylindrical handle of the pallet to the pallet plane. L The vertical projection of the center point of the central axis of the pallet cylindrical handle onto the pallet plane is a straight line from the edge of the pallet. Length, α t Let be the angle between the central axis of the cylindrical handle of the pallet and the plane of the pallet. Furthermore, in the pallet coordinate system ∑ T The following represents the gripping points of both hands, ∑ i The pose can be represented as:

[0139] ;

[0140] in, Let the i-th grab point be in the tray coordinate system ∑ T The homogeneous transformation matrix (4x4 matrix) contains position and orientation information. R i Let be the attitude matrix of the reference coordinate system at the i-th grasping point. This refers to the location of the capture point.

[0141] Optionally, controlling the robot to grasp the object being transported with both hands includes:

[0142] Based on the gripping points of both hands, and considering the reachability conditions of both arms, the reachability data of the robot's two arms is determined;

[0143] When the reachability data of the two arms is less than the reachability threshold, the robot's hands are controlled to grasp the object being transported through coordinated waist and arm movements.

[0144] When the reachability data of the two arms is greater than or equal to the reachability threshold, the robot's two arms are controlled to grasp the object being transported through coordinated movement of the two arms.

[0145] Specifically, after obtaining the gripping points of both hands, the robot's hands re-grasp the tray handles at these points. However, because there are differences between the gripping points and the robot's hand positions, and these differences vary with each transport operation, the robot's hands need to employ different motion strategies to reach the gripping points. Therefore, the robot's two-arm reachability data is determined through the two-arm reachability conditions, and the final motion strategy, such as waist-arm coordinated movement or two-arm coordinated movement, is determined based on this data. The two-arm reachability data can be obtained by querying an online reachability mapping map; the formula is as follows:

[0146] ;

[0147] in, , S i For two-arm accessibility scoring, i.e., two-arm accessibility data, For the first i Coordinate transformation of the coordinate system of a robotic arm relative to the global coordinate system. Let be the pose of the i-th real capture point relative to the global coordinate system (homogeneous transformation matrix, containing position and pose information). This is the pose transformation matrix (with known preset parameters) of the pallet coordinate system relative to the global coordinate system, responsible for transforming the pose from the local coordinate system of the pallet to the global coordinate system. Let be the pose (homogeneous transformation matrix) of the i-th real grasping point in the pallet coordinate system, and ReachabilityMap be the dual-arm reachability mapping function. When the dual-arm reachability data is less than the reachability threshold, it indicates that the robot's dual arms have very low reachability at the current waist joint angle. This may be because the pallet is far from the robot and the desired grasping point is on the edge of the pallet away from the robot. Therefore, waist-arm coordinated motion is needed to control the robot to move to the grasping point, as waist joint movement can expand the workspace of the arms. When the dual-arm reachability data is greater than or equal to the reachability threshold, dual-arm coordinated motion is sufficient to control the robot's hands to move to the grasping point to grasp the object being transported, while the waist joint angle remains at its current value. Based on the dual-arm reachability condition, the robot can prioritize using dual-arm coordinated motion. Only when dual-arm reachability is very poor will the robot utilize both waist and arm movements simultaneously. This approach improves the stability of the two-handed operation and minimizes robot energy consumption.

[0148] Optionally, after determining the desired posture of the robot when carrying the object with both hands, the method further includes:

[0149] Based on the collision-free translational trajectory of the robot's hands and the desired posture of the robot's hands when carrying the object, joint speed commands are generated.

[0150] Specifically, the specific solving process of the above-mentioned collision-free translational motion trajectory of the robot double hands and the expected pose of the robot double hands when carrying the carried object is as follows:

[0151] The position and pose of the tray control point in the global coordinate system represented by the homogeneous transformation matrix are:

[0152] ; wherein, is the collision-free translational motion trajectory of the carried object, which can obtain the collision-free translational motion trajectory of the robot double hands after coordinate conversion, is the expected pose of the tray, i.e. the expected pose of the carried object when the robot double hands carry the carried object, which can obtain the expected pose of the robot double hands when carrying the carried object after coordinate conversion.

[0153] Further, the double-hand pose represented by the homogeneous transformation matrix relative to the double-arm coordinate system can be represented as:

[0154] ; wherein, is the transformation matrix of the double-arm coordinate system relative to the global coordinate system, i=1 corresponds to the left arm, and i=2 corresponds to the right arm. c is the pose of the tray control point represented by the homogeneous transformation matrix, i.e. , is the pose matrix of the i-th real grasp point in the tray coordinate system, i=1 corresponds to the left-hand grasp point, and i=2 corresponds to the right-hand grasp point. includes the collision-free translational motion trajectory of the robot double hands and the expected pose of the robot double hands when carrying the carried object.

[0155] The double-hand pose represented by the homogeneous transformation matrix can also be converted into a compact form:

[0156] ;

[0157] is represented as , the collision-free translational motion trajectory of the robot double hands is represented as , and the expected pose of the robot double hands when carrying the carried object is represented by Euler angles , and , is a six-dimensional real vector space, is a three-dimensional real vector space.

[0158] ​According to the desired pose of the robot double hands and the collision-free translational motion trajectory of the robot double hands, a joint velocity command formula is adopted to generate joint velocity, and the joint velocity command formula is obtained as follows:

[0159]

[0160] wherein, is the joint velocity, and , is a fourteen-dimensional real vector space, is a twelve-dimensional real vector space, , is the translational velocity, and is obtained by derivative with respect to time, is the angular velocity, and the angular velocity can be obtained by Euler angles derivative with respect to time, and is obtained through matrix transformation, that is, , is a transformation matrix. J1 and J2 represent the Jacobian matrix of one of the mechanical arms and the other mechanical arm, respectively, and J m is the joint Jacobian matrix of the double arms.

[0161] Optionally, before the step of acquiring the force data of the robot double hands and determining the center of mass position of the object to be carried by using the trained neural network model, the method further comprises the steps of:

[0162] controlling the robot double hands to grasp the object to be carried at an initial position, and acquiring the force data by using a six-dimensional force-torque sensor;

[0163] controlling the robot to put down the object to be carried.

[0164] Specifically, when acquiring the force data of the robot double hands, the object to be carried is pre-grasped, that is, the robot double hands first grasp the tray at the initial position, that is, the center position of the tray handle, and then carry the object to be carried vertically upward for a distance, so that the object to be carried is separated from the support plane. At this time, the robot double hands are subjected to the force and torque caused by the gravity of the object to be carried, and the force and torque, that is, the force data, are acquired by using a six-dimensional force-torque sensor. Then, the object to be carried is put down.

[0165] Exemplarily, some parameters are set according to actual conditions, for example, the preset modified auxiliary acceleration is set as [0, 0, 10] T , and the parameters of model predictive control (MPC) are as follows: N = 19, Q = diag(10 5 I 5, 10 5 ​I 5, 10 5 I 5), P = diag(10 5 I 5, 10 5 I 5, 10 5 I 5), R = 10 5 I 3, S = 10 4 I 3, = 10, the parameter gamma of the control barrier function technology (CBF) is set to 0.4, the radius of the collision detection point surrounding sphere is set to 10mm, the obstacle radius r c,j is set to 50mm, the safety distance r o,i is set to 10mm. δ s is set to 10mm.

[0166] As Figure 5 shown, the control device of the robot manipulator provided by the embodiment of the application comprises:

[0167] a centroid module, configured to determine the centroid position of the object to be carried according to the force data of the robot double hands obtained by using the trained neural network model;

[0168] a grabbing module, configured to determine the grabbing point position of the object to be carried grabbed by the robot double hands according to the centroid position based on the double hand force-torque balance condition, and control the robot double hands to grab the object to be carried;

[0169] an obstacle avoidance module, configured to generate the collision-free translational motion trajectory of the object to be carried and the collision-free translational motion trajectory of the robot double hands according to the obstacle information obtained based on the model predictive control technology and the control barrier function technology;

[0170] a carrying module, configured to determine the expected posture of the robot double hands when carrying the object to be carried according to the translational acceleration of the robot double hands when carrying the object to be carried.

[0171] As Figure 6 shown, the electronic device 600 provided by the embodiment of the application comprises a memory 610 and a processor 620; the memory 610 is configured to store a computer program; the processor 620 is configured to implement the control method of the robot manipulator as described above when the computer program is executed.

[0172] Or, an electronic device 600, comprising a memory 610 and a processor 620 coupled to the memory 610; the memory 610 is configured to store a computer program; the processor 620 is configured to execute the following operations when executing the computer program:

[0173] According to the acquired force data of the robot double hands, a trained neural network model is used to determine the centroid position of the carried object;

[0174] Based on the double-hand force-torque balance condition, the centroid position is used to determine the grasping point position of the robot double hands grasping the carried object, and the robot double hands are controlled to grasp the carried object;

[0175] Based on the model predictive control technology and the control barrier function technology, the acquired obstacle information is used to generate the collision-free translation motion trajectory of the carried object and the collision-free translation motion trajectory of the robot double hands;

[0176] According to the acquired translation acceleration of the robot double hands carrying the carried object, the expected posture of the robot double hands carrying the carried object is determined.

[0177] The embodiment of the application provides a computer readable storage medium, and the storage medium stores a computer program.

[0178] Or, a non-volatile computer readable storage medium, the storage medium stores a computer program, when the computer program is executed by a processor, the processor executes the following operations:

[0179] According to the acquired force data of the robot double hands, a trained neural network model is used to determine the centroid position of the carried object;

[0180] Based on the double-hand force-torque balance condition, the centroid position is used to determine the grasping point position of the robot double hands grasping the carried object, and the robot double hands are controlled to grasp the carried object;

[0181] Based on the model predictive control technology and the control barrier function technology, the acquired obstacle information is used to generate the collision-free translation motion trajectory of the carried object and the collision-free translation motion trajectory of the robot double hands;

[0182] According to the acquired translation acceleration of the robot double hands carrying the carried object, the expected posture of the robot double hands carrying the carried object is determined.

[0183] An electronic device 600, which can be a server or a client of the present application, will now be described, which is an example of a hardware device that can be applied to aspects of the present application. The electronic device 600 is intended to represent various forms of digital electronic computer devices such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device 600 can also represent various forms of mobile devices such as personal digital assistants, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0184] The electronic device 600 includes a computing unit that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) or a computer program loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The computing unit, the ROM, and the RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0185] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware by a computer program, and the program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM), or a random access memory (RAM), etc. In this application, the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application. In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0186] Although the present application is disclosed as above, the protection scope of the present application is not limited to this. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application, and these changes and modifications will fall within the protection scope of the present application.

Claims

1. A control method for a robotic manipulator, characterized in that, include: Based on the force data obtained from the robot's two hands, a trained neural network model is used to determine the center of mass of the object being moved. Based on the force-torque balance condition of both hands, and according to the position of the center of mass, the gripping point position of the robot's two hands to grasp the object being transported is determined, and the robot's two hands are controlled to grasp the object being transported. Based on model predictive control technology and obstacle control function technology, the collision-free translational motion trajectory of the object being transported and the collision-free translational motion trajectory of the robot's hands are generated according to the acquired obstacle information. Based on the translational acceleration obtained when the robot uses both hands to carry the object, the desired posture of the robot using both hands to carry the object is determined. The model predictive control technology and obstacle function control technology generate, based on the acquired obstacle information, the collision-free translational trajectory of the object being transported and the collision-free translational trajectory of the robot's hands, including: A discrete-point mass kinematic model of the transported object is constructed using a five-level integrator and a second-order forward difference. Based on the model predictive control technology and the control obstacle function technology, constraints are determined according to the obstacle information and the discrete point mass kinematics model, and slack variables and regularization terms are added to construct the MPC-CBF optimization problem. Solve the MPC-CBF optimization problem to generate the collision-free translational trajectory of the object being transported, and determine the collision-free translational trajectory of the robot's hands based on the collision-free translational trajectory of the object being transported. The control of the robot to grasp the object being transported by both hands includes: Based on the gripping points of both hands, and considering the reachability conditions of both arms, the reachability data of the robot's two arms is determined; When the reachability data of the two arms is less than the reachability threshold, the robot's hands are controlled to grasp the object being transported through coordinated waist and arm movements. When the reachability data of the two arms is greater than or equal to the reachability threshold, the robot's two arms are controlled to grasp the object being transported through coordinated movement of the two arms.

2. The control method for the robotic manipulator according to claim 1, characterized in that, The step of determining the desired posture of the robot when carrying the object by its two hands, based on the obtained translational acceleration, includes: The composite acceleration is determined based on the translational acceleration and the preset correction auxiliary acceleration, wherein the translational acceleration is determined based on the collision-free translational trajectory of the transported object. Based on the desired posture rotation matrix constructed from the synthetic acceleration, the desired posture of the robot when carrying the object is determined by both hands.

3. The control method for the robotic manipulator according to claim 1, characterized in that, The method of determining the gripping point position for the robot to grasp the object by both hands based on the force-torque balance condition of both hands and the position of the center of mass includes: Based on the position of the center of mass, establish the force-torque balance relationship of the robot's two hands; Based on the force-torque balance relationship, a grasping point optimization problem is constructed that satisfies the force-torque balance condition of both hands. The grasping point optimization problem is solved to obtain the position of the grasping point.

4. The control method for the robotic manipulator according to claim 1, characterized in that, After determining the desired posture of the robot when carrying the object with both hands, the method further includes: Based on the collision-free translational trajectory of the robot's hands and the desired posture of the robot's hands when carrying the object, joint speed commands are generated.

5. The control method for a robotic manipulator according to claim 1, characterized in that, Before determining the center of mass of the object being transported using a trained neural network model based on the acquired force data from the robot's two hands, the process further includes: The robot's hands are controlled to grasp the object to be moved from the initial position, and the force data is acquired using a six-dimensional force-torque sensor. Control the robot to lower the object being transported.

6. A control device for a robotic manipulator, characterized in that, include: The center of mass module is used to determine the position of the center of mass of the object being transported by using a trained neural network model based on the force data acquired from the robot's two hands. The grasping module is used to determine the grasping point position of the robot's two hands to grasp the object being transported based on the force-torque balance condition of both hands and the position of the center of mass, and to control the robot's two hands to grasp the object being transported. The obstacle avoidance module is used to generate the collision-free translational motion trajectory of the object being transported and the collision-free translational motion trajectory of the robot's hands based on model predictive control technology and obstacle control function technology, according to the acquired obstacle information. The transport module is used to determine the desired posture of the robot's two hands when transporting the object based on the acquired translational acceleration of the robot's two hands when transporting the object. The model predictive control technology and obstacle function control technology generate, based on the acquired obstacle information, the collision-free translational trajectory of the object being transported and the collision-free translational trajectory of the robot's hands, including: A discrete-point mass kinematic model of the transported object is constructed using a five-level integrator and a second-order forward difference. Based on the model predictive control technology and the control obstacle function technology, constraints are determined according to the obstacle information and the discrete point mass kinematics model, and slack variables and regularization terms are added to construct the MPC-CBF optimization problem. Solve the MPC-CBF optimization problem to generate the collision-free translational trajectory of the object being transported, and determine the collision-free translational trajectory of the robot's hands based on the collision-free translational trajectory of the object being transported. The control of the robot to grasp the object being transported by both hands includes: Based on the gripping points of both hands, and considering the reachability conditions of both arms, the reachability data of the robot's two arms is determined; When the reachability data of the two arms is less than the reachability threshold, the robot's hands are controlled to grasp the object being transported through coordinated waist and arm movements. When the reachability data of the two arms is greater than or equal to the reachability threshold, the robot's two arms are controlled to grasp the object being transported through coordinated movement of the two arms.

7. An electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement the control method for the robotic manipulator as described in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the control method for the robotic arm as described in any one of claims 1 to 5.

Images