Multi-fingered dexterous hand generalization object operation method and system based on model prediction control

Abstract

The invention provides a multi-fingered dexterous hand generalization object operation method and system based on model prediction control, relates to the technical field of dexterous hand control, and aims to control a dexterous hand to operate an object to move stably. The method comprises the steps that sequence planning is conducted according to expected movement of a target object based on a quasi-dynamic dynamics model of the dexterous hand and the target object, a reference sequence is obtained, and the quasi-dynamic dynamics model has a smooth contact dynamics gradient; the reference information of each time step in the reference sequence is corrected according to a contact dynamical model to obtain corrected reference information, the contact force in the contact dynamical model changes along with the change of the contact distance between the objects, and the corrected reference information comprises the expected joint speed and the expected joint position of the dexterous hand; and controlling the dexterous hand according to the correction reference information so as to operate the target object to realize the expected movement.

Description

Technical Field

[0001] The present application relates to the technical field of dexterous hand control, and in particular to a method and system for generalizing object manipulation by a multi-finger dexterous hand based on model predictive control. Background Art

[0002] In-hand manipulation tasks for dexterous hands refer to manipulating objects using only finger or palm contact without the need for re-grasping. In-hand manipulation with multi-finger dexterous hands is a crucial skill for humanoid robots and is widely used in various scenarios, such as home service and the operation of various tools and machines on production lines. Such a skill is mechanistically based on the problem of multiple contacts, where the robot (i.e., the dexterous hand) needs to interact with objects by constantly making and breaking contact.

[0003] Since the contact mechanism is not smooth, the dynamics are difficult to solve. Problems involving contact dynamics usually have highly non-smooth loss functions, which makes the control of multi-contact operation problems challenging. In the related art, two types of control methods, learning-based and model-based control, are used to address these challenges. Although learning-based methods show amazing robustness, thanks to domain randomization and large-scale training, their low data efficiency hinders further deployment and generalization. Model-based control methods can be divided into two categories. The first type of method uses an explicit contact model. This method can achieve robust operation performance through a simple controller on hardware, but problems may occur when processing long sequence tasks. For example, the actual contact point is inconsistent with the planning result and needs to be re-planned, but the planning process is usually time-consuming; the second type of method uses an implicit contact model. This method does not require a predefined contact sequence and can calculate potential contacts online, but the implicit modeling method always sacrifices the fidelity of the model for computational efficiency, so it has poor stability when processing long sequence tasks. For example, it cannot guarantee that the expected contact is executed.

[0004] Therefore, how to control the dexterous hand to manipulate the stable movement of objects is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of the above problems, embodiments of the present application provide a method and system for generalizing object manipulation using a multi-finger dexterous hand based on model predictive control, so as to overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect of an embodiment of the present application, a method for generalizing object manipulation by a multi-finger dexterous hand based on model predictive control is disclosed, the method comprising: Based on the quasi-dynamic dynamics model of the dexterous hand and the target object, sequence planning is performed according to the expected movement of the target object to obtain a reference sequence, wherein the reference sequence includes reference information of multiple time steps, and the reference information of each time step includes the joint position of the dexterous hand, the expected joint position of the dexterous hand, and the contact force between the dexterous hand and the target object, and the quasi-dynamic dynamics model has a smooth contact dynamics gradient; Correcting the reference information of each time step in the reference sequence according to the flexible contact model to obtain corrected reference information, wherein the contact force in the flexible contact model changes with the change of the contact distance between the objects, and the corrected reference information includes the expected joint velocity and the expected joint position of the dexterous hand; The dexterous hand is controlled according to the corrected reference information to manipulate the target object to achieve the desired movement.

[0007] Optionally, based on the quasi-dynamic dynamics model of the dexterous hand and the target object, sequence planning is performed according to the expected movement of the target object to obtain a reference sequence, including: With the goal of minimizing the difference between the position of the target object and the expected position of the target object, and the difference between the joint positions of the dexterous hand and the expected joint positions of the dexterous hand, a sequence planning optimal control model is constructed according to the quasi-dynamic dynamics model and the expected motion of the target object; The sequence planning optimal control model is solved to obtain a reference sequence.

[0008] Optionally, with the goal of minimizing the difference between the position of the target object and the expected position of the target object, and the difference between the joint positions of the dexterous hand and the expected joint positions of the dexterous hand, a sequence planning optimal control model is constructed according to the quasi-dynamic dynamics model and the expected motion of the target object, including: Taking the desired joint position of the dexterous hand as the first control input, based on the quasi-dynamic dynamics model, determining a first constraint condition, wherein the first constraint condition characterizes the dynamic relationship between a first state variable and the first control input, wherein the first state variable includes the joint position of the dexterous hand and the position of the target object; determining a second constraint, the second constraint characterizing a range of the first control input; constructing a first objective function according to the difference between the position of the target object and the expected position, and the difference between the joint position of the dexterous hand and the expected joint position of the dexterous hand; A sequence programming optimal control model is obtained according to the first constraint condition, the second constraint condition, and the first objective function.

[0009] Optionally, the quasi-dynamic dynamics model is a model based on implicit dynamics of convex problem optimization and conforms to quasi-dynamic assumptions, and the quasi-dynamic dynamics model relaxes the nonlinear contact mechanism into a cone complementarity problem.

[0010] Optionally, the reference information of each time step in the reference sequence is corrected according to the flexible contact model to obtain corrected reference information, including: Determine the tracking target according to the reference information of each time step, wherein the tracking target includes the reference joint position of the dexterous hand, the reference contact force between the target objects, and the reference joint velocity of the dexterous hand; A sequence correction optimal control model is constructed according to the flexible contact model and the tracking target with the goal of minimizing the difference between the joint position of the dexterous hand and the reference joint position of the dexterous hand, the difference between the contact force between the dexterous hand and the target object and the reference contact force between the dexterous hand and the target object, and the difference between the expected joint velocity of the dexterous hand and the reference joint velocity of the dexterous hand; The sequence modified optimal control model is solved to obtain modified reference information.

[0011] Optionally, with the goal of minimizing the difference between the joint position of the dexterous hand and the reference joint position of the dexterous hand, the difference between the contact force between the dexterous hand and the target object and the reference contact force between the dexterous hand and the target object, and the difference between the expected joint velocity of the dexterous hand and the reference joint velocity of the dexterous hand, a sequence correction optimal control model is constructed according to the flexible contact model and the tracking target, including: A spring-damper system including a target object, a contact point and a dexterous hand is constructed according to a flexible contact model and a joint impedance model, wherein the spring-damper system describes the relationship between the joint angle and the contact force of the dexterous hand through flexible contact; Taking the desired joint velocity of the dexterous hand as the second control input, determining a third constraint condition according to the spring damping system, wherein the third constraint condition characterizes the dynamic relationship between the second state variable and the second control input, wherein the second state variable includes information about the joint position of the dexterous hand, the desired joint position of the dexterous hand, and the contact force between the dexterous hand and the target object; constructing a second objective function according to the difference between the joint position of the dexterous hand and the reference joint position of the dexterous hand, the difference between the contact force between the dexterous hand and the target object and the reference contact force between the dexterous hand and the target object, and the difference between the expected joint velocity of the dexterous hand and the reference joint velocity of the dexterous hand; According to the third constraint condition and the second objective function, a sequence corrected optimal control model is obtained.

[0012] Optionally, controlling the dexterous hand according to the corrected reference information includes: converting the corrected reference information into joint torque; The dexterous hand is controlled in a closed loop based on the joint torque.

[0013] In a second aspect of the embodiments of the present application, a generalizable object operating system for a multi-finger dexterous hand based on model predictive control is disclosed, the system comprising: A reference sequence planner is used for performing sequence planning according to the expected motion of the target object based on the quasi-dynamic dynamics model of the dexterous hand and the target object to obtain a reference sequence, wherein the reference sequence includes reference information of multiple time steps, and the reference information of each time step includes the joint position of the dexterous hand, the expected joint position of the dexterous hand, and the contact force between the dexterous hand and the target object, and the quasi-dynamic dynamics model has a smooth contact dynamics gradient; A contact planner, for correcting the reference information of each time step in the reference sequence according to a flexible contact model to obtain corrected reference information, wherein the contact force in the flexible contact model changes with the change of the contact distance between the objects, and the corrected reference information includes the desired joint velocity and the desired joint position of the dexterous hand; An impedance controller is used to control the dexterous hand according to the corrected reference information to manipulate the target object to achieve the desired movement.

[0014] According to a third aspect of an embodiment of the present application, an electronic device is disclosed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method for generalizing object manipulation of a multi-finger dexterous hand based on model predictive control described in the first aspect of the embodiment of the present application are implemented.

[0015] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is disclosed, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for generalizing object manipulation of a multi-finger dexterous hand based on model predictive control described in the first aspect of the embodiment of the present application are implemented.

[0016] In a fifth aspect of an embodiment of the present application, a computer program product is disclosed, including a computer program, which, when executed by a processor, implements the steps of the method for generalizing object manipulation of a multi-finger dexterous hand based on model predictive control as described in the first aspect of an embodiment of the present application.

[0017] The embodiments of the present application include the following advantages: In the embodiment of the present application, the expected motion of the target object is used as input, and the reference sequence planning is performed according to the quasi-dynamic dynamics model of the dexterous hand and the target object. Since the quasi-dynamic dynamics model has a smooth contact dynamics gradient, it allows the information about the expected motion to be transmitted to the dexterous hand joints via contact, and can implicitly allow switching between different contact modes, so a reference sequence containing contact information is planned; and the reference information of each time step in the reference sequence is corrected based on the flexible contact model. Since the contact force in the flexible contact model changes with the change of the contact distance between the objects, local corrections can be made near the reference sequence to ensure that the planned contact force is accurately executed. Compared with the method of explicit modeling contact, this method does not require a predefined contact sequence or an offline planning process. Compared with the method of implicit modeling contact, it has stronger robustness because it can compensate for the errors existing in the inaccurate dynamics model. Therefore, this method can stably perform long-time in-hand operations in actual operations; in addition, this method only requires a simple initial value (i.e., the expected motion of the target object) and can be quickly generalized to objects of different shapes without any pre-training, which has significant advantages over the learning-based method. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 It is a flowchart of the steps of a method for generalizing object manipulation by a multi-finger dexterous hand based on model predictive control provided in an embodiment of the present application; Figure 2 is a schematic diagram of modeling a spring damping system provided in an embodiment of the present application; Figure 3 This is an overall architecture diagram of a generalized object manipulation method for a multi-finger dexterous hand based on model predictive control provided in an embodiment of the present application; Figure 4 It is a schematic diagram of the structure of a generalizable object operating system for a multi-finger dexterous hand based on model predictive control provided in an embodiment of the present application; Figure 5 This is a control architecture diagram of a multi-finger dexterous hand generalizable object operating system based on model predictive control provided in an embodiment of the present application; Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] In-hand manipulation of multi-fingered dexterous hands is a crucial skill for humanoid robots and is widely used in various scenarios. However, the dynamics are difficult to solve due to the non-smooth contact mechanisms. Problems involving contact dynamics usually have highly non-smooth loss functions, which makes the control of multi-contact manipulation problems challenging. Although a lot of effort has been made and significant success has been achieved in the field of legged robot locomotion (which also involves multi-contact problems), the robot manipulation problem is different. For example, the robot must actively explore contacts instead of relying on gravity to generate contacts. In addition, the number and location of contact points change over time, resulting in a system model that is much more complex than that of a legged robot.

[0022] In the related art, two types of methods, learning-based and model-based control, are used to achieve the in-hand operation task of dexterous hands. Learning-based (reinforcement learning or imitation learning) methods: They have shown excellent performance and stability in dexterous in-hand operation. The success of reinforcement learning methods (in most cases, this success is called "experimental") can be attributed to the inherent randomness contained in its Markov decision process, the large amount of domain randomization done during training, the flexible selection of perception modalities and large-scale training, and the use of experience. In contrast, imitation learning methods require less exploration of the action space, but are highly dependent on expert teaching data. Some methods use a paradigm that combines model learning with model-based control, or other explorations combine learning-based and non-learning methods. Nevertheless, learning-based methods mainly rely on data to understand the inherent mechanism of in-hand operation, and do not combine theory / mechanism to understand the commonality of multi-contact problems such as in-hand operation. Therefore, these methods may perform poorly when faced with objects not included in the training set or when there is a significant gap between simulation and reality.

[0023] Model-based control methods can be divided into two categories. Model-based algorithms provide deployment and generalization solutions without training. The first category of methods uses explicit contact models: In the in-hand operation of multi-fingered dexterous hands, this category of methods is characterized by: planning and control usually involve explicitly represented contacts, including the location of contacts, the mode of contact points, and the contact forces. This information about contacts constitutes the contact sequence. Existing works use different methods to obtain these discrete sequences. After obtaining the contact sequence, the algorithm is simplified to solving continuous control inputs such as joint torques under the premise of following the predefined contact sequence. Therefore, the underlying control usually involves solving optimization problems with force closure and constraining contact points to avoid sliding. Due to the predefined contact sequence, the actual optimization problem to be processed is usually much simpler than the original problem. Although these methods are easier to implement in hardware and the control effect is more accurate, the process of obtaining the contact sequence becomes the bottleneck of the algorithm and the link with the highest time cost. Online re-planning often significantly reduces the real-time performance of the algorithm. In addition, if the contact state deviates from the predefined contact sequence due to interference, for the underlying control, still using the predefined contact sequence as a constraint will cause significant errors.

[0024] The second category of methods uses implicit contact models: by using relaxed complementary constraints, smoothed contact models, or massively parallel sampling of control variables directly in the object engine, the original, accurate contact dynamics are avoided. A specific approach is an optimization algorithm framework based on operator splitting, which decouples the variables between the time steps of the original control problem and accelerates the otherwise intractable linear complementarity problem (LCP). Smoothing the contact model is considered to be critical for solving multiple contact problems, allowing the system to efficiently explore a variety of possible contact modes. Related studies have shown that randomized smoothing through random sampling of states and control inputs and analytic smoothing through barrier function methods can produce similar results for planning problems for dexterous manipulation. Another way to smooth the contact dynamics is through a flexible contact model. Another specific approach is to propose a variant of this model and perform gradient propagation through inverse dynamics and contact dynamics. Although such methods of implicit contact modeling improve the efficiency of online solutions by eliminating the reliance on a specific contact sequence, the smoothing of the dynamics (effectively a relaxation or approximation of the original dynamics) introduces the side effect of "force-at-a-distance" (which means that "virtual" contact forces are calculated when the contact distance between objects is non-zero). Therefore, these methods often fail due to small differences between the actual contact modes and the planned contact modes.

[0025] In summary, although learning-based methods show amazing robustness, thanks to domain randomization and large-scale training, their data inefficiency hinders further deployment and generalization. In contrast, model-based control methods provide plug-and-play solutions. The first type of methods adopts explicit contact models to control the robot to make contact at predefined contact points and follow a predefined contact sequence to make and break contact. This transforms the manipulation problem into two stages: discrete search for contact sequences and continuous optimization of control inputs. Although this method can achieve robust operation performance on hardware with a simple controller, it may have problems when dealing with long sequence tasks; this is because operating under the constraints of a predefined contact sequence is prone to falling into local optimality, and replanning a new contact sequence is also very time-consuming. The second type of methods adopts implicit contact models. For example, smooth contact models are used, or relaxed complementary constraints are used to efficiently explore a large number of possible contact modes. This type of method does not require a predefined contact sequence, but is able to calculate potential contacts online. However, such methods always sacrifice model fidelity for computational efficiency, and therefore encounter difficulties in processing long sequence tasks, because when the contact mode of the system changes, the underlying control may produce sliding contact or even lose contact because it does not consider the tracking of contact force.

[0026] In order to overcome the limitations of related technologies, the present application provides a generalizable object manipulation method and system for a multi-finger dexterous hand based on model predictive control, which is used to realize long-sequence in-hand dexterous manipulation, wherein the fingers of the dexterous hand need to establish and disengage contact alternately to realize the tracking of the desired trajectory (desired motion) of the manipulated target object. Through the desired motion of the target object, a reference sequence is planned, and the reference sequence is partially modified, and finally the dexterous hand is controlled according to the modified result to manipulate the target object to achieve the desired motion. Compared with the method of explicit modeling contact, this method does not require a predefined contact sequence or an offline planning process. Compared with the method of implicit modeling contact, it has stronger robustness because it can compensate for the errors existing in the inaccurate dynamic model. Therefore, this method can stably perform long-sequence in-hand manipulation in actual operation; in addition, this method only requires a simple initial value (i.e., the desired motion of the target object) and can be quickly generalized to objects of different shapes without any pre-training, which has significant advantages over the learning-based method.

[0027] The following describes the method and system for generalizing object manipulation using a multi-finger dexterous hand based on model predictive control provided in an embodiment of the present application in conjunction with the accompanying drawings.

[0028] The present application embodiment provides a method for manipulating objects with a multi-finger dexterous hand based on model predictive control, referring to Figure 1 As shown, Figure 11 is a flowchart of a method for generalizing object manipulation using a multi-finger dexterous hand based on model predictive control provided by an embodiment of the present application. Figure 1 As shown, the method for generalizing object manipulation by a multi-finger dexterous hand based on model predictive control may include steps S110 to S130: Step S110: Based on the quasi-dynamic dynamics model of the dexterous hand and the target object, sequence planning is performed according to the expected movement of the target object to obtain a reference sequence, wherein the reference sequence includes reference information of multiple time steps, and the reference information of each time step includes the joint positions of the dexterous hand, the expected joint positions of the dexterous hand, and the contact force between the dexterous hand and the target object, and the quasi-dynamic dynamics model has a smooth contact dynamics gradient.

[0029] In the embodiment of the present application, considering the quasi-dynamic operation achieved by frictional contact between rigid bodies, the target object being operated is regarded as a single rigid body, and the actuator can be any multi-link system (for example, a dexterous hand). Therefore, the embodiment of the present application uses a dexterous hand to operate the target object so that it can track the reference sequence.

[0030] Quasi-dynamic means that the inertia of the target object can be ignored during the operation, that is, the momentum of the target object in the current time step will not be accumulated to the next time step. At this time, the dexterous hand and the target object can be described by first-order dynamics. The quasi-dynamic operation assumption is consistent with many objects in daily life; for example, the friction between the contact surface can make the pushed object stop at any time, and the hinge with damping can make the rotating object stop at any time.

[0031] A key issue in achieving long-sequence in-hand manipulation is how to plan the process of establishing and breaking contact between fingers and objects. Therefore, based on the quasi-dynamic dynamics model of the dexterous hand and the target object, sequence planning is performed according to the expected motion of the target object to obtain a reference sequence. The reference sequence refers to the reference information (or reference trajectory) for the dexterous hand to manipulate the target object to achieve the expected motion, and the reference sequence includes reference information of multiple time steps. In this way, only the expected motion of the target object is relied on as input, without giving a predefined contact sequence.

[0032] Since the quasi-dynamic dynamics model is a contact dynamics model with smooth contact dynamics gradients, it allows information about the desired motion to be transferred to the dexterous hand joints via contact and can implicitly allow switching between different contact modes, thus planning a reference sequence containing contact information.

[0033] Step S120: Correct the reference information of each time step in the reference sequence according to the flexible contact model to obtain corrected reference information. The contact force in the flexible contact model changes with the change of the contact distance between objects. The corrected reference information includes the expected joint velocity and the expected joint position of the dexterous hand.

[0034] In the embodiment of the present application, it is considered that the quasi-dynamic dynamics model will introduce modeling errors after smoothing, so that the object will solve the "false" external force and generate displacement when the contact distance is non-zero. Therefore, directly executing the reference sequence will lead to sliding contact or even loss of contact. In order to alleviate this modeling error, the reference information of each time step in the reference sequence is corrected based on the flexible contact model and tactile feedback, so as to obtain the corrected reference information corresponding to each time.

[0035] In the flexible contact model, the contact force changes with the contact distance between the objects, rather than being generated only when the contact distance is 0 as in rigid contact modeling. Therefore, local corrections can be made near the reference sequence to ensure that the planned contact (contact point position, contact force) is accurately executed.

[0036] Step S130: Controlling the dexterous hand according to the corrected reference information to manipulate the target object to achieve the desired movement.

[0037] In the embodiment of the present application, the correction reference information includes the desired joint velocity and the desired joint position of the dexterous hand, and the correction reference information corresponding to each time step is used as the control input of the dexterous hand, so that the dexterous hand can operate the target object to achieve the desired movement. Specifically, the dexterous hand can be closed-loop controlled based on a PD (proportional differential) controller until the target object is operated to achieve the desired movement.

[0038] According to the technical solution of the embodiment of the present application, the expected motion of the target object is used as input, and the reference sequence planning is performed according to the quasi-dynamic dynamics model of the dexterous hand and the target object. Since the quasi-dynamic dynamics model has a smooth contact dynamics gradient, the information about the expected motion is allowed to be transmitted to the dexterous hand joints via contact, and the switching between different contact modes can be implicitly allowed, so that a reference sequence containing contact information is planned; and the reference information of each time step in the reference sequence is corrected based on the flexible contact model. Since the contact force in the flexible contact model changes with the change of the contact distance between the objects, local corrections can be made near the reference sequence to ensure that the planned contact force is accurately executed. Compared with the method of explicit modeling contact, this method does not require a predefined contact sequence or an offline planning process. Compared with the method of implicit modeling contact, it has stronger robustness because it can compensate for the errors existing in the inaccurate dynamics model. Therefore, this method can stably perform long-time in-hand operations in actual operations; in addition, this method only requires a simple initial value (i.e., the expected motion of the target object) and can be quickly generalized to objects of different shapes without any pre-training, which has significant advantages over the learning-based method.

[0039] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a method for manipulating objects with a multi-finger dexterous hand based on model predictive control. In this method, the above step S110 of "based on the quasi-dynamic dynamics model of the dexterous hand and the target object, sequence planning is performed according to the expected motion of the target object to obtain a reference sequence" may specifically include sub-steps S110-1 to S110-2: Step S110-1: With the goal of minimizing the difference between the target object position and the expected position of the target object, and the difference between the dexterous hand joint position and the expected joint position of the dexterous hand, a sequence planning optimal control model is constructed according to the quasi-dynamic dynamics model and the expected motion of the target object.

[0040] Step S110 - 2 : solving the sequence planning optimal control model to obtain a reference sequence.

[0041] In the embodiment of the present application, a key problem in achieving long-sequence in-hand operation is how to plan the process of establishing and breaking contact between fingers and objects, that is, to plan reference information for each time step. The embodiment of the present application models the reference sequence planning problem as an optimal control problem in a finite time domain, that is, a sequence planning optimal control model is constructed through a quasi-dynamic dynamics model and the expected motion of the target object, and then the reference sequence is obtained by solving the sequence planning optimal control model.

[0042] Specifically, with the goal of minimizing the difference between the position of the target object and the expected position of the target object, and the difference between the joint position of the dexterous hand and the expected joint position of the dexterous hand, a sequence planning optimal control model is constructed according to the quasi-dynamic dynamics model and the expected motion of the target object, including steps A1 to A4: Step A1: Taking the desired joint position of the dexterous hand as the first control input, based on the quasi-dynamic dynamics model, determine the first constraint condition, wherein the first constraint condition characterizes the dynamic relationship between the first state variable and the first control input, and the first state variable includes the joint position of the dexterous hand and the position of the target object.

[0043] In the embodiments of the present application, in order to ensure the accuracy of modeling and the real-time performance of the solution, it is hoped that the dynamics has the following good properties: 1) satisfying the "quasi-dynamic" assumption; 2) the solution is smooth and has a smooth gradient; 3) the calculation is efficient. The field of legged robots points out the importance of smooth property solutions, because smooth contact dynamics gradients allow information about the desired motion of the object to be transmitted to the dexterous hand joints via contact, and can implicitly allow switching between different contact modes. In addition, it should be noted that, unlike the exact integral used in the forward dynamics commonly used in the field of legged robots, the dynamics of the operating field itself also needs to be smooth (approximate). Therefore, a quasi-dynamic dynamics model is adopted for the dynamics.

[0044] Specifically, the quasi-dynamic dynamics model is a model based on implicit dynamics of convex problem optimization and conforms to the quasi-dynamic assumption, and the quasi-dynamic dynamics model relaxes the nonlinear contact mechanism into a cone complementarity problem. For example, the quasi-dynamic dynamics model can be a CQDC model, which is an implicit dynamics based on a convex optimization problem and conforms to the "quasi-dynamic" assumption. Both the forward solution and the reverse gradient solution can obtain a smooth approximate solution, so it has good properties expected by dynamics, and the convex optimization problem contained in the CQDC model is to relax the nonlinear contact mechanism into a cone complementarity problem.

[0045] For example, the quasi-dynamic kinetic model can be expressed as: (1) in, It is the generalized velocity of the system, which is usually composed of the angular velocity of the dexterous hand joints, the translation velocity of the object, and the rotation velocity of the object, and is expressed as the generalized coordinate change Divide by the time step h; is the signed distance of the ith contact point; It is The friction coefficient of each contact point; is the contact Jacobian matrix. In addition, and , is a parameter vector (matrix) determined by the inertia of the target object and the impedance model of the manipulator (i.e., the impedance model of the dexterous hand), where is the inertia matrix, is the joint impedance, is the generalized torque, is the discrete time step.

[0046] Specifically, the quasi-dynamic kinetic model can be solved using the barrier function method, and the solution process is explained using the logarithmic barrier function as an example.

[0047] For example, for a logarithmic barrier function: (2) Assume that the penalty problem constructed by the logarithmic barrier function has the coefficient Solve the problem below and you will get the solution to the original problem. , that is, generalized velocity (the velocity of the joints of the dexterous hand and the velocity of the target object. Specifically, for the joint angle, it can be expressed as its differential with respect to time, and its physical meaning is angular velocity; for the translation of the object, it can be expressed as its differential with respect to time, and its physical meaning is linear velocity; for the rotation of the object, there are many ways to express it, with clear physical meanings such as angular velocity, and it can also be expressed by the differential of Euler angles and axis angles with respect to time, which can be selected as needed), and the solution to the dual problem, that is, the force at the contact point (Contact force between the dexterous hand and the target object): (3) in, . Then, the gradient of the penalty problem is calculated by the chain rule, that is: (4) in, .

[0048] Thus, the dynamic gradients can be efficiently computed using the kinematic Jacobian matrix and numerical differentiation of the Jacobian matrix from the Pinocchio library Since the barrier function method is used in the solution, the above gradient is equivalent to the smoothing of the original kinetic gradient.

[0049] Finally, the first constraint condition is obtained according to the kinetic gradient, namely: (5) Among them, the first state variable , whose dimensions are , including the dexterous hand joint positions and the target object positions; Indicates The first state variable of the time step, Indicates The first state variable of the time step, represents the first control input at the i-th time step, i.e., the desired joint position of the dexterous hand, Represents a dynamic relationship.

[0050] Step A2: Determine a second constraint condition, where the second constraint condition represents a range of the first control input.

[0051] Among them, the first control can be expressed as , with the same lower and upper bounds at different time steps , where the selection matrix Pick Active degrees of freedom included.

[0052] The second constraint can be expressed as: (6) Where N is the number of time steps, from 0th to There are N in total.

[0053] Step A3: construct a first objective function according to the difference between the position of the target object and the expected position, and the difference between the joint position of the dexterous hand and the expected joint position of the dexterous hand.

[0054] Specifically, the first objective function can be expressed as: (7) Among them, the terms related to the desired motion of the target object and the dexterous hand joints are key to planning the process of establishing and breaking contact between fingers and objects.

[0055] (8) in, Drive the dexterous hand to achieve the desired object movement; Constrain the finger to do local exploration near the desired position and ignore This usually causes the system to fall into a local optimal solution, that is, the finger moves with the object, cannot actively break contact, and eventually reaches the joint limit; represents the regularization term; is the expected position of the target object, is the target object position, Represents the weighted norm between the target object position and the target object's expected position, that is, the difference between the target object position and the target object's expected position; is the desired joint position of the dexterous hand, It is the position of the dexterous hand joints. It represents the weighted norm between the joint positions of the dexterous hand and the expected joint positions of the dexterous hand, that is, the difference between the joint positions of the dexterous hand and the expected joint positions of the dexterous hand.

[0056] Step A4: Obtain a sequence planning optimal control model according to the first constraint condition, the second constraint condition, and the first objective function.

[0057] For example, the sequential programming optimal control model is expressed as: (9) In some embodiments, it is allowed to add other types of differentiable objective function terms to the sequence planning optimal control model. For example, by adding "differentiable objective function terms" to the sequence planning optimal control model, the fingers are constrained not to exceed the joint angle limit and collisions between fingers are avoided.

[0058] The optimal control model for sequential programming can be solved efficiently by using a control-constrained version of the differential dynamic programming (DDP) algorithm. Specifically, the second-order term can be omitted. , because the second-order term contains higher-order information than the kinematic Hessian matrix, it has little effect on solving the dexterous hand motion under the assumption of "quasi-dynamic". Finally, the reference sequence obtained by solving ,in, is the position of the dexterous hand joints and the position of the target object, including the position of the dexterous hand joints and the target object in N time steps, is the expected joint position of the dexterous hand, including the expected joint position of the dexterous hand in N time steps, is the contact force between the dexterous hand and the target object, which contains the contact force between the dexterous hand and the target object in N time steps.

[0059] The technical solution of the embodiment of the present application only relies on the expected motion of the target object as input, without providing a predefined contact sequence, and the reference sequence planning is modeled as an optimal control problem in a finite time domain based on the quasi-dynamic dynamics model, and then the reference sequence is quickly obtained by solving. Since the quasi-dynamic dynamics model has a smooth contact dynamics gradient, it allows information about the expected motion to be transmitted to the dexterous hand joints via contact, and can implicitly allow switching between different contact modes, thereby planning a reference sequence containing contact information.

[0060] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a method for generalizing object manipulation using a multi-finger dexterous hand based on model predictive control. In this method, the "correcting the reference information of each time step in the reference sequence according to the flexible contact model to obtain corrected reference information" in the above step S120 may specifically include sub-steps S120-1 to S120-3: Step S120-1: determining a tracking target according to reference information of each time step, wherein the target includes a reference joint position of the dexterous hand, a reference contact force between target objects, and a reference joint velocity of the dexterous hand; Step S120-2: constructing a sequence correction optimal control model according to the flexible contact model and the tracking target with the goal of minimizing the difference between the joint position of the dexterous hand and the reference joint position of the dexterous hand, the difference between the contact force between the dexterous hand and the target object and the reference contact force between the dexterous hand and the target object, and the difference between the expected joint velocity of the dexterous hand and the reference joint velocity of the dexterous hand; Step S120 - 3 : solving the sequence corrected optimal control model to obtain corrected reference information.

[0061] In the embodiment of the present application, in order to alleviate the modeling error based on the quasi-dynamic dynamics model in step S110, the reference information of each time step is corrected based on the flexible contact model and the quasi-dynamic dynamics model. Specifically, after the reference information is interpolated, it is used as a tracking target (wherein the reference joint velocity of the dexterous hand is generally 0 to suppress the high dynamic movement of the joint to meet the quasi-dynamic assumption), and then a sequence correction optimal control model is constructed based on the tracking target and the flexible contact model, and the corrected reference information is obtained by solving the sequence correction optimal control model.

[0062] Specifically, with the goal of minimizing the difference between the joint position of the dexterous hand and the reference joint position of the dexterous hand, the difference between the contact force between the dexterous hand and the target object and the reference contact force between the dexterous hand and the target object, and the difference between the expected joint velocity of the dexterous hand and the reference joint velocity of the dexterous hand, a sequence correction optimal control model is constructed according to the flexible contact model and the tracking target, including steps B1 to B4: Step B1: construct a spring-damper system including a target object, contact points and a dexterous hand according to a flexible contact model and a joint impedance model, wherein the spring-damper system describes the relationship between the joint angle and contact force of the dexterous hand through flexible contact.

[0063] In the embodiment of the present application, since the spring damping system describes the relationship between the joint angle and contact force of the dexterous hand through flexible contact, reference information correction based on the spring damping system can achieve a balance between tracking the joint angle and tracking the contact force.

[0064] Among them, the flexible contact model can be a variety of flexible contact models, but the various types of flexible contact models assume that the contact force changes with the change of the contact distance between the objects, rather than being generated only when the contact distance is 0 like the rigid body contact modeling. The flexible contact model proposed in the embodiment of the present application ignores the damping effect related to the contact speed under the assumption of "quasi-dynamic". The flexible contact model has a smooth gradient.

[0065] Specifically, Figure 2 As shown in the figure, the modeling module process of the spring damper system is as follows: First, define the equivalent stiffness in the contact frame : (10) in, The axis is aligned with the sliding direction, The axis coincides with the contact normal, It is an adjustable parameter, which is used to express the numerical relationship between normal force and tangential force.

[0066] In order to track the contact forces and the joints of the dexterous hand simultaneously, a model that couples the contact forces and the motion of the dexterous hand is needed. For this model, the following assumptions are made: 1) "quasi-dynamic" operation, 2) the contact stiffness and Jacobian matrix remain unchanged each time the optimization problem is solved (only solved once before solving according to the state of the system).

[0067] The following derivation process omits subscripts , because the derivation is the same for each contact point, under the quasi-dynamic assumption, the contact force applied by each finger is It can be expressed as: (11) in, , is the rotation matrix, It is the relative displacement between the closest point of the target object and the dexterous fingertip.

[0068] A virtual spring is attached to each contact point, and its stiffness matrix is , equivalent stiffness It can be expressed as: (12) in, Representative Grip matrix with contact points, stacking matrix Defined as: (13) To avoid ambiguity, use .according to Under gravity compensation, we get: (14) in, is the stack of contact Jacobian matrices, is the combination of contact forces. Usually choose , As a diagonal matrix. Use And ignoring the velocity term, we get: (15) Furthermore, substitute formula (11) and (15) into formula (16) and multiply on the left : (16) You can get: (17) in, , for all contact points, we can get: (18) in, is the rate of change of contact force with respect to time, yes The stacking of yes From formula (14), we can get: (19) in, represents the joint velocity of the dexterous hand, represents the expected joint velocity of the dexterous hand.

[0069] That is, formula (18) and formula (19) can be characterized as a spring-damper system.

[0070] Step B2: Taking the desired joint velocity of the dexterous hand as the second control input, determine the third constraint condition according to the spring-damping system, wherein the third constraint condition characterizes the dynamic relationship between the second state variable and the second control input, wherein the second state variable includes the joint position of the dexterous hand, the desired joint position of the dexterous hand, and the contact force between the dexterous hand and the target object.

[0071] For example, by combining formula (18) and formula (19), the desired joint velocity of the dexterous hand is used as the second control input , then the second-order contact dynamics (spring-damper system) can be further expressed as: (20) Formula (20) describes the second state variable The linear dynamics of For the position of the dexterous hand joints, The desired joint positions for the dexterous hand, is the contact force between the dexterous hand and the target object. In practical applications, when the target object is not free to move, the inclusion of Item.

[0072] Finally, formula (20) is discretized to obtain the third constraint, which can be expressed as: (twenty one) in, Indicates The second state variable for the time step is Indicates The second state variable for the time step is represents the second control input at the i-th time step, The state matrix and input matrix respectively.

[0073] Step B3: Construct a second objective function based on the difference between the joint position of the dexterous hand and the reference joint position of the dexterous hand, the difference between the contact force between the dexterous hand and the target object and the reference contact force between the dexterous hand and the target object, and the difference between the expected joint velocity of the dexterous hand and the reference joint velocity of the dexterous hand.

[0074] Step B4: Obtaining a sequence-corrected optimal control model according to the third constraint condition and the second objective function.

[0075] Specifically, the sequence corrected optimal control model can be expressed as: (twenty two) in, is the reference joint position of the dexterous hand at the Nth time step, is the joint position of the dexterous hand at N time steps, is the difference between the joint position of the dexterous hand at N time steps and the reference joint position of the dexterous hand; is the contact force between the dexterous hand and the target object at the Nth time step, is the reference contact force between the dexterous hand and the target object at the Nth time step, is the difference between the contact force between the dexterous hand and the target object at the Nth time step and the reference contact force between the dexterous hand and the target object; is the reference joint position of the dexterous hand at the i-th time step, is the joint position of the dexterous hand at time step i, is the difference between the joint position of the dexterous hand at time step i and the reference joint position of the dexterous hand; is the contact force between the dexterous hand and the target object at the i-th time step, is the reference contact force between the dexterous hand and the target object at the i-th time step, is the difference between the contact force between the dexterous hand and the target object at the i-th time step and the reference contact force between the dexterous hand and the target object; is the expected joint velocity of the dexterous hand at the i-th time step, is the reference joint velocity of the dexterous hand at the i-th time step, is the difference between the desired joint velocity of the dexterous hand at the i-th time step and the reference joint velocity of the dexterous hand.

[0076] By adopting the technical solution of the embodiment of the present application, a spring-damping system including a target object, contact points and a dexterous hand is constructed to achieve a balance between tracking joint angles and tracking contact forces. Based on the spring-damping system, local corrections can be made near the reference sequence to ensure that the planned contact force is accurately executed.

[0077] In combination with the above embodiments, in one implementation, the present application also provides a method for manipulating objects with a multi-finger dexterous hand based on model predictive control. In this method, the "controlling the dexterous hand according to the modified reference information" in the above step S130 may specifically include sub-steps S130-1 to S130-2: Step S130 - 1 : Convert the corrected reference information into joint torque.

[0078] Step S130 - 2 : performing closed-loop control on the dexterous hand based on the joint torque.

[0079] In the embodiment of the present application, the PD controller based on the shutdown level performs gravity compensation, and the correction reference information ( ) is converted to joint torque : (twenty three) in, Generalized gravity.

[0080] In this way, the dexterous hand is controlled based on the joint torque, and the real-time joint position and joint speed of the dexterous hand are obtained for closed-loop control until the dexterous hand manipulates the target object to achieve the desired movement.

[0081] Reference Figure 3 As shown, Figure 3 This is an overall architecture diagram of a multi-finger dexterous hand generalizable object manipulation method based on model predictive control provided in an embodiment of the present application. Specifically, the multi-finger dexterous hand generalizable object manipulation process can be divided into three stages. In the first stage, with the goal of minimizing the difference between the position of the target object and the expected position of the target object, and the difference between the joint position of the dexterous hand and the expected joint position of the dexterous hand, a sequence planning optimal control model is constructed according to the quasi-dynamic dynamics model and the expected motion of the target object. By solving the sequence planning optimal control model, a reference sequence is obtained, wherein the reference sequence includes reference information of multiple time steps, and the reference information of each time step includes the joint position of the dexterous hand, the expected joint position of the dexterous hand, and the contact force between the dexterous hand and the target object.

[0082] In the second stage, according to the reference information of each time step, the tracking target is determined, with the goal of minimizing the difference between the joint position of the dexterous hand and the reference joint position of the dexterous hand, the difference between the contact force between the dexterous hand and the target object and the reference contact force between the dexterous hand and the target object, and the difference between the expected joint velocity of the dexterous hand and the reference joint velocity of the dexterous hand. According to the flexible contact model and the tracking target, a sequence correction optimal control model is constructed; by solving the sequence correction optimal control model, the correction reference information is obtained, wherein the correction reference information includes the expected joint velocity of the dexterous hand and the expected joint position of the dexterous hand.

[0083] In the third stage, based on a controller (eg, a PD controller), the corrected reference information is converted into joint torques, and the dexterous hand is closed-loop controlled based on the joint torques to manipulate the target object to achieve the desired motion.

[0084] In this way, the embodiment of the present application only takes the expected motion of the target object as input, and performs reference sequence planning according to the quasi-dynamic dynamics model of the dexterous hand and the target object. Since the quasi-dynamic dynamics model has a smooth contact dynamics gradient, it allows information about the expected motion to be transmitted to the dexterous hand joints via contact, and can implicitly allow switching between different contact modes, thereby planning a reference sequence containing contact information; and, based on the flexible contact model, the reference information of each time step in the reference sequence is corrected. Since the contact force in the flexible contact model changes with the change of the contact distance between the objects, local corrections can be made near the reference sequence to ensure that the planned contact force is accurately executed. Compared with the method of explicit modeling contact, this method does not require a predefined contact sequence or an offline planning process. Compared with the method of implicit modeling contact, it has stronger robustness because it can compensate for the errors existing in the inaccurate dynamics model. Therefore, this method can stably perform long-time in-hand operations in actual operations; in addition, this method only requires a simple initial value (i.e., the expected motion of the target object) and can be quickly generalized to objects of different shapes without any pre-training, which has significant advantages over the learning-based method.

[0085] The present application also provides a multi-finger dexterous hand generalizable object operating system based on model predictive control, referring to Figure 4 As shown, Figure 4 : is a schematic diagram of a generalizable object operating system for a multi-finger dexterous hand based on model predictive control provided in an embodiment of the present application, the system comprising: A reference sequence planner 410 is used to perform sequence planning according to the expected motion of the target object based on the quasi-dynamic dynamics model of the dexterous hand and the target object to obtain a reference sequence, wherein the reference sequence includes reference information of multiple time steps, and the reference information of each time step includes the joint position of the dexterous hand, the expected joint position of the dexterous hand, and the contact force between the dexterous hand and the target object, and the quasi-dynamic dynamics model has a smooth contact dynamics gradient; A contact planner 420, configured to modify the reference information of each time step in the reference sequence according to a flexible contact model to obtain modified reference information, wherein the contact force in the flexible contact model changes with the change of the contact distance between the objects, and the modified reference information includes the desired joint velocity and the desired joint position of the dexterous hand; The impedance controller 430 is used to control the dexterous hand according to the corrected reference information to manipulate the target object to achieve the desired movement.

[0086] In the present application embodiment, Figure 5 As shown, the reference sequence planner at the upper layer takes the desired motion of the target object as input, and plans the reference sequence according to the quasi-dynamic dynamics model of the dexterous hand and the target object. Since the quasi-dynamic dynamics model has a smooth contact dynamics gradient, it allows information about the desired motion to be transmitted to the dexterous hand joints via contact, and can implicitly allow switching between different contact modes, thereby planning a reference sequence containing contact information. Next, the contact planner at the lower layer corrects the reference information of each time step in the reference sequence based on the flexible contact model. Since the contact force in the flexible contact model changes with the contact distance between objects, local corrections can be made near the reference sequence to ensure that the planned contact force is accurately executed. Finally, the dexterous hand is controlled according to the corrected reference information through the impedance controller. Based on this system, there is no need for predefined contact sequences or offline planning processes. Compared with the implicit modeling contact scheme, it is more robust because it can compensate for the errors of inaccurate dynamic models, and can thus stably perform long-time in-hand operations in actual operations. In addition, only simple initial values ​​(i.e., the expected motion of the target object) are required, and it can be quickly generalized to objects of different shapes without any pre-training, which has significant advantages over learning-based methods.

[0087] The present application also provides an electronic device, referring to Figure 6 , Figure 6 Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6As shown, the electronic device 600 includes: a memory 610 and a processor 620. The memory 610 and the processor 620 are connected via a bus communication. A computer program is stored in the memory 810. The computer program can be executed on the processor 620 to implement the steps of the method for generalizing object manipulation of a multi-finger dexterous hand based on model predictive control described in the embodiment of the present application.

[0088] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for generalizing object manipulation of a multi-finger dexterous hand based on model predictive control described in the embodiment of the present application are implemented.

[0089] The embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method for generalizing object manipulation of a multi-finger dexterous hand based on model predictive control described in the embodiment of the present application.

[0090] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0091] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods and systems according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0092] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0094] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0095] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0096] The above is a detailed introduction to the generalized object manipulation method and system of a multi-finger dexterous hand based on model predictive control provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A generalizable object manipulation method for a multi-finger dexterous hand based on model predictive control, characterized in that: include: Based on the quasi-dynamic dynamics model of the dexterous hand and the target object, sequence planning is performed according to the expected movement of the target object to obtain a reference sequence, wherein the reference sequence includes reference information of multiple time steps, and the reference information of each time step includes the joint position of the dexterous hand, the expected joint position of the dexterous hand, and the contact force between the dexterous hand and the target object, and the quasi-dynamic dynamics model has a smooth contact dynamics gradient; Correcting the reference information of each time step in the reference sequence according to the flexible contact model to obtain corrected reference information, wherein the contact force in the flexible contact model changes with the change of the contact distance between the objects, and the corrected reference information includes the expected joint velocity and the expected joint position of the dexterous hand; The dexterous hand is controlled according to the corrected reference information to manipulate the target object to achieve the desired movement.

2. The method according to claim 1, characterized in that Based on the quasi-dynamic dynamics model of the dexterous hand and the target object, sequence planning is performed according to the expected motion of the target object to obtain a reference sequence, including: With the goal of minimizing the difference between the position of the target object and the expected position of the target object, and the difference between the joint positions of the dexterous hand and the expected joint positions of the dexterous hand, a sequence planning optimal control model is constructed according to the quasi-dynamic dynamics model and the expected motion of the target object; The sequence planning optimal control model is solved to obtain a reference sequence.

3. The method according to claim 2, characterized in that With the goal of minimizing the difference between the position of the target object and the expected position of the target object, and the difference between the joint position of the dexterous hand and the expected joint position of the dexterous hand, a sequence planning optimal control model is constructed according to the quasi-dynamic dynamics model and the expected motion of the target object, including: Taking the desired joint position of the dexterous hand as the first control input, based on the quasi-dynamic dynamics model, determining a first constraint condition, wherein the first constraint condition characterizes the dynamic relationship between a first state variable and the first control input, wherein the first state variable includes the joint position of the dexterous hand and the position of the target object; determining a second constraint, the second constraint characterizing a range of the first control input; constructing a first objective function according to the difference between the position of the target object and the expected position, and the difference between the joint positions of the dexterous hand and the expected joint positions of the dexterous hand; A sequence programming optimal control model is obtained according to the first constraint condition, the second constraint condition, and the first objective function.

4. The method according to any one of claims 1 to 3, characterized in that: The quasi-dynamic dynamics model is a model based on implicit dynamics of convex problem optimization and conforms to the quasi-dynamic assumption. The quasi-dynamic dynamics model relaxes the nonlinear contact mechanism into a cone complementarity problem.

5. The method according to claim 1, characterized in that The reference information of each time step in the reference sequence is corrected according to the flexible contact model to obtain corrected reference information, including: Determine the tracking target according to the reference information of each time step, wherein the tracking target includes the reference joint position of the dexterous hand, the reference contact force between the target objects, and the reference joint velocity of the dexterous hand; constructing a sequence correction model according to the flexible contact model and the tracking target with the goal of minimizing the difference between the joint position of the dexterous hand and the reference joint position of the dexterous hand, the difference between the contact force between the dexterous hand and the target object and the reference contact force between the dexterous hand and the target object, and the difference between the expected joint velocity of the dexterous hand and the reference joint velocity of the dexterous hand; The sequence correction model is solved to obtain correction reference information.

6. The method according to claim 1, characterized in that With the goal of minimizing the difference between the joint position of the dexterous hand and the reference joint position of the dexterous hand, the difference between the contact force between the dexterous hand and the target object and the reference contact force between the dexterous hand and the target object, and the difference between the expected joint velocity of the dexterous hand and the reference joint velocity of the dexterous hand, a sequence correction model is constructed according to the flexible contact model and the tracking target, including: A spring-damper system including a target object, a contact point and a dexterous hand is constructed according to a flexible contact model and a joint impedance model, wherein the spring-damper system describes the relationship between the joint angle and the contact force of the dexterous hand through flexible contact; Taking the desired joint velocity of the dexterous hand as the second control input, and determining a third constraint condition according to the spring damping system, wherein the third constraint condition characterizes the dynamic relationship between the second state variable and the second control input, wherein the second state variable includes the joint position of the dexterous hand, the desired joint position of the dexterous hand, and the contact force between the dexterous hand and the target object; constructing a second objective function according to the difference between the joint position of the dexterous hand and the reference joint position of the dexterous hand, the difference between the contact force between the dexterous hand and the target object and the reference contact force between the dexterous hand and the target object, and the difference between the expected joint velocity of the dexterous hand and the reference joint velocity of the dexterous hand; A sequence correction model is obtained according to the third constraint condition and the second objective function.

7. The method according to claim 1, characterized in that Controlling the dexterous hand according to the corrected reference information includes: converting the corrected reference information into joint torque; The dexterous hand is controlled in a closed loop based on the joint torque.

8. A multi-finger dexterous hand generalizable object operating system based on model predictive control, characterized in that: include: A reference sequence planner is used for performing sequence planning according to the expected motion of the target object based on the quasi-dynamic dynamics model of the dexterous hand and the target object to obtain a reference sequence, wherein the reference sequence includes reference information of multiple time steps, and the reference information of each time step includes the joint position of the dexterous hand, the expected joint position of the dexterous hand, and the contact force between the dexterous hand and the target object, and the quasi-dynamic dynamics model has a smooth contact dynamics gradient; A contact planner, for correcting the reference information of each time step in the reference sequence according to a flexible contact model to obtain corrected reference information, wherein the contact force in the flexible contact model changes with the change of the contact distance between the objects, and the corrected reference information includes the desired joint velocity and the desired joint position of the dexterous hand; An impedance controller is used to control the dexterous hand according to the corrected reference information to manipulate the target object to achieve the desired movement.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for generalizing object manipulation by a multi-finger dexterous hand based on model predictive control described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for generalizing object manipulation by a multi-finger dexterous hand based on model predictive control described in any one of claims 1 to 7 are implemented.

Images