Generalizable object manipulation method and system for multi-fingered dexterous hands based on model predictive control

Abstract

The present application provides a generalizable object manipulation method and system for a multi-finger dexterous hand based on model predictive control, which relates to the field of dexterous hand control technology and aims to control the dexterous hand to manipulate the stable movement of an object. The method includes: based on a quasi-dynamic dynamic model of the dexterous hand and the target object, sequence planning is performed according to the expected movement of the target object to obtain a reference sequence, wherein the quasi-dynamic dynamic model has a smooth contact dynamic gradient; according to the contact dynamic model, the reference information of each time step in the reference sequence is corrected to obtain corrected reference information, wherein the contact force in the contact dynamic model changes with the change of the contact distance between the objects, and the corrected reference information includes the expected joint velocity and expected joint position of the dexterous hand; and according to the corrected reference information, the dexterous hand is controlled to manipulate the target object to achieve the expected movement.

Description

Technical Field

[0001] The present application relates to the technical field of dexterous hand control, and in particular to a method and system for generalizing object manipulation by a multi-finger dexterous hand based on model predictive control. Background Art

[0002] In-hand manipulation tasks for dexterous hands involve manipulating objects using only the contact of the fingers or palm, without the need for re-grasping. Multi-finger dexterous hands are a crucial skill for humanoid robots, with applications in a wide range of scenarios, such as domestic service and operating tools and machines on production lines. This skill is mechanistically based on the multi-contact problem, where the robot (i.e., the dexterous hand) interacts with objects by continuously making and breaking contact.

[0003] Since the dynamics of the contact mechanism are difficult to solve due to its non-smooth nature, problems involving contact dynamics often have highly non-smooth loss functions, which makes the control of multi-contact manipulation problems challenging. In the related art, two types of control methods, learning-based and model-based, are used to address these challenges. Although learning-based methods have shown remarkable robustness, thanks to domain randomization and large-scale training, their low data efficiency hinders further deployment and generalization. Model-based control methods can be divided into two categories. The first type of methods uses explicit contact models. This method can achieve robust operation performance on hardware with a simple controller, but may encounter problems when processing long sequence tasks. For example, the actual contact point may be inconsistent with the planned result, requiring re-planning, but the planning process is usually time-consuming. The second type of methods uses implicit contact models. This method does not require a predefined contact sequence and can calculate potential contacts online. However, implicit modeling methods always sacrifice model fidelity for computational efficiency, and therefore have poor stability when processing long sequence tasks. For example, the expected contact cannot be guaranteed to be executed.

[0004] Therefore, how to control the dexterous hand to manipulate the stable movement of objects is a technical problem that needs to be solved urgently. Summary of the Invention

[0005] In view of the above problems, embodiments of the present application provide a method and system for generalized object manipulation using a multi-finger dexterous hand based on model predictive control, so as to overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect of the embodiments of the present application, a method for generalizing object manipulation using a multi-finger dexterous hand based on model predictive control is disclosed, the method comprising:

[0007] Based on a quasi-dynamic dynamic model of the dexterous hand and the target object, sequence planning is performed according to the expected motion of the target object to obtain a reference sequence, wherein the reference sequence includes reference information of multiple time steps, and the reference information of each time step includes the joint position of the dexterous hand, the expected joint position of the dexterous hand, and the contact force between the dexterous hand and the target object. The quasi-dynamic dynamic model has a smooth contact dynamic gradient;

[0008] Correcting reference information of each time step in the reference sequence according to a flexible contact model to obtain corrected reference information, wherein the contact force in the flexible contact model changes as the contact distance between the objects changes, and the corrected reference information includes an expected joint velocity and an expected joint position of the dexterous hand;

[0009] The dexterous hand is controlled according to the corrected reference information to manipulate the target object to achieve the desired movement.

[0010] Optionally, based on a quasi-dynamic dynamics model of the dexterous hand and the target object, sequence planning is performed according to the desired motion of the target object to obtain a reference sequence, including:

[0011] With the goal of minimizing the difference between the position of the target object and the desired position of the target object, and the difference between the joint positions of the dexterous hand and the desired joint positions of the dexterous hand, a sequence planning optimal control model is constructed based on the quasi-dynamic dynamics model and the desired motion of the target object;

[0012] The sequence planning optimal control model is solved to obtain a reference sequence.

[0013] Optionally, with the goal of minimizing the difference between the target object position and the desired target object position, and the difference between the dexterous hand joint positions and the desired dexterous hand joint positions, a sequence planning optimal control model is constructed based on the quasi-dynamic dynamics model and the desired motion of the target object, including:

[0014] Determining a first constraint condition based on the quasi-dynamic dynamics model, with the desired joint position of the dexterous hand as a first control input, wherein the first constraint condition represents a dynamic relationship between a first state variable and the first control input, wherein the first state variable includes the joint position of the dexterous hand and the position of the target object;

[0015] determining a second constraint, the second constraint characterizing a range of the first control input;

[0016] constructing a first objective function based on the difference between the target object position and the expected position, and the difference between the joint positions of the dexterous hand and the expected joint positions of the dexterous hand;

[0017] A sequential programming optimal control model is obtained according to the first constraint condition, the second constraint condition, and the first objective function.

[0018] Optionally, the quasi-dynamic dynamics model is a model based on implicit dynamics of convex problem optimization and conforms to the quasi-dynamic assumption, and the quasi-dynamic dynamics model relaxes the nonlinear contact mechanism into a cone complementarity problem.

[0019] Optionally, correcting the reference information of each time step in the reference sequence according to the flexible contact model to obtain corrected reference information includes:

[0020] Determining a tracking target based on reference information at each time step, wherein the tracking target includes a reference joint position of the dexterous hand, a reference contact force between the target object and the reference joint velocity of the dexterous hand;

[0021] constructing a sequence-corrected optimal control model based on the flexible contact model and the tracking target, with the goal of minimizing the difference between the joint position of the dexterous hand and the reference joint position of the dexterous hand, the difference between the contact force between the dexterous hand and the target object and the reference contact force between the dexterous hand and the target object, and the difference between the desired joint velocity of the dexterous hand and the reference joint velocity of the dexterous hand;

[0022] The sequence correction optimal control model is solved to obtain correction reference information.

[0023] Optionally, with the goal of minimizing the difference between the joint position of the dexterous hand and the reference joint position of the dexterous hand, the difference between the contact force between the dexterous hand and the target object and the reference contact force between the dexterous hand and the target object, and the difference between the expected joint velocity of the dexterous hand and the reference joint velocity of the dexterous hand, a sequence correction optimal control model is constructed according to the flexible contact model and the tracking target, including:

[0024] A spring-damper system including a target object, contact points, and a dexterous hand is constructed based on a flexible contact model and a joint impedance model. The spring-damper system describes the relationship between the joint angle and contact force of the dexterous hand through flexible contact.

[0025] Determining a third constraint condition based on the spring-damper system, taking the desired joint velocity of the dexterous hand as a second control input, wherein the third constraint condition represents a dynamic relationship between a second state variable and the second control input, wherein the second state variable includes information about the joint position of the dexterous hand, the desired joint position of the dexterous hand, and the contact force between the dexterous hand and the target object;

[0026] constructing a second objective function according to differences between joint positions of the dexterous hand and reference joint positions of the dexterous hand, differences between contact forces between the dexterous hand and the target object and reference contact forces between the dexterous hand and the target object, and differences between desired joint velocities of the dexterous hand and reference joint velocities of the dexterous hand;

[0027] A sequence-corrected optimal control model is obtained according to the third constraint condition and the second objective function.

[0028] Optionally, controlling the dexterous hand according to the corrected reference information includes:

[0029] converting the corrected reference information into joint torque;

[0030] The dexterous hand is controlled in a closed loop based on the joint torque.

[0031] In a second aspect of the embodiments of the present application, a generalizable object operating system for a multi-finger dexterous hand based on model predictive control is disclosed, the system comprising:

[0032] A reference sequence planner is configured to perform sequence planning based on a quasi-dynamic dynamic model of the dexterous hand and the target object and the desired motion of the target object to obtain a reference sequence, wherein the reference sequence includes reference information for multiple time steps, and the reference information for each time step includes the joint positions of the dexterous hand, the desired joint positions of the dexterous hand, and the contact forces between the dexterous hand and the target object, wherein the quasi-dynamic dynamic model has a smooth contact dynamic gradient;

[0033] a contact planner for modifying reference information of each time step in the reference sequence according to a flexible contact model to obtain modified reference information, wherein the contact force in the flexible contact model changes with the change of the contact distance between the objects, and the modified reference information includes the desired joint velocity and the desired joint position of the dexterous hand;

[0034] An impedance controller is used to control the dexterous hand according to the corrected reference information to manipulate the target object to achieve the desired movement.

[0035] According to a third aspect of an embodiment of the present application, an electronic device is disclosed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method for generalizing object manipulation of a multi-finger dexterous hand based on model predictive control described in the first aspect of the embodiment of the present application are implemented.

[0036] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is disclosed, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for generalizing object manipulation of a multi-finger dexterous hand based on model predictive control described in the first aspect of the embodiment of the present application are implemented.

[0037] In a fifth aspect of an embodiment of the present application, a computer program product is disclosed, including a computer program, which, when executed by a processor, implements the steps of the method for generalizing object manipulation of a multi-finger dexterous hand based on model predictive control described in the first aspect of an embodiment of the present application.

[0038] The embodiments of the present application include the following advantages:

[0039] In an embodiment of the present application, a reference sequence is planned based on a quasi-dynamic dynamic model of the dexterous hand and the target object, using the desired motion of the target object as input. Because the quasi-dynamic dynamic model has a smooth contact dynamics gradient, it allows information about the desired motion to be transferred to the dexterous hand joints via contact and implicitly allows switching between different contact modes. Thus, a reference sequence containing contact information is planned. Furthermore, the reference information at each time step in the reference sequence is corrected based on a flexible contact model. Because the contact force in the flexible contact model changes with the contact distance between objects, local corrections can be made near the reference sequence to ensure that the planned contact force is accurately executed. Compared to methods that explicitly model contact, this method does not require a predefined contact sequence or an offline planning process. Compared to methods that implicitly model contact, this method is more robust because it can compensate for errors in an inaccurate dynamic model. Therefore, this method can stably execute long-time in-hand operations in practice. Furthermore, this method requires only a simple initial value (i.e., the desired motion of the target object) and can rapidly generalize to objects of different shapes without any pre-training, which is a significant advantage over learning-based methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0041] Figure 1 This is a flowchart of a method for generalizing object manipulation using a multi-finger dexterous hand based on model predictive control provided by an embodiment of the present application;

[0042] Figure 2 This is a schematic diagram of modeling a spring damping system provided in an embodiment of the present application;

[0043] Figure 3 This is an overall architecture diagram of a generalizable object manipulation method for a multi-finger dexterous hand based on model predictive control provided by an embodiment of the present application;

[0044] Figure 4 This is a schematic structural diagram of a generalizable object operating system for a multi-finger dexterous hand based on model predictive control provided by an embodiment of the present application;

[0045] Figure 5 This is a control architecture diagram of a generalizable object operating system for a multi-finger dexterous hand based on model predictive control provided by an embodiment of the present application;

[0046] Figure 6 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] To make the above-mentioned purposes, features, and advantages of this application more clearly understood, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of this application.

[0048] In-hand manipulation, a crucial skill for multi-fingered dexterous hands, is widely used in a variety of scenarios. However, the dynamics of multi-contact manipulation are difficult to solve due to the non-smooth nature of the contact mechanisms. Problems involving contact dynamics often have highly non-smooth loss functions, making control of multi-contact manipulation challenging. While extensive research has been conducted and achieved significant success in the field of legged robot locomotion (which also involves multi-contact problems), robotic manipulation presents a different problem. For example, robots must actively explore contacts rather than relying on gravity to generate them. Furthermore, the number and location of contact points vary over time, making the system model far more complex than for legged robots.

[0049] In the related art, two approaches, learning-based and model-based control, are employed to achieve in-hand manipulation tasks for dexterous hands. Learning-based methods (reinforcement learning or imitation learning) have demonstrated excellent performance and stability in dexterous in-hand manipulation. The success of reinforcement learning methods (often described as "experimental") can be attributed to the inherent stochasticity inherent in their Markov decision processes, extensive domain randomization during training, flexible selection of perceptual modalities, large-scale training, and the utilization of experience. In contrast, imitation learning methods require less exploration of the action space but rely heavily on expert instructional data. Some methods combine model learning with model-based control, while others explore a combination of learning-based and non-learning methods. However, learning-based methods primarily rely on data to understand the underlying mechanisms of in-hand manipulation and lack theoretical / mechanistic understanding of the common characteristics of multi-touch problems such as in-hand manipulation. Consequently, these methods may perform poorly when faced with objects not included in the training set or when there is a significant gap between simulation and reality.

[0050] Model-based control methods can be divided into two categories. Model-based algorithms offer deployment and generalization solutions without training. The first category employs explicit contact models. In the context of in-hand manipulation in multi-fingered dexterous hands, this category is characterized by the fact that planning and control typically involve explicit representation of contacts, including contact locations, contact point modalities, and contact forces. This contact information constitutes a contact sequence. Existing work uses various methods to obtain these discrete sequences. Once the contact sequence is obtained, the algorithm is simplified to solving a continuous control input, such as joint torque, while adhering to a predefined contact sequence. Therefore, the underlying control typically involves solving an optimization problem with force closure and constraining contact points to prevent sliding. The predefined contact sequence simplifies the actual optimization problem significantly. While these methods are easier to implement in hardware and produce more precise control, obtaining the contact sequence becomes the algorithm's bottleneck and the most time-consuming step. Online replanning often significantly reduces the algorithm's real-time performance. Furthermore, if the contact state deviates from the predefined contact sequence due to disturbances, the underlying control still uses the predefined contact sequence as a constraint, resulting in significant errors.

[0051] The second category of approaches employs implicit contact models: these avoid directly solving the original, exact contact dynamics by using relaxed complementarity constraints, smoothed contact models, or massively parallel sampling of control variables directly in the object engine. A specific approach employs an optimization algorithm framework based on operator splitting, decoupling the variables between time steps of the original control problem and accelerating the otherwise intractable linear complementarity problem (LCP). Smoothing the contact model is considered crucial for solving multi-contact problems, allowing the system to efficiently explore a wide range of possible contact modes. Related research has demonstrated that randomized smoothing, achieved through random sampling of states and control inputs, and analytical smoothing using barrier function methods can produce similar results for planning problems involving dexterous manipulation. Another approach to smoothing contact dynamics is through the use of flexible contact models. Another specific approach proposes a variant of this model and implements gradient propagation through inverse dynamics and contact dynamics. While implicit contact modeling methods improve the efficiency of online solutions by eliminating the need for a specific contact sequence, the smoothing of the dynamics (effectively relaxing or approximating the original dynamics) introduces a side effect of "force-at-a-distance" (which means that "virtual" contact forces are calculated when the contact distance between objects is non-zero). As a result, these methods often fail due to slight discrepancies between the actual contact modes and the planned contact modes.

[0052] In summary, while learning-based approaches exhibit remarkable robustness, owing to domain randomization and large-scale training, their data inefficiency hinders further deployment and generalization. In contrast, model-based control approaches offer plug-and-play solutions. The first category employs explicit contact models, controlling the robot to make contact at predefined contact points and to follow a predefined contact sequence to establish and disengage contact. This transforms the manipulation problem into a discrete search for a contact sequence and a continuous optimization of the control input. While this approach can achieve robust manipulation performance in hardware with a simple controller, it can struggle with long sequences. This is because operating within the constraints of a predefined contact sequence can easily lead to local optima, and replanning new contact sequences is time-consuming. The second category employs implicit contact models. For example, smoothed contact models or relaxed complementary constraints can be used to efficiently explore a large number of possible contact modes. These approaches do not require a predefined contact sequence and instead compute potential contacts online. However, such methods always sacrifice model fidelity for computational efficiency, and therefore encounter difficulties when processing long sequence tasks. This is because when the contact mode of the system changes, the underlying control does not consider the tracking of contact forces, which may cause sliding contact or even loss of contact.

[0053] To overcome the limitations of related technologies, this application provides a generalizable object manipulation method and system for a multi-finger dexterous hand based on model predictive control. This method is used to achieve long sequences of in-hand dexterous manipulation, in which the fingers of the dexterous hand alternately establish and break contact to track the desired trajectory (desired motion) of the manipulated target object. A reference sequence is planned based on the desired motion of the target object, which is then locally modified. Ultimately, the dexterous hand is controlled based on the modified results to manipulate the target object to achieve the desired motion. Compared to methods that explicitly model contact, this method does not require a predefined contact sequence or offline planning process. Compared to methods that implicitly model contact, this method is more robust because it can compensate for errors in imprecise dynamic models. Therefore, this method can stably perform long sequences of in-hand manipulation in practice. Furthermore, this method requires only a simple initial value (i.e., the desired motion of the target object) and can rapidly generalize to objects of different shapes without any pre-training, which offers significant advantages over learning-based methods.

[0054] The following describes the generalized object manipulation method and system of the multi-finger dexterous hand based on model predictive control provided by the embodiments of the present application in conjunction with the accompanying drawings.

[0055] The present invention provides a method for manipulating objects with a multi-finger dexterous hand based on model predictive control. Figure 1 As shown, Figure 1 This is a flowchart of a method for manipulating objects with a multi-finger dexterous hand based on model predictive control provided by an embodiment of the present application. Figure 1 As shown, the method for generalizing object manipulation of the multi-finger dexterous hand based on model predictive control may include steps S110 to S130:

[0056] Step S110: Based on the quasi-dynamic dynamic model of the dexterous hand and the target object, sequence planning is performed according to the expected movement of the target object to obtain a reference sequence, wherein the reference sequence includes reference information of multiple time steps, and the reference information of each time step includes the joint position of the dexterous hand, the expected joint position of the dexterous hand, and the contact force between the dexterous hand and the target object. The quasi-dynamic dynamic model has a smooth contact dynamic gradient.

[0057] In this embodiment of the present application, considering the quasi-dynamic manipulation achieved through frictional contact between rigid bodies, the manipulated target object is considered a single rigid body, while the actuator can be any multi-link system (for example, a dexterous hand). Therefore, this embodiment of the present application uses a dexterous hand to manipulate the target object, enabling it to track a reference sequence.

[0058] Quasi-dynamic manipulation assumes that the inertia of the target object can be ignored during manipulation. This means that the momentum of the target object at the current time step does not accumulate into the next time step. In this case, the dexterous hand and the target object can be described using first-order dynamics. Quasi-dynamic manipulation assumptions are consistent with many objects in everyday life; for example, friction with contact surfaces can readily bring a pushed object to a stop, and a damped hinge can readily bring a rotating object to a stop.

[0059] A key issue in achieving long-sequence in-hand manipulation is how to plan the process of establishing and breaking contact between the fingers and the object. Therefore, based on a quasi-dynamic model of the dexterous hand and the target object, sequence planning is performed based on the target object's desired motion, generating a reference sequence. The reference sequence is the reference information (or reference trajectory) for the dexterous hand to achieve the desired motion of the target object. The reference sequence includes reference information for multiple time steps. This approach relies solely on the target object's desired motion as input, eliminating the need for a predefined contact sequence.

[0060] Since the quasi-dynamic dynamics model is a contact dynamics model with smooth contact dynamics gradients, it allows information about the desired motion to be transferred to the dexterous hand joints via contact and implicitly allows switching between different contact modes, thus planning a reference sequence containing contact information.

[0061] Step S120: Correct the reference information of each time step in the reference sequence according to the flexible contact model to obtain corrected reference information. The contact force in the flexible contact model changes with the change of the contact distance between objects. The corrected reference information includes the expected joint velocity and the expected joint position of the dexterous hand.

[0062] In the embodiment of the present application, it is considered that the quasi-dynamic dynamics model will introduce modeling errors after smoothing, causing the object to calculate a "false" external force when the contact distance is non-zero, and thus generate displacement. Therefore, directly executing the reference sequence will result in sliding contact or even loss of contact. In order to alleviate this modeling error, the reference information of each time step in the reference sequence is corrected based on the flexible contact model and using tactile feedback, thereby obtaining the corrected reference information corresponding to each time.

[0063] In the flexible contact model, the contact force changes with the change of the contact distance between the objects, rather than being generated only when the contact distance is 0 as in rigid contact modeling. Therefore, local corrections can be made near the reference sequence to ensure that the planned contact (contact point position, contact force) is accurately executed.

[0064] Step S130: Controlling the dexterous hand according to the corrected reference information to manipulate the target object to achieve the desired movement.

[0065] In this embodiment of the present application, the correction reference information includes the desired joint velocities and positions of the dexterous hand. The correction reference information corresponding to each time step is used as the control input for the dexterous hand to manipulate the target object to achieve the desired motion. Specifically, closed-loop control of the dexterous hand can be performed using a proportional-derivative (PD) controller until the desired motion is achieved.

[0066] The technical solution of the embodiment of the present application uses the desired motion of the target object as input and performs reference sequence planning based on the quasi-dynamic dynamic model of the dexterous hand and the target object. Because the quasi-dynamic dynamic model has a smooth contact dynamic gradient, it allows information about the desired motion to be transmitted to the dexterous hand joints via contact and implicitly allows switching between different contact modes, thereby planning a reference sequence containing contact information. Furthermore, the reference information of each time step in the reference sequence is corrected based on the flexible contact model. Because the contact force in the flexible contact model changes with the contact distance between the objects, local corrections can be made near the reference sequence to ensure that the planned contact force is accurately executed. Compared with methods that explicitly model contact, this method does not require a predefined contact sequence or an offline planning process. Compared with methods that implicitly model contact, it has greater robustness because it can compensate for errors in imprecise dynamic models. Therefore, this method can stably execute long time-series in-hand operations in actual operations. In addition, this method only requires a simple initial value (i.e., the desired motion of the target object) and can quickly generalize to objects of different shapes without any pre-training, which is a significant advantage over learning-based methods.

[0067] In conjunction with the above embodiments, in one embodiment, the present application also provides a method for generalizing object manipulation using a multi-finger dexterous hand based on model predictive control. In this method, the above step S110 of "based on the quasi-dynamic dynamics model of the dexterous hand and the target object, sequence planning is performed according to the desired motion of the target object to obtain a reference sequence" may specifically include sub-steps S110-1 to S110-2:

[0068] Step S110-1: With the goal of minimizing the difference between the target object position and the expected target object position, and the difference between the dexterous hand joint position and the expected dexterous hand joint position, a sequence planning optimal control model is constructed according to the quasi-dynamic dynamics model and the expected motion of the target object.

[0069] Step S110 - 2 : Solve the sequence planning optimal control model to obtain a reference sequence.

[0070] In the embodiments of the present application, a key issue in achieving long-sequence in-hand manipulation is how to plan the process of establishing and breaking contact between the fingers and the object, that is, to plan the reference information for each time step. The embodiments of the present application model the reference sequence planning problem as an optimal control problem in a finite time domain, that is, to construct a sequence planning optimal control model using a quasi-dynamic dynamics model and the expected motion of the target object, and then to obtain the reference sequence by solving the sequence planning optimal control model.

[0071] Specifically, with the goal of minimizing the difference between the target object position and the desired target object position, as well as the difference between the dexterous hand joint positions and the desired dexterous hand joint positions, a sequence planning optimal control model is constructed based on the quasi-dynamic dynamics model and the desired motion of the target object, including steps A1 to A4:

[0072] Step A1: Taking the desired joint position of the dexterous hand as the first control input, based on the quasi-dynamic dynamics model, determine the first constraint condition, which characterizes the dynamic relationship between the first state variable and the first control input. The first state variable includes the joint position of the dexterous hand and the position of the target object.

[0073] In the embodiments of the present application, in order to ensure modeling accuracy and real-time solution, it is hoped that the dynamics have the following good properties: 1) satisfying the "quasi-dynamic" assumption; 2) the solution is smooth and has a smooth gradient; 3) the calculation is efficient. The field of legged robots points out the importance of smooth property solutions, because smooth contact dynamics gradients allow information about the desired motion of the object to be transmitted to the dexterous hand joints via contact, and can implicitly allow switching between different contact modes. In addition, it should be noted that, unlike the exact integral used in the forward dynamics commonly used in the field of legged robots, the dynamics of the operation domain itself also need to be smooth (approximate). Therefore, a quasi-dynamic dynamics model is adopted for the dynamics.

[0074] Specifically, the quasi-dynamic dynamics model is based on implicit dynamics for convex optimization and conforms to the quasi-dynamic assumption. The quasi-dynamic dynamics model relaxes the nonlinear contact mechanism into a cone complementarity problem. For example, the quasi-dynamic dynamics model can be a CQDC model. This CQDC model is based on implicit dynamics for convex optimization and conforms to the quasi-dynamic assumption. Both forward and backward gradient calculations yield smooth approximate solutions, thus exhibiting favorable properties expected from dynamics. Furthermore, the convex optimization problem encompassed by the CQDC model relaxes the nonlinear contact mechanism into a cone complementarity problem.

[0075] For example, the quasi-dynamic kinetic model can be expressed as:

[0076] (1)

[0077] in, It is the generalized velocity of the system, which is usually composed of the angular velocity of the dexterous hand joints, the translation velocity of the object, and the rotation velocity of the object, and is expressed as the generalized coordinate change Divide by the time step h; is the signed distance of the i-th contact point; It is The friction coefficient of each contact point; is the contact Jacobian matrix. In addition, and , is a parameter vector (matrix) determined by the target object inertia and the manipulator impedance model (i.e., the dexterous hand impedance model), where is the inertia matrix, is the joint impedance, is the generalized torque, is the discrete time step.

[0078] Specifically, the quasi-dynamic dynamics model can be solved using the obstacle function method, and the solution process is explained using the logarithmic obstacle function as an example.

[0079] For example, for the logarithmic barrier function:

[0080] (2)

[0081] Assume that the penalty problem constructed by the logarithmic barrier function has the coefficient Solve the following and you will get the solution to the original problem , that is, generalized velocity (the velocity of the dexterous hand joints and the velocity of the target object. Specifically, for the joint angle, it can be expressed as its differential with respect to time, and its physical meaning is angular velocity; for the translation of the object, it can be expressed as its differential with respect to time, and its physical meaning is linear velocity; for the rotation of the object, there are many ways to express it, with clear physical meanings such as angular velocity, and it can also be expressed as the differential of Euler angles or axis angles with respect to time, which can be selected as needed), and the solution to the dual problem, that is, the force at the contact point (Contact force between the dexterous hand and the target object):

[0082] (3)

[0083] in, . Then, the gradient of the penalty problem is calculated by the chain rule, that is:

[0084] (4)

[0085] in, .

[0086] In this way, the dynamic gradient can be efficiently calculated using the motion Jacobian matrix and the numerical differentiation of the Jacobian matrix in the Pinocchio library Since the barrier function method is used in the solution, the above gradient is equivalent to the smoothing of the original dynamic gradient.

[0087] Finally, the first constraint condition is obtained according to the kinetic gradient, namely:

[0088] (5)

[0089] Among them, the first state variable , whose dimensions are , including the dexterous hand joint positions and the target object positions; Indicates the The first state variable of the time step, Indicates the The first state variable of the time step, represents the first control input at the i-th time step, i.e., the desired joint position of the dexterous hand, Represents a dynamic relationship.

[0090] Step A2: Determine a second constraint condition, where the second constraint condition represents a range of the first control input.

[0091] Among them, the first control can be expressed as , with the same lower and upper bounds at different time steps , where the selection matrix Pick Active degrees of freedom included.

[0092] The second constraint can be expressed as:

[0093] (6)

[0094] Where N represents the number of time steps, from 0th to There are N in total.

[0095] Step A3: Construct a first objective function based on the difference between the target object position and the expected position, and the difference between the dexterous hand joint position and the expected dexterous hand joint position.

[0096] Specifically, the first objective function can be expressed as:

[0097] (7)

[0098] Among them, the terms related to the desired motion of the target object and the dexterous hand joints are key to planning the process of establishing and breaking contact between the fingers and the object.

[0099] (8)

[0100] in, Drive the dexterous hand to achieve the desired object movement; Constrain the finger to do local exploration near the desired position, ignoring This usually causes the system to fall into a local optimal solution, where the finger moves with the object, cannot actively break contact, and eventually reaches the joint limit; represents the regularization term; is the expected position of the target object, is the target object position, Represents the weighted norm between the target object position and the target object's expected position, that is, the difference between the target object position and the target object's expected position; is the desired joint position of the dexterous hand, It is the position of the dexterous hand joints, It represents the weighted norm between the joint positions of the dexterous hand and the expected joint positions of the dexterous hand, that is, the difference between the joint positions of the dexterous hand and the expected joint positions of the dexterous hand.

[0101] Step A4: Obtain a sequential programming optimal control model based on the first constraint condition, the second constraint condition, and the first objective function.

[0102] For example, the sequential programming optimal control model is expressed as:

[0103] (9)

[0104] In some embodiments, other types of differentiable objective function terms are allowed to be added to the sequence planning optimal control model. For example, by adding a "differentiable objective function term" to the sequence planning optimal control model, the fingers can be constrained not to exceed the joint angle limit and collisions between fingers can be avoided.

[0105] The optimal control model for sequential programming can be solved efficiently by using the control-constrained version of the differential dynamic programming (DDP) algorithm. Specifically, the second-order terms can be omitted. , because the second-order term contains higher-order information than the kinematic Hessian matrix, it has little effect on solving the dexterous hand motion under the assumption of "quasi-dynamic". ,in, is the position of the dexterous hand joint and the position of the target object, including the position of the dexterous hand joint and the target object in N time steps, is the expected joint position of the dexterous hand, including the expected joint position of the dexterous hand in N time steps, is the contact force between the dexterous hand and the target object, which contains the contact force between the dexterous hand and the target object in N time steps.

[0106] The technical solution of the embodiments of this application relies solely on the desired motion of the target object as input, eliminating the need for a predefined contact sequence. Based on a quasi-dynamic dynamics model, reference sequence planning is modeled as a finite-time optimal control problem, which is then quickly solved to obtain the reference sequence. Because the quasi-dynamic dynamics model has a smooth contact dynamics gradient, it allows information about the desired motion to be transferred to the dexterous hand joints via contact, and implicitly allows switching between different contact modes, thereby planning a reference sequence that includes contact information.

[0107] In conjunction with the above embodiments, in one embodiment, the present application also provides a method for generalizing object manipulation using a multi-fingered dexterous hand based on model predictive control. In this method, the step S120 of "correcting the reference information of each time step in the reference sequence according to the flexible contact model to obtain corrected reference information" may specifically include sub-steps S120-1 to S120-3:

[0108] Step S120-1: determining a tracking target based on reference information at each time step, wherein the tracking target includes a reference joint position of the dexterous hand, a reference contact force between the target object and the reference joint velocity of the dexterous hand;

[0109] Step S120-2: constructing a sequence-corrected optimal control model based on the flexible contact model and the tracking target, with the goal of minimizing the difference between the joint position of the dexterous hand and the reference joint position of the dexterous hand, the difference between the contact force between the dexterous hand and the target object and the reference contact force between the dexterous hand and the target object, and the difference between the expected joint velocity of the dexterous hand and the reference joint velocity of the dexterous hand;

[0110] Step S120 - 3 : Solve the sequence corrected optimal control model to obtain corrected reference information.

[0111] In the embodiment of the present application, to mitigate the modeling error in step S110 based on the quasi-dynamic dynamics model, the reference information for each time step is corrected based on the flexible contact model and the quasi-dynamic dynamics model. Specifically, the reference information is interpolated and used as the tracking target (wherein the reference joint velocity of the dexterous hand is generally 0 to suppress the highly dynamic motion of the joints and conform to the quasi-dynamic assumption). A sequentially corrected optimal control model is then constructed based on this tracking target and the flexible contact model. The corrected reference information is obtained by solving the sequentially corrected optimal control model.

[0112] Specifically, with the goal of minimizing the difference between the joint position of the dexterous hand and the reference joint position of the dexterous hand, the difference between the contact force between the dexterous hand and the target object and the reference contact force between the dexterous hand and the target object, and the difference between the expected joint velocity of the dexterous hand and the reference joint velocity of the dexterous hand, a sequence correction optimal control model is constructed according to the flexible contact model and the tracking target, including steps B1 to B4:

[0113] Step B1: Construct a spring-damper system including a target object, contact points, and a dexterous hand based on a flexible contact model and a joint impedance model. The spring-damper system describes the relationship between the joint angle and contact force of the dexterous hand through flexible contact.

[0114] In the embodiment of the present application, since the spring damping system describes the relationship between the joint angle and contact force of the dexterous hand through flexible contact, reference information correction based on the spring damping system can achieve a balance between tracking the joint angle and tracking the contact force.

[0115] The flexible contact model can be any of a variety of types, but these models assume that the contact force changes as the contact distance between objects changes, rather than being generated only when the contact distance is zero, as in rigid body contact modeling. The flexible contact model proposed in the present embodiment, under the "quasi-dynamic" assumption, ignores the damping effect associated with contact velocity. The flexible contact model has a smooth gradient.

[0116] Specifically, if Figure 2 As shown in the figure, the modeling module process of the spring damper system is as follows:

[0117] First, define the equivalent stiffness in the contact frame :

[0118] (10)

[0119] in, The axis is aligned with the sliding direction, The axis coincides with the contact normal, It is an adjustable parameter that is used to express the numerical relationship between normal force and tangential force.

[0120] To simultaneously track contact forces and dexterous hand joints, a model that couples contact forces and hand motion is required. This model makes the following assumptions: 1) quasi-dynamic operation, and 2) the contact stiffness and Jacobian matrix remain constant across each optimization problem (they are only calculated once based on the system state before solving the problem).

[0121] The following derivation process omits subscripts , since the derivation is the same for each contact point, under the quasi-dynamic assumption, the contact force exerted by each finger is It can be expressed as:

[0122] (11)

[0123] in, , is the rotation matrix, It is the relative displacement between the closest point of the target object and the dexterous fingertip.

[0124] A virtual spring is attached to each contact point, and its stiffness matrix is , equivalent stiffness It can be expressed as:

[0125] (12)

[0126] in, Representative contact points, a grasping matrix, and a stacking matrix Defined as:

[0127] (13)

[0128] To avoid ambiguity, use .according to Under gravity compensation, we get:

[0129] (14)

[0130] in, is the stack of contact Jacobian matrices, is the combination of contact forces. Usually choose , As a diagonal matrix. Use And ignoring the velocity term, we can get:

[0131] (15)

[0132] Furthermore, substitute formulas (11) and (15) into formula (16) and multiply on the left by :

[0133] (16)

[0134] You can get:

[0135] (17)

[0136] in, , for all contact points, we can get:

[0137] (18)

[0138] in, is the rate of change of contact force with time, yes The stacking, yes From formula (14), we can get:

[0139] (19)

[0140] in, represents the joint velocity of the dexterous hand, represents the expected joint velocity of the dexterous hand.

[0141] That is, formula (18) and formula (19) can be characterized as a spring-damper system.

[0142] Step B2: Taking the desired joint velocity of the dexterous hand as the second control input, determine a third constraint condition based on the spring-damping system, wherein the third constraint condition characterizes the dynamic relationship between the second state variable and the second control input, and the second state variable includes the joint position of the dexterous hand, the desired joint position of the dexterous hand, and the contact force between the dexterous hand and the target object.

[0143] For example, combining formula (18) and formula (19), the desired joint velocity of the dexterous hand is used as the second control input , then the second-order contact dynamics (spring-damper system) can be further expressed as:

[0144] (20)

[0145] Formula (20) describes the second state variable The linear dynamics of For the position of the dexterous hand joints, The desired joint positions for the dexterous hand, is the contact force between the dexterous hand and the target object. In practical applications, when the target object is not free to move, the inclusion of Item.

[0146] Finally, formula (20) is discretized to obtain the third constraint, which can be expressed as:

[0147] (twenty one)

[0148] in, Indicates the The second state variable of the time step, Indicates the The second state variable of the time step, represents the second control input at the i-th time step, The state matrix and input matrix respectively.

[0149] Step B3: Construct a second objective function based on the difference between the joint position of the dexterous hand and the reference joint position of the dexterous hand, the difference between the contact force between the dexterous hand and the target object and the reference contact force between the dexterous hand and the target object, and the difference between the expected joint velocity of the dexterous hand and the reference joint velocity of the dexterous hand.

[0150] Step B4: Obtain a sequence-corrected optimal control model based on the third constraint condition and the second objective function.

[0151] Specifically, the sequential corrected optimal control model can be expressed as:

[0152] (twenty two)

[0153] in, is the reference joint position of the dexterous hand at the Nth time step, is the joint position of the dexterous hand at N time steps, is the difference between the joint position of the dexterous hand at N time steps and the reference joint position of the dexterous hand; is the contact force between the dexterous hand and the target object at the Nth time step, is the reference contact force between the dexterous hand and the target object at the Nth time step, is the difference between the contact force between the dexterous hand and the target object at the Nth time step and the reference contact force between the dexterous hand and the target object; is the reference joint position of the dexterous hand at the i-th time step, is the joint position of the dexterous hand at time step i, is the difference between the joint position of the dexterous hand at time step i and the reference joint position of the dexterous hand; is the contact force between the dexterous hand and the target object at the i-th time step, is the reference contact force between the dexterous hand and the target object at the i-th time step, is the difference between the contact force between the dexterous hand and the target object at the i-th time step and the reference contact force between the dexterous hand and the target object; is the expected joint velocity of the dexterous hand at the i-th time step, is the reference joint velocity of the dexterous hand at the i-th time step, is the difference between the desired joint velocity of the dexterous hand at the i-th time step and the reference joint velocity of the dexterous hand.

[0154] By adopting the technical solution of the embodiment of the present application, a spring-damping system including a target object, contact points and a dexterous hand is constructed to achieve a balance between tracking joint angles and tracking contact forces. Based on the spring-damping system, local corrections can be made near the reference sequence to ensure that the planned contact forces are accurately executed.

[0155] In conjunction with the above embodiments, in one embodiment, the present application also provides a method for generalizing object manipulation using a multi-fingered dexterous hand based on model predictive control. In this method, the "controlling the dexterous hand according to the modified reference information" in step S130 may specifically include sub-steps S130-1 to S130-2:

[0156] Step S130 - 1 : Convert the corrected reference information into joint torque.

[0157] Step S130 - 2 : performing closed-loop control on the dexterous hand based on the joint torque.

[0158] In the embodiment of the present application, the PD controller based on the shutdown level performs gravity compensation, and the correction reference information ( ) is converted to joint torque :

[0159] (twenty three)

[0160] in, Generalized gravity.

[0161] In this way, the dexterous hand is controlled based on the joint torque, and the real-time joint position and joint speed of the dexterous hand are obtained for closed-loop control until the dexterous hand manipulates the target object to achieve the desired movement.

[0162] Reference Figure 3 As shown, Figure 3 This is an overall architecture diagram of a multi-finger dexterous hand generalizable object manipulation method based on model predictive control provided by an embodiment of the present application. Specifically, the multi-finger dexterous hand generalizable object manipulation process can be divided into three stages. In the first stage, with the goal of minimizing the difference between the target object position and the expected position of the target object, as well as the difference between the dexterous hand joint position and the dexterous hand expected joint position, a sequence planning optimal control model is constructed according to the quasi-dynamic dynamics model and the expected motion of the target object. By solving the sequence planning optimal control model, a reference sequence is obtained, wherein the reference sequence includes reference information of multiple time steps, and the reference information of each time step includes the dexterous hand joint position, the dexterous hand expected joint position, and the contact force between the dexterous hand and the target object.

[0163] In the second stage, based on the reference information of each time step, the tracking target is determined, with the goal of minimizing the difference between the joint position of the dexterous hand and the reference joint position of the dexterous hand, the difference between the contact force between the dexterous hand and the target object and the reference contact force between the dexterous hand and the target object, and the difference between the expected joint velocity of the dexterous hand and the reference joint velocity of the dexterous hand. According to the flexible contact model and the tracking target, a sequence correction optimal control model is constructed; by solving the sequence correction optimal control model, the correction reference information is obtained, wherein the correction reference information includes the expected joint velocity of the dexterous hand and the expected joint position of the dexterous hand.

[0164] In the third stage, based on a controller (eg, a PD controller), the corrected reference information is converted into joint torques, and closed-loop control is performed on the dexterous hand based on the joint torques to manipulate the target object to achieve the desired motion.

[0165] Thus, the embodiment of the present application uses only the desired motion of the target object as input and performs reference sequence planning based on the quasi-dynamic dynamic model of the dexterous hand and the target object. Because the quasi-dynamic dynamic model has a smooth contact dynamic gradient, it allows information about the desired motion to be transmitted to the dexterous hand joints via contact and implicitly allows switching between different contact modes, thereby planning a reference sequence containing contact information. Furthermore, the reference information of each time step in the reference sequence is corrected based on the flexible contact model. Because the contact force in the flexible contact model changes with the contact distance between the objects, local corrections can be made near the reference sequence to ensure that the planned contact force is accurately executed. Compared with the explicit contact modeling method, this method does not require a predefined contact sequence or an offline planning process. Compared with the implicit contact modeling method, because it can compensate for the errors of the inaccurate dynamic model, it has stronger robustness. Therefore, this method can stably perform long time series of in-hand operations in actual operations. In addition, this method only requires a simple initial value (i.e., the desired motion of the target object) and can quickly generalize to objects of different shapes without any pre-training, which has significant advantages over learning-based methods.

[0166] The present application also provides a multi-finger dexterous hand generalizable object operating system based on model predictive control, referring to Figure 4 As shown, Figure 4 : This is a schematic diagram of a generalizable object operating system for a multi-finger dexterous hand based on model predictive control provided by an embodiment of the present application. The system includes:

[0167] a reference sequence planner 410 for performing sequence planning based on a quasi-dynamic dynamic model of the dexterous hand and the target object and a desired motion of the target object to obtain a reference sequence, wherein the reference sequence includes reference information for a plurality of time steps, and the reference information for each time step includes a joint position of the dexterous hand, an expected joint position of the dexterous hand, and a contact force between the dexterous hand and the target object, wherein the quasi-dynamic dynamic model has a smooth contact dynamic gradient;

[0168] a contact planner 420 configured to modify reference information of each time step in the reference sequence according to a flexible contact model to obtain modified reference information, wherein the contact force in the flexible contact model changes as the contact distance between the objects changes, and the modified reference information includes expected joint velocities and expected joint positions of the dexterous hand;

[0169] The impedance controller 430 is used to control the dexterous hand according to the corrected reference information to manipulate the target object to achieve the desired movement.

[0170] In the embodiment of this application, Figure 5 As shown, the reference sequence planner at the upper layer takes the desired motion of the target object as input, and plans the reference sequence according to the quasi-dynamic dynamics model of the dexterous hand and the target object. Since the quasi-dynamic dynamics model has a smooth contact dynamics gradient, it allows information about the desired motion to be transmitted to the dexterous hand joints via contact, and can implicitly allow switching between different contact modes, thereby planning a reference sequence containing contact information. Next, the contact planner at the lower layer corrects the reference information of each time step in the reference sequence based on the flexible contact model. Since the contact force in the flexible contact model changes with the contact distance between objects, it can make local corrections near the reference sequence to ensure that the planned contact force is accurately executed. Finally, the dexterous hand is controlled according to the corrected reference information through the impedance controller.

[0171] Based on this system, there is no need for predefined contact sequences or offline planning processes. Compared with the implicit contact modeling scheme, it has stronger robustness because it can compensate for the errors of inaccurate dynamic models, and thus can stably perform long-time in-hand operations in actual operations. In addition, only simple initial values ​​are required (i.e., the expected motion of the target object), and it can be quickly generalized to objects of different shapes without any pre-training, which has significant advantages compared with learning-based methods.

[0172] The present application also provides an electronic device, Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6As shown, the electronic device 600 includes: a memory 610 and a processor 620. The memory 610 and the processor 620 are connected via a bus communication. A computer program is stored in the memory 810. The computer program can be run on the processor 620 to implement the steps of the method for generalizing object manipulation of a multi-finger dexterous hand based on model predictive control described in the embodiment of the present application.

[0173] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method for generalizing object manipulation of a multi-finger dexterous hand based on model predictive control described in the embodiment of the present application are implemented.

[0174] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method for generalizing object manipulation of a multi-finger dexterous hand based on model predictive control described in the embodiment of the present application.

[0175] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0176] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods and systems according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0177] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0178] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0179] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0180] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0181] The above is a detailed introduction to the generalized object manipulation method and system of a multi-finger dexterous hand based on model predictive control provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A generalizable object manipulation method for a multi-finger dexterous hand based on model predictive control, characterized in that: include: Based on a quasi-dynamic dynamic model of the dexterous hand and the target object, sequence planning is performed according to the expected motion of the target object to obtain a reference sequence, wherein the reference sequence includes reference information of multiple time steps, and the reference information of each time step includes the joint position of the dexterous hand, the expected joint position of the dexterous hand, and the contact force between the dexterous hand and the target object. The quasi-dynamic dynamic model has a smooth contact dynamic gradient; Correcting reference information of each time step in the reference sequence according to a flexible contact model to obtain corrected reference information, wherein the contact force in the flexible contact model changes as the contact distance between the objects changes, and the corrected reference information includes an expected joint velocity and an expected joint position of the dexterous hand; The dexterous hand is controlled according to the corrected reference information to manipulate the target object to achieve the desired movement.

2. The method according to claim 1, characterized in that Based on the quasi-dynamic dynamics model of the dexterous hand and the target object, sequence planning is performed according to the expected motion of the target object to obtain a reference sequence, including: With the goal of minimizing the difference between the position of the target object and the desired position of the target object, and the difference between the joint positions of the dexterous hand and the desired joint positions of the dexterous hand, a sequence planning optimal control model is constructed based on the quasi-dynamic dynamics model and the desired motion of the target object; The sequence planning optimal control model is solved to obtain a reference sequence.

3. The method according to claim 2, characterized in that With the goal of minimizing the difference between the position of the target object and the desired position of the target object, and the difference between the joint positions of the dexterous hand and the desired joint positions of the dexterous hand, a sequence planning optimal control model is constructed based on the quasi-dynamic dynamics model and the desired motion of the target object, including: Determining a first constraint condition based on the quasi-dynamic dynamics model, with the desired joint position of the dexterous hand as a first control input, wherein the first constraint condition represents a dynamic relationship between a first state variable and the first control input, wherein the first state variable includes the joint position of the dexterous hand and the position of the target object; determining a second constraint, the second constraint characterizing a range of the first control input; constructing a first objective function based on the difference between the target object position and the expected position, and the difference between the joint positions of the dexterous hand and the expected joint positions of the dexterous hand; A sequential programming optimal control model is obtained according to the first constraint condition, the second constraint condition, and the first objective function.

4. The method according to any one of claims 1 to 3, characterized in that: The quasi-dynamic dynamics model is a model based on implicit dynamics of convex problem optimization and conforms to the quasi-dynamic assumption. The quasi-dynamic dynamics model relaxes the nonlinear contact mechanism into a cone complementarity problem.

5. The method according to claim 1, wherein The reference information of each time step in the reference sequence is corrected according to the flexible contact model to obtain corrected reference information, including: Determining a tracking target based on reference information at each time step, wherein the tracking target includes a reference joint position of the dexterous hand, a reference contact force between the target object and the reference joint velocity of the dexterous hand; constructing a sequence correction model according to the flexible contact model and the tracking target with the goal of minimizing the difference between the joint positions of the dexterous hand and the reference joint positions of the dexterous hand, the difference between the contact force between the dexterous hand and the target object and the reference contact force between the dexterous hand and the target object, and the difference between the expected joint velocity of the dexterous hand and the reference joint velocity of the dexterous hand; The sequence correction model is solved to obtain correction reference information.

6. The method according to claim 1, wherein With the goal of minimizing the difference between the joint position of the dexterous hand and the reference joint position of the dexterous hand, the difference between the contact force between the dexterous hand and the target object and the reference contact force between the dexterous hand and the target object, and the difference between the expected joint velocity of the dexterous hand and the reference joint velocity of the dexterous hand, a sequence correction model is constructed according to the flexible contact model and the tracking target, including: A spring-damper system including a target object, contact points, and a dexterous hand is constructed based on a flexible contact model and a joint impedance model. The spring-damper system describes the relationship between the joint angle and contact force of the dexterous hand through flexible contact. Determining a third constraint condition based on the spring-damper system, using the desired joint velocity of the dexterous hand as a second control input, wherein the third constraint condition represents a dynamic relationship between a second state variable and the second control input, wherein the second state variable includes a joint position of the dexterous hand, a desired joint position of the dexterous hand, and a contact force between the dexterous hand and a target object; constructing a second objective function according to differences between joint positions of the dexterous hand and reference joint positions of the dexterous hand, differences between contact forces between the dexterous hand and the target object and reference contact forces between the dexterous hand and the target object, and differences between desired joint velocities of the dexterous hand and reference joint velocities of the dexterous hand; A sequence correction model is obtained according to the third constraint condition and the second objective function.

7. The method according to claim 1, characterized in that Controlling the dexterous hand according to the corrected reference information includes: converting the corrected reference information into joint torque; The dexterous hand is controlled in a closed loop based on the joint torque.

8. A multi-finger dexterous hand with generalizable object operation system based on model predictive control, characterized in that: include: A reference sequence planner is configured to perform sequence planning based on a quasi-dynamic dynamic model of the dexterous hand and the target object and the desired motion of the target object to obtain a reference sequence, wherein the reference sequence includes reference information for multiple time steps, and the reference information for each time step includes the joint positions of the dexterous hand, the desired joint positions of the dexterous hand, and the contact forces between the dexterous hand and the target object, wherein the quasi-dynamic dynamic model has a smooth contact dynamic gradient; a contact planner for modifying reference information of each time step in the reference sequence according to a flexible contact model to obtain modified reference information, wherein the contact force in the flexible contact model changes with the change of the contact distance between the objects, and the modified reference information includes the desired joint velocity and the desired joint position of the dexterous hand; An impedance controller is used to control the dexterous hand according to the corrected reference information to manipulate the target object to achieve the desired movement.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method for generalizing object manipulation of a multi-finger dexterous hand based on model predictive control according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for generalizing object manipulation of a multi-finger dexterous hand based on model predictive control described in any one of claims 1 to 7 are implemented.

Images