Pneumatic humanoid dexterous hand collaborative planning and robust control method based on digital twinning

Abstract

A pneumatic humanoid dexterous hand collaborative planning and robust control method based on digital twinning belongs to the technical field of robot dexterous hand control and pneumatic driving, and comprises the following steps: establishing a digital twinning model and realizing virtual-real bidirectional mapping; constructing a task target and a constraint set, and obtaining a gesture and a capture intention; generating an initial reference trajectory by adopting a multi-quadratic radial basis function interpolation, and performing constraint optimization by taking the initial reference trajectory as a priori and combining full multiple targeting differential dynamic programming and an augmented Lagrange method to obtain a five-finger collaborative trajectory meeting joint boundary and task constraints; establishing a finger Lagrange dynamics and tendon rope transmission model; designing a nonlinear extended state observer to estimate coupling online; and constructing self-adaptive backstepping super-torsion sliding mode control and introducing disturbance compensation to realize high-precision robust tracking. According to the method, the planning feasibility is improved through assistance of the twin environment, and the precision and robustness of the dexterous hand in the grabbing process are remarkably enhanced.

Description

Technical Field

[0001] This invention relates to the fields of robot dexterous hand control, pneumatic drive and digital twin modeling technology, and in particular to a method for collaborative planning and robust control of a pneumatic anthropomorphic dexterous hand based on digital twins. Background Technology

[0002] Humanoid dexterous hands have garnered widespread attention due to their applications in service robots, rehabilitation aids, industrial flexible grasping, and human-computer interaction. Compared to rigid motor drives, pneumatic actuators such as artificial muscles and cylinders offer advantages such as lightweight structure, smooth output, and inherently higher safety, making them suitable for tasks involving contact with the human body or vulnerable objects. However, pneumatic systems also exhibit significant nonlinear and time-varying characteristics, such as gas compressibility, hysteresis and friction, and coupling and gaps caused by tendon-wire transmission. These issues result in strong coupling, parameter uncertainty, and significant disturbances in the joint dynamics of dexterous hands, easily leading to increased trajectory tracking errors, response lag, and decreased grasping stability. Furthermore, dexterous hands typically possess redundant degrees of freedom with multi-joint and multi-finger coordination. Practical tasks such as gesture imitation, fine grasping, and manipulation not only require smooth trajectories but also need to meet multiple conditions, including joint constraints, actuator saturation, contact force constraints, obstacle avoidance constraints, and energy consumption constraints. Traditional trajectory generation methods based on empirical rules or single interpolation struggle to simultaneously achieve feasibility and optimality under complex constraints. Meanwhile, model control methods relying solely on offline calibration are ill-equipped to handle uncertainties and external disturbances in pneumatic systems and chord drives, leading to significant fluctuations in control performance with varying operating conditions. Furthermore, with the accelerated iteration of complex robot system development, digital twin technology offers a new approach for achieving virtual-real consistency assessment, online state mapping, and closed-loop optimization. Therefore, this invention proposes a technical solution for a pneumatic anthropomorphic dexterous hand that integrates digital twin closed-loop updates, constraint-optimized trajectory planning, and adaptive robust control. This solution aims to obtain executable five-finger collaborative trajectories under multiple constraints and effectively suppress the impact of chord coupling, friction, and unmodeled dynamics on system performance, thereby improving the dexterous hand's response speed, tracking accuracy, and grasping operation stability.

[0003] The main results of the cooperative motion planning and adaptive robust control method used for pneumatic dexterous hands, as obtained from the search, are as follows:

[0004] Reference 1: Fu, J., Yao, W., Sun, G., Liu, J., Wu, L. (2024). Full coverage path planning recombination framework for unmanned vehicles with multi-objective constraints, IEEE Transactions on Industrial Electronics, 71(8), 9276-9286 (Fu, J., Yao, W., Sun, G., Liu, J., Wu, L. (2024). Full coverage path planning recombination framework for unmanned vehicles with multi-objective constraints, IEEE Transactions on Industrial Electronics, 71(8), 9276-9286);

[0005] Reference 2: Brahmi, B., Ghommam, J., Saad, M. (2025). Disturbance observer-based backstepping-super twisting control for robust trajectory tracking in robot manipulators, IEEE / ASME Transactions on Mechatronics, 30(6), 5686-5697.

[0006] Reference 3: Zhao, L., Nie, Z., Xia, Y., Li, H. (2024). Virtual-physical tracking control for a car-like mobile robot based on digital twin technology, IEEE Transactions on Industrial Electronics, 71(12), 16348-16356.

[0007] To obtain feasible and optimized motion trajectories under multiple objectives and constraints, Reference 1 addresses the limitation of direct application of traditional differential dynamic programming methods due to "cross-temporal coupled non-Markov constraints," proposing a planning reorganization approach oriented towards multi-objective constraints. This provides a methodological basis for subsequent use of augmented Lagrangian constraint handling mechanisms and multi-target iterative optimization. To achieve high-precision trajectory tracking and obtain smoother control inputs, Reference 2 proposes a backstepping super-torsional control strategy based on a disturbance observer. This strategy avoids excessive feedback gain through disturbance estimation and robust term design, effectively suppressing uncertainties and external disturbances. For online adaptation and real-time updates, Reference 3 introduces digital twin technology into the virtual-real tracking control framework, enabling real-time mapping, state prediction, and dynamic trajectory adjustment, thereby improving the consistency and robustness of online control and task execution.

[0008] It can be seen that the above studies provide key support from three dimensions: "constrained trajectory optimization," "robust tracking control," and "virtual-real mapping and online updating." However, a unified integrated solution for scenarios involving pneumatic artificial muscles and strongly coupled dexterous hands with tendons and chords is still lacking. In particular, under multiple constraints such as non-Markovian smoothness requirements and actuator saturation, systematic methods that organically integrate "interpolation initialization + constrained full multi-target differential dynamic programming trajectory optimization," "adaptive backstepping-super-torsional sliding mode control supported by disturbance observers," and "digital twin virtual-real closed-loop updating" are still rare. Therefore, co-designing the above three technical routes in a pneumatic humanoid dexterous hand is of great significance for realizing executable five-finger collaborative trajectory generation, online dynamic correction, and multi-source disturbance suppression. Summary of the Invention

[0009] The purpose of this invention is to provide a cooperative motion planning and control method for grasping tasks using a chord-driven pneumatic bionic dexterous hand. By introducing multi-shot differential dynamic programming and augmented Lagrangian constraints in the motion planning stage, the non-Markov trajectory optimization problem caused by cross-temporal coupling terms is solved. In the motion control stage, a nonlinear extended state observer and an adaptive backstepping super-torsion control algorithm are designed to suppress disturbances such as chord coupling and friction, thereby achieving efficient generation and high-precision tracking of the grasping trajectory. While satisfying actuator constraints and system stability, the smoothness and reliability of the grasping action are improved.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] A collaborative planning and robust control method for a pneumatic humanoid dexterous hand based on digital twins includes the following steps:

[0012] Step 1: Based on the structural characteristics of the pneumatic anthropomorphic dexterous hand entity and the tendon-wire transmission and pneumatic artificial muscle actuation methods, a virtual structure model of the dexterous hand is established in structural simulation software, and the virtual structure is imported into a digital twin computing environment. In the digital twin computing environment, virtual sensors not present on the physical entity are configured based on the virtual structure model to acquire or calculate state and interaction quantities that cannot be directly measured on the physical entity side, so as to output joint angular velocity, joint angular acceleration, and fingertip contact force information without adding physical hardware sensors. The virtual sensors include joint angular velocity sensors, joint angular acceleration sensors, fingertip force sensors, and joint torque sensors.

[0013] Step 2: Collect operator gestures or grasping task target information. When it is a gesture imitation task, obtain the set of hand key points based on the visual key point extraction algorithm and convert it into a dexterous hand joint spatial target point sequence. In the grasping task, construct an optimization problem that includes fingertip target points, actuator saturation constraints and energy consumption constraints, and generate the corresponding planning boundary conditions and constraint set in the digital twin.

[0014] Step 3: Construct a quadratic radial basis function interpolation trajectory in the digital twin using joint interpolation nodes, and use this trajectory as the initial guess trajectory for trajectory optimization of full multi-target differential dynamic programming based on augmented Lagrange constraints.

[0015] Step 4: Establish a dynamics and disturbance model for the dexterous hand. Develop a dynamic model for each finger based on the Lagrange method, including inertial, centrifugal, and gravitational terms. Model the joint coupling changes and joint friction caused by the tendon cord looping as disturbance terms. Establish the mapping relationship from tendon cord tension to joint torque and the mechanical relationship from pneumatic artificial muscle internal pressure to tendon cord tension for control quantity solving and actuator constraint judgment.

[0016] Step 5, Disturbance Observation and Estimation: Based on the dynamic model, a disturbance observer is designed to estimate the tendon-chord coupling disturbance online; the disturbance observer is a nonlinear extended state observer, and the output disturbance estimate is used for control compensation;

[0017] Step 6: Design an adaptive robust tracking control based on a pneumatic humanoid dexterous hand: Using the reference trajectory output in Step 3 as the desired input, construct the tracking error and sliding surface; combine the disturbance estimate from Step 5 to design an adaptive backstepping super-torsional sliding mode control law to achieve high-precision tracking of the five finger joint trajectories and suppress tracking degradation caused by friction and tendon coupling; further convert the desired control torque into a valve-controlled voltage and apply it to the physical actuator.

[0018] Step 7: The convergence of the nonlinear extended state observer and the adaptive backstepping supertorsion sliding mode controller is verified using the Lyapunov method.

[0019] Furthermore, step one specifically includes: based on the number of joints of each finger... Establish a modular digital twin model and define the first knuckle angle vector

[0020]

[0021] In the formula, Indicates finger number; Indicates the first The number of joints in the fingers; Indicates the first Refers to the first The joint angles of each joint; Indicates the first Joint angle vector; Represent the angle vectors of all five finger joints; establish a global coordinate system in the virtual volume. With fingertip coordinate system And obtain the fingertip pose mapping through forward kinematics.

[0022]

[0023] In the formula, Indicates the first The homogeneous transformation matrix of the fingertip coordinate system relative to the global coordinate system; Represents the attitude rotation matrix; Represents the fingertip position vector; the sampled values ​​from the physical angle sensor. Mapped to virtual body joint angles, denoted as

[0024]

[0025] And update in real time within the digital twin Obtain the pose information of each fingertip; in the formula, Represents the vector of sampled values ​​from the angle sensor on the physical side; This represents the calibration mapping function from sensor sampled values ​​to joint angles; simultaneously, the digital twin configures virtual sensors not set on the physical side, outputting joint angular velocity, angular acceleration, joint torque, and fingertip force information.

[0026] Further, step two specifically includes: acquiring a hand image and extracting a set of key points of the hand, where the number of key points is m, and the j-th key point is represented in the image pixel coordinate system as...

[0027]

[0028] In the formula, This represents the pixel coordinates of the j-th keypoint; These represent the horizontal and vertical coordinates of the pixels, respectively; m represents the number of keypoints.

[0029] Transform the key points from the image coordinate system to the 3D hand coordinate system: Obtain the depth value corresponding to each key point. And using the camera intrinsic parameter matrix The pixels are back-projected onto the camera coordinate system to obtain 3D points.

[0030]

[0031] In the formula, This indicates that the j-th keypoint is in the camera coordinate system. The three-dimensional coordinates below; Indicates the corresponding depth value; Represent the camera intrinsic parameter matrix; let the three-dimensional coordinate system of the hand be... The rigid body transformation from the camera coordinate system to the hand coordinate system is as follows: The key points are represented in the hand coordinate system as follows:

[0032]

[0033] In the formula, This indicates that the j-th key point is in the hand coordinate system. The three-dimensional coordinates below; It is a rotation matrix; It is a translation vector;

[0034] Based on the definition of a dexterous hand joint, the 3D keypoint sequence is converted into a joint space path node sequence, denoted as .

[0035]

[0036] The nodes are then denoised, clipped, and temporally resampled to obtain the interpolation node set used in step three. In the formula, k represents the timing number; N represents the number of nodes; This represents a mapping function from the set of key points to joint angles; This represents the joint corner node corresponding to the k-th frame.

[0037] Furthermore, step three specifically includes: using the joint space interpolation nodes obtained in step two... Constructing the interpolation trajectory of multiple quadratic radial basis functions:

[0038]

[0039] In the formula, For continuous reference joint trajectory; This represents the number of interpolation nodes. This is the weight vector; For node timestamps; These are multiple quadratic radial basis functions; The radial basis function with time distance as the independent variable is used to characterize the interpolation nodes. For time The impact; For shape parameters; determined by interpolation conditions Solving for weights This yields a reference trajectory as an initial guess for trajectory optimization in the full multi-target differential dynamic programming based on augmented Lagrangian constraints. Using this reference trajectory as the initial trajectory, the trajectory optimization is formulated as a discrete optimal control problem:

[0040]

[0041]

[0042] In the formula, and Discrete time points Status and control inputs; The number of steps taken; For discrete dynamics mapping; and These are stage costs and terminal costs, respectively. Equality constraint; Inequality constraints;

[0043] The augmented Lagrangian method is used to handle constraints, and the augmented cost function is constructed as follows:

[0044]

[0045] In the formula, For Lagrange multipliers; The penalty parameter is used. The constraints are progressively satisfied through iterative trajectory optimization using full multi-target differential dynamic programming based on the augmented cost, and a reference trajectory is output.

[0046] Furthermore, step four specifically includes: establishing a dynamic model of an n-joint finger; assuming the joint angle vector of the finger is θ(t)∈R. n Its dynamic equation is

[0047]

[0048] In the formula, M( C( ) is the inertia matrix; ) is the eccentricity term matrix; G( ) represents the gravity term; The disturbance is written as the joint driving torque:

[0049]

[0050] Joint coupling interference is defined as

[0051]

[0052] In the formula, , , These are the nominal dynamic parameters; , , The parameter changes are caused by coupling changes due to chord entanglement. For equivalent coupling interference; joint friction interference is expressed as a smooth approximation using a saturation function.

[0053]

[0054] In the formula, The equivalent friction amplitude vector; ψ>0 is the smoothing coefficient; sat( () represents the element-wise saturation function; further establish the mapping relationship between "chord tension and joint torque": let the chord tension vector be... ,but

[0055]

[0056] In the formula, Jacobi for tendon-chord transmission; This is a diagonal matrix that is equivalent to the joint's force arm; The routing matrix for chordal traversal is established; and the relationship between the intramuscular pressure of the pneumatic artificial muscle and the tension of the chordal traversal is simplified as follows:

[0057]

[0058] In the formula, This represents the pressure vector within the artificial muscle. This is the length vector of the artificial muscle. It is a pressure-tension mapping function; , This is the equivalent coefficient matrix, used for solving control variables and determining actuator constraints.

[0059] Furthermore, step five specifically includes: based on the nominal dynamics of step four, writing the single-joint system in state form, letting...

[0060]

[0061] The coupling uncertainty is equivalent to the extended state. Then there is

[0062]

[0063] in

[0064]

[0065] In the formula, Joint angle vector; This refers to the joint driving torque; This is the equivalent total disturbance;

[0066] The nonlinear extended state observer is designed as follows:

[0067]

[0068] Where the observation error is

[0069]

[0070] In the formula, They are respectively Observed values; This is due to the error in joint angle observation; The observable gain matrix is ​​typically taken in diagonal form: when hour ,when hour ;

[0071] Nonlinear injection function Defined by element

[0072]

[0073] in

[0074]

[0075] In the formula, For nonlinear gain coefficients; For exponential parameters.

[0076] Furthermore, the adaptive backstepping super-torsional sliding mode control in step six is ​​designed as follows:

[0077]

[0078] In the formula, This refers to the joint driving torque; The super-torsional sliding mode control term is used to improve the system's robustness to disturbances and uncertainties and reduce chattering; Estimating compensation terms for tendon-chord coupling perturbations; This is the compensation term for the estimation of unmodeled dynamics and external disturbances;

[0079] Among them, the super torsional sliding mode item It can be constructed using the super-torsion algorithm as follows:

[0080]

[0081] In the formula, For super-torsional gain parameters; It is an integral sliding surface.

[0082] Furthermore, step seven includes: constructing the Lyapunov function as follows:

[0083]

[0084] In the formula, The backstepping error variable; For integral sliding surfaces; For internal variables related to torsion; Error in estimating unknown parameters; It is a positive definite adaptive gain matrix; The non-negative function corresponding to the hypertorsion algorithm is obtained from the virtual control law.

[0085]

[0086] Substituting the control law from step six and the disturbance estimate from step five into the closed-loop system, we obtain...

[0087]

[0088] In the formula, For feedback gain; This represents the error in perturbation estimation. This indicates element-wise multiplication; Let be the friction regression vector; the derivative of the Lyapunov function is

[0089] in The bounded terms introduced by perturbation residuals and observation errors are summarized; by constructing an adaptive law using projection operators, the key inequalities can be obtained.

[0090]

[0091] This cancels out the parameter coupling terms; further, by using Young's inequality and Cauchy–Schwarz inequality to scale the cross terms, and combining this with the external perturbation suppression property of the super-torsion term, we can obtain...

[0092]

[0093] In the formula, It is a constant related to the control gain; This is a constant related to the upper bound of the disturbance residual; by selecting a sufficiently large control gain, To be positive, and make The closed-loop system error can be obtained if it can be suppressed within a given range. and sliding surfaces The global uniformity is eventually bounded; when the perturbation residual approaches zero or a stronger condition is met, the above error further converges to zero, thus completing the stability proof.

[0094] By adopting the above technical solution, the present invention has the following technical effects:

[0095] (1) Based on the integrated architecture of pneumatic humanoid dexterous hand entity-digital twin, without adding physical hardware sensors, the system obtains non-measurable information such as joint angular velocity, angular acceleration, joint torque and fingertip contact force through virtual sensors, providing more complete state and interactive input for grasping planning and control, and improving the observability of the system and the reliability of task execution.

[0096] (2) A two-stage online planning method combining interpolation parameterization and augmented Lagrange multi-target differential dynamic programming is proposed: first, a smooth and time-adjustable continuous reference trajectory is generated using multiple quadratic radial basis functions; then, rapid optimization is performed under complex equality and inequality constraints and cross-temporal coupling cost terms to output a five-finger collaborative grasping trajectory that satisfies the constraints of smoothness, timeliness, trackability and energy consumption, thereby improving the trajectory generation efficiency and feasibility.

[0097] (3) A dynamic and driving mapping model including tendon cord coupling and friction disturbance was established, and a nonlinear extended state observer was designed to estimate the coupled disturbance, unmodeled dynamics and external disturbances online. With the help of adaptive backstepping super torsional sliding mode control, disturbance compensation and robust tracking were realized, which reduced the tracking degradation caused by friction and tendon cord coupling. The stability and convergence of observation error and tracking error were ensured by Lyapunov analysis, thereby improving the grasping accuracy and the stable operation capability of the system. Attached Figure Description

[0098] Figure 1 This is a schematic diagram of the overall framework of the pneumatic humanoid dexterous hand based on digital twins for collaborative motion planning and adaptive robust control of the present invention.

[0099] Figure 2 This is a flowchart of the collaborative motion planning and control method for a pneumatic anthropomorphic dexterous hand based on digital twins according to the present invention.

[0100] Figure 3 This is a schematic diagram of the physical structure of the pneumatic anthropomorphic dexterous hand of the present invention;

[0101] Figure 4 A schematic diagram of the three-dimensional cooperative motion trajectory of the five fingertips generated by the planning algorithm of this invention;

[0102] Figure 5 This is a schematic diagram of the joint angle trajectory tracking response and expansion state observer output curve of the present invention. Detailed Implementation

[0103] To make the objectives of this invention clearer and the technical solutions more explicit, this invention will be described in detail with reference to the following accompanying drawings and specific embodiments.

[0104] The following is combined with Figure 1-5 The cooperative motion planning and control method described in this invention is described in detail, but is not intended to limit the invention.

[0105] Figure 1 The diagram illustrates the principle of the cooperative motion planning and control method of this invention, demonstrating the trajectory generation and adaptive robust tracking control method for a pneumatic humanoid dexterous hand based on digital twins. This method starts with the task input, constructs joint reference information through key point extraction and joint interpolation node generation, introduces the digital twin hand for interpolation to generate the initial trajectory, and uses augmented Lagrangian constraints to optimize the reference trajectory. In the control loop, for the coupled and frictional disturbances experienced by the physical hand, a nonlinear extended state observer is used for disturbance estimation, and a super-torsional backstepping sliding mode controller is combined to achieve stable tracking and error convergence of the reference trajectory, thus forming an integrated closed-loop control framework of "planning-optimization-observation-control".

[0106] Figure 2 A collaborative planning and robust control method for a pneumatic humanoid dexterous hand based on digital twins, comprising the following steps:

[0107] Step 1: Based on the structural characteristics of the pneumatic anthropomorphic dexterous hand entity and the tendon-wire transmission and pneumatic artificial muscle actuation methods, a virtual structure model of the dexterous hand is established in structural simulation software, and the virtual structure is imported into a digital twin computing environment. In the digital twin computing environment, virtual sensors not present on the physical entity are configured based on the virtual structure model to acquire or calculate state and interaction quantities that cannot be directly measured on the physical entity side, so as to output joint angular velocity, joint angular acceleration, and fingertip contact force information without adding physical hardware sensors. The virtual sensors include joint angular velocity sensors, joint angular acceleration sensors, fingertip force sensors, and joint torque sensors.

[0108] Step 2: Collect operator gestures or grasping task target information. When it is a gesture imitation task, obtain the set of hand key points based on the visual key point extraction algorithm and convert it into a dexterous hand joint spatial target point sequence. In the grasping task, construct an optimization problem that includes fingertip target points, contact constraints, actuator saturation constraints and energy consumption constraints, and generate the corresponding planning boundary conditions and constraint set in the digital twin.

[0109] Step 3: In the digital twin, construct a quadratic radial basis function interpolation trajectory using joint interpolation nodes, and use this trajectory as the initial guess trajectory for full multi-target differential dynamic programming trajectory optimization based on augmented Lagrange constraints. In this embodiment, the online optimization includes at least the following steps: First, trajectory parameterization and sequential quadratic programming optimization based on multiple quadratic radial basis function interpolation are used to generate a smooth and time-adjustable reference joint trajectory. On this basis, using the generated reference trajectory as the initial trajectory, full multi-target differential dynamic programming trajectory optimization based on augmented Lagrange constraints is used to handle cross-temporal coupling cost terms and complex inequality and equality constraints, and output the five-finger collaborative grasping trajectory.

[0110] Step 4: Establish a dynamics and disturbance model for the dexterous hand. Develop a dynamic model for each finger based on the Lagrange method, including inertial, centrifugal, and gravitational terms. Model the joint coupling changes and joint friction caused by the tendon cord looping as disturbance terms. Establish the mapping relationship from tendon cord tension to joint torque and the mechanical relationship from pneumatic artificial muscle internal pressure to tendon cord tension for control quantity solving and actuator constraint judgment.

[0111] Step 5, Disturbance Observation and Estimation: Based on the dynamic model, a disturbance observer is designed to estimate the tendon-chord coupling disturbance online; the disturbance observer is a nonlinear extended state observer, and the output disturbance estimate is used for control compensation;

[0112] Step 6: Design an adaptive robust tracking control based on a pneumatic humanoid dexterous hand: Using the reference trajectory output in Step 3 as the desired input, construct the tracking error and sliding surface; combine the disturbance estimate from Step 5 to design an adaptive backstepping super-torsional sliding mode control law to achieve high-precision tracking of the five finger joint trajectories and suppress tracking degradation caused by friction and tendon coupling; further convert the desired control torque into a valve-controlled voltage and apply it to the physical actuator.

[0113] Step 7: The convergence of the nonlinear extended state observer and the adaptive backstepping supertorsion sliding mode controller is verified using the Lyapunov method.

[0114] Step one specifically includes determining the number of joints in each finger. Establish a modular digital twin model, dividing the five fingers into five finger modules, defining the first finger module... knuckle angle vector

[0115]

[0116] In the formula, Indicates finger number; Indicates the first The number of joints in the fingers; Indicates the first Refers to the first The joint angles of each joint; Indicates the first Joint angle vector; Represents the angle vector of all five finger joints of the hand;

[0117] Establish a global coordinate system in the virtual volume With fingertip coordinate system And obtain the fingertip pose mapping through forward kinematics.

[0118]

[0119] In the formula, Indicates the first The homogeneous transformation matrix of the fingertip coordinate system relative to the global coordinate system; Represents the attitude rotation matrix; This represents the fingertip position vector.

[0120] Sampled values ​​from solid angle sensor Mapped to virtual body joint angles, denoted as

[0121]

[0122] And update in real time within the digital twin Obtain the pose information of each fingertip; in the formula, Represents the vector of sampled values ​​from the angle sensor on the physical side; This represents the calibration mapping function from sensor sampled values ​​to joint angles. Simultaneously, the digital twin is equipped with virtual sensors not present on the physical side, outputting measurement information such as joint angular velocity, angular acceleration, joint torque, and fingertip force.

[0123]

[0124] And based on the fingertip Jacobian matrix Establish the mapping relationship between velocity and force:

[0125]

[0126] In the formula, and These represent the joint angular velocity vector and the joint angular acceleration vector, respectively. Indicates the first Jacobian matrix at your fingertips; Indicates the first Finger speed; Indicates the first Joint torque vector; Indicates the first The force vector at the fingertip.

[0127] Step two specifically includes: acquiring a hand image and extracting a set of key points for the hand. Let the number of key points be m, and the j-th key point be represented in the image pixel coordinate system as...

[0128]

[0129] In the formula, This represents the pixel coordinates of the j-th keypoint; m ... And using the camera intrinsic parameter matrix The pixels are back-projected onto the camera coordinate system to obtain 3D points.

[0130]

[0131] In the formula, This indicates that the j-th keypoint is in the camera coordinate system. The three-dimensional coordinates below; Indicates the corresponding depth value; Represent the camera intrinsic parameter matrix; establish a global coordinate system for the hand in the virtual volume, assuming the hand's three-dimensional coordinate system is... The rigid body transformation from the camera coordinate system to the hand coordinate system is as follows: The key points are represented in the hand coordinate system as follows:

[0132]

[0133] In the formula, This indicates that the j-th key point is in the hand coordinate system. The three-dimensional coordinates below; It is a rotation matrix; Let be the translation vector; according to the definition of a dexterous hand joint, the 3D keypoint sequence is converted into a joint space path node sequence, denoted as .

[0134]

[0135] The nodes are then denoised, clipped, and temporally resampled to obtain the interpolation node set used in step three. In the formula, k represents the timing number; N represents the number of nodes; This represents a mapping function from the set of key points to joint angles; This represents the joint corner node corresponding to the k-th frame.

[0136] Step three specifically includes: using the joint space interpolation nodes obtained in step two. Constructing the interpolation trajectory of multiple quadratic radial basis functions:

[0137]

[0138] In the formula, For continuous reference joint trajectory; This represents the number of interpolation nodes. This is the weight vector; For node timestamps; These are multiple quadratic radial basis functions; The radial basis function with time distance as the independent variable is used to characterize the interpolation nodes. For time The impact; For shape parameters; determined by interpolation conditions Solving for weights This yields a reference trajectory as an initial guess for trajectory optimization in the full multi-target differential dynamic programming based on augmented Lagrangian constraints. Using this reference trajectory as the initial trajectory, the trajectory optimization is formulated as a discrete optimal control problem:

[0139]

[0140]

[0141] In the formula, and Discrete time points Status and control inputs; The number of steps taken; For discrete dynamics mapping; and These are stage costs and terminal costs, respectively. Equality constraint; Inequality constraints;

[0142] The stage cost function contains only a smoothing term and an aerodynamic gas consumption term, defined as follows:

[0143]

[0144] in The weight matrix for the smoothing term; Here is the weight matrix for the gas consumption term; the augmented Lagrangian method is used to handle constraints, and the augmented cost function is constructed as follows:

[0145]

[0146] In the formula, For Lagrange multipliers; The penalty parameter is used as the basis for the stepwise satisfaction of constraints through full multi-target differential dynamic programming trajectory optimization iteration based on the augmented cost, and outputs a five-finger cooperative reference trajectory and corresponding sequence control that satisfy the smoothness and energy consumption constraints; in this embodiment, the penalty parameter is taken as... and in the outer iteration according to Gradually increase and do not exceed Lagrange multipliers initialized to , and according to Update until the constraint residuals are satisfied. .

[0147] Step four specifically includes: establishing a dynamic model of an n-joint finger. Let the joint angle vector of this finger be θ(t)∈R. n Its dynamic equation is

[0148]

[0149] In the formula, M( C( ) is the inertia matrix; G( is the Coriolis / eccentric term matrix; G( ) represents the gravity term; To represent the joint driving torque, the disturbance is written as:

[0150]

[0151] Joint coupling interference is defined as

[0152]

[0153] In the formula, , , These are the nominal dynamic parameters; , , The parameter changes are caused by coupling changes due to the tendon ligament wrapping. For equivalent coupling interference; joint friction interference is expressed as a smooth approximation using a saturation function.

[0154]

[0155] In the formula, The equivalent friction amplitude vector; ψ>0 is the smoothing coefficient; sat( () represents the element-wise saturation function; further establish the mapping relationship between "chord tension and joint torque": let the chord tension vector be... ,but

[0156]

[0157] In the formula, Jacobi for tendon-chord transmission; This is a diagonal matrix that is equivalent to the joint's force arm; A chordal routing matrix is ​​constructed, and the relationship between the intramuscular pressure of the pneumatic artificial muscle and the tension of the chordal is established, which is simplified as follows:

[0158]

[0159] In the formula, This represents the pressure vector within the artificial muscle. This is the length vector of the artificial muscle. It is a pressure-tension mapping function; , This is the equivalent coefficient matrix, used for solving control variables and determining actuator constraints.

[0160] Step five specifically includes: based on the nominal dynamics of step four, writing the single-joint system in state form, letting...

[0161]

[0162] The coupling uncertainty is equivalent to the extended state. Then there is

[0163]

[0164] in

[0165]

[0166] In the formula, Joint angle vector; This refers to the joint driving torque; This is the equivalent total disturbance;

[0167] The nonlinear extended state observer is designed as follows:

[0168]

[0169] Where the observation error is

[0170]

[0171] In the formula, They are respectively Observed values; This is due to the error in joint angle observation; The observable gain matrix is ​​typically taken in diagonal form: when hour ,when hour ;

[0172] Nonlinear injection function Defined by element

[0173]

[0174] in

[0175]

[0176] In the formula, For nonlinear gain coefficients; For exponential parameters.

[0177] The adaptive backstepping super-torsional sliding mode control design in step six is ​​as follows:

[0178]

[0179] In the formula, This is the joint control torque vector; The super-torsional sliding mode control term is used to improve the system's robustness to disturbances and uncertainties and reduce chattering; Estimating compensation terms for tendon-chord coupling perturbations; This is the compensation term for the estimation of unmodeled dynamics and external disturbances;

[0180] Among them, the super torsional sliding mode item It can be constructed using the super-torsion algorithm as follows:

[0181]

[0182] In the formula, For the super-torsional gain parameter, It is an integral sliding surface.

[0183] Step seven includes: proving the stability and convergence of the nonlinear extended state observer and the adaptive backstepping supertorsion sliding mode controller using the Lyapunov method; constructing the Lyapunov function as follows:

[0184]

[0185] In the formula, The backstepping error variable; For integral sliding surfaces; For internal variables related to torsion; Error in estimating unknown parameters; It is a positive definite adaptive gain matrix; The non-negative function corresponding to the hypertorsion algorithm is obtained from the virtual control law.

[0186]

[0187] Substituting the control law from step six and the disturbance estimate from step five into the closed-loop system, we can obtain...

[0188]

[0189] In the formula For feedback gain; This represents the error in perturbation estimation. This indicates element-wise multiplication; The regression vector is the one with respect to friction.

[0190] The derivative of the Lyapunov function satisfies

[0191] in The bounded terms introduced by perturbation residuals and observation errors are summarized; an adaptive law is constructed using the projection operator, and the key inequality is obtained.

[0192]

[0193] This cancels out the parameter coupling terms; further, by using Young's inequality and Cauchy–Schwarz inequality to scale the cross terms, and combining this with the external perturbation suppression property of the super-torsion term, we obtain...

[0194]

[0195] In the formula, It is a constant related to the control gain; This is a constant related to the upper bound of the disturbance residual. By selecting a sufficiently large control gain, it becomes... To be positive, and make The closed-loop system error can be obtained if it can be suppressed within a given range. and sliding surfaces The global uniformity is eventually bounded; when the perturbation residual approaches zero or a stronger condition is met, the above error further converges to zero, thus completing the stability proof.

[0196] The present invention will now be described in conjunction with specific embodiments:

[0197] To verify the effectiveness of the proposed pneumatic humanoid dexterous hand collaborative planning and robust control method based on digital twin, an integrated experimental platform of pneumatic humanoid dexterous hand entity and digital twin virtual body was built. Experiments were conducted to verify the five-finger collaborative motion and grasping tasks, so as to illustrate the feasibility of trajectory generation under multiple constraints and the high-precision robust tracking capability under the presence of disturbances.

[0198] The experimental platform includes: a pneumatic humanoid dexterous hand entity (such as...) Figure 3 The system includes a valve-controlled drive and air supply system, an industrial control computer, and a digital twin computing environment. A virtual body model consistent with the physical structure is established within the digital twin environment, and virtual sensors not present on the physical side are configured to output or calculate state and interaction quantities such as joint angular velocity, joint angular acceleration, joint torque, and fingertip contact force.

[0199] In this embodiment, the controller runs on an industrial control computer with a sampling period of 0.01s; air supply is provided by an air compressor with a supply pressure of 0.50MPa; trajectory smoothness weight and energy consumption weight are set to the same order of magnitude to balance trajectory trackability and aerodynamic air consumption.

[0200] The initial state is set as follows: the initial angle of each joint of the five fingers is 0°, and the virtual body and the physical body complete the joint angle synchronization and zero-position calibration at the initial moment; under this initial condition, the digital twin side generates and outputs the five-finger collaborative reference trajectory and corresponding sequence control quantity for subsequent tracking.

[0201] The planning objectives and constraints are set as follows: Based on the input of a gesture imitation or grasping task, construct a discrete optimal control problem that includes joint limits, actuator saturation, and smoothness and energy consumption indices; firstly, generate a smooth initial trajectory in a digital twin based on multiple quadratic radial basis function interpolation, and then further optimize the initial trajectory online using full multi-target differential dynamic programming with augmented Lagrangian constraints to obtain a five-finger cooperative optimal reference trajectory that satisfies multiple constraints.

[0202] in, Figure 4The figure shows the coordinated motion trajectory curves of the five fingertips in three-dimensional space, which are used to characterize the spatial displacement and temporal evolution of the five fingers during the execution of a task. The smoothness and coordination characteristics of the five fingertips demonstrate the optimization effect. At the same time, the fingertip trajectory is synthesized from the joint trajectory. We can also calculate the optimized trajectory to be tracked for each joint from the optimization method.

[0203] The control method is set as follows: establish the dynamics and drive mapping relationship of the finger on the solid side, and treat uncertainties such as tendon ligament coupling and friction as disturbances; use a nonlinear extended state observer to estimate the disturbances online, and combine the reference trajectory to construct the tracking error and sliding surface, and design an adaptive backstepping super torsional sliding mode control law to output the joint control quantity.

[0204] To test the superiority of the control algorithm in this example, the optimized inter-joint trajectory was used for tracking by a physical dexterous hand, such as... Figure 5 As shown. Figure 5 The time-domain curves for joint angle tracking response and observer disturbance estimation are used to characterize how, under the action of the designed controller, each joint output can quickly track the reference trajectory, and how the observer output can effectively estimate coupled and frictional disturbances, thus providing a basis for control compensation. In summary, from Figure 4 and Figure 5 It is evident that the method of the present invention can achieve stable generation and high-precision tracking of five-finger collaborative trajectories, and maintain good dynamic response and convergence performance under the condition of disturbance.

[0205] Based on the disclosure and teachings of the foregoing specification, those skilled in the art can make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments described above, and any obvious improvements, substitutions, or modifications made by those skilled in the art based on the present invention are within the scope of protection of the present invention.

Claims

1. A method for collaborative planning and robust control of a pneumatic humanoid dexterous hand based on digital twins, characterized in that, Includes the following steps: Step 1: Based on the structural characteristics of the pneumatic anthropomorphic dexterous hand entity and the tendon-wire transmission and pneumatic artificial muscle actuation methods, a virtual structure model of the dexterous hand is established in structural simulation software, and the virtual structure is imported into a digital twin computing environment. In the digital twin computing environment, virtual sensors not present on the physical entity are configured based on the virtual structure model to acquire or calculate state and interaction quantities that cannot be directly measured on the physical entity side, so as to output joint angular velocity, joint angular acceleration, and fingertip contact force information without adding physical hardware sensors. The virtual sensors include joint angular velocity sensors, joint angular acceleration sensors, fingertip force sensors, and joint torque sensors. Step 2: Collect operator gestures or grasping task target information. When it is a gesture imitation task, obtain the set of hand key points based on the visual key point extraction algorithm and convert it into a dexterous hand joint spatial target point sequence. In the grasping task, construct an optimization problem that includes fingertip target points, actuator saturation constraints and energy consumption constraints, and generate the corresponding planning boundary conditions and constraint set in the digital twin. Step 3: Construct a quadratic radial basis function interpolation trajectory in the digital twin using joint interpolation nodes, and use this trajectory as the initial guess trajectory for trajectory optimization of full multi-target differential dynamic programming based on augmented Lagrange constraints. Step 4: Establish a dynamics and perturbation model for the dexterous hand. Establish a dynamics model for each finger based on the Lagrange method, including inertial terms, centrifugal terms, and gravity terms. The joint coupling changes caused by chord looping and joint friction are modeled as a unified perturbation term. Establish the mapping relationship from tendon chord tension to joint torque and the mechanical relationship from pneumatic artificial muscle internal pressure to tendon chord tension, for use in control quantity solving and actuator constraint judgment; Step 5, Disturbance Observation and Estimation: Based on the dynamic model, a disturbance observer is designed to estimate the tendon-chord coupling disturbance online; the disturbance observer is a nonlinear extended state observer, and the output disturbance estimate is used for control compensation; Step 6, Design Adaptive Robust Tracking Control Based on Pneumatic Humanoid Dexterous Hand: Using the reference trajectory output in Step 3 as the desired input, construct the tracking error and sliding surface; Based on the disturbance estimate from step five, an adaptive backstepping super-torsional sliding mode control law is designed to achieve high-precision tracking of the five-finger joint trajectory and suppress tracking degradation caused by friction and tendon coupling; the desired control torque is further converted into a valve-controlled voltage and applied to the physical actuator. Step 7: The convergence of the nonlinear extended state observer and the adaptive backstepping supertorsion sliding mode controller is verified using the Lyapunov method.

2. The method for collaborative planning and robust control of a pneumatic humanoid dexterous hand based on digital twins according to claim 1, characterized in that, Step one specifically includes: Based on the number of joints in each finger Establish a modular digital twin model and define the first knuckle angle vector In the formula, Indicates finger number; Indicates the first The number of joints in the fingers; Indicates the first Refers to the first The joint angles of each joint; Indicates the first Joint angle vector; Represents the angle vector of all five finger joints of the hand; Establish a global coordinate system in the virtual volume With fingertip coordinate system And obtain the fingertip pose mapping through forward kinematics. In the formula, Indicates the first The homogeneous transformation matrix of the fingertip coordinate system relative to the global coordinate system; Represents the attitude rotation matrix; This represents the fingertip position vector.

3. The method for collaborative planning and robust control of a pneumatic humanoid dexterous hand based on digital twins according to claim 2, characterized in that, Step two specifically includes: Acquire hand images and extract a set of key points. Let the number of key points be m, and the j-th key point be represented in the image pixel coordinate system as follows: In the formula, This represents the pixel coordinates of the j-th keypoint; These represent the horizontal and vertical coordinates of the pixels, respectively; m represents the number of keypoints. Transform the key points from the image coordinate system to the 3D hand coordinate system: Obtain the depth value corresponding to each key point. And using the camera intrinsic parameter matrix The pixels are back-projected onto the camera coordinate system to obtain 3D points. In the formula, This indicates that the j-th keypoint is in the camera coordinate system. The three-dimensional coordinates below; Indicates the corresponding depth value; Represents the camera intrinsic parameter matrix; Then, let the three-dimensional coordinate system of the hand be... The rigid body transformation from the camera coordinate system to the hand coordinate system is as follows: The key points are represented in the hand coordinate system as follows: In the formula, This indicates that the j-th key point is in the hand coordinate system. The three-dimensional coordinates below; It is a rotation matrix; It is a translation vector.

4. The method for collaborative planning and robust control of a pneumatic humanoid dexterous hand based on digital twins according to claim 3, characterized in that, Step three specifically includes: The multi-quadratic radial basis function interpolation trajectory is constructed using the joint space target point sequence obtained in step two, as follows: In the formula, For continuous reference joint trajectory; This represents the number of interpolation nodes. This is the weight vector; For node timestamps; These are multiple quadratic radial basis functions; The radial basis function with time distance as the independent variable is used to characterize the interpolation nodes. For time The impact; For shape parameters; determined by interpolation conditions Solving for weights This yields a reference trajectory as an initial guess for trajectory optimization in the full multi-target differential dynamic programming based on augmented Lagrangian constraints. Using this reference trajectory as the initial trajectory, the trajectory optimization is formulated as a discrete optimal control problem: In the formula, and Discrete time points Status and control inputs; The number of steps taken; For discrete dynamics mapping; and These are stage costs and terminal costs, respectively. Equality constraint; Inequality constraints; The augmented Lagrangian method is used to handle constraints, and the augmented cost function is constructed as follows: In the formula, For Lagrange multipliers; The penalty parameter is used to gradually satisfy the constraints through full multi-target differential dynamic programming trajectory optimization iteration based on the augmented cost, and outputs a five-finger collaborative reference trajectory and corresponding sequence control that satisfy the smoothness and energy consumption constraints.

5. The method for collaborative planning and robust control of a pneumatic humanoid dexterous hand based on digital twins according to claim 4, characterized in that, Step four specifically includes: Establish a dynamic model of an n-joint finger, and let the joint angle vector of the finger be θ(t)∈R. n Its dynamic equation is In the formula, M( ) is the inertia matrix; C( ) is the eccentricity term matrix; G( ) represents the gravity term; The disturbance is written as the joint driving torque: Joint coupling interference is defined as In the formula, , , These are the nominal dynamic parameters; , , This refers to the parameter changes caused by variations in the ligament's circumference. For equivalent coupling interference; joint friction interference is expressed as a smooth approximation using a saturation function. In the formula, The equivalent friction amplitude vector; ψ>0 is the smoothing coefficient; sat( ) represents the element-wise saturation function.

6. The method for collaborative planning and robust control of a pneumatic humanoid dexterous hand based on digital twins according to claim 5, characterized in that, Step five specifically includes: Based on the finger dynamics model from step four, the single-joint system is written in state form, letting... The coupling uncertainty is equivalent to the extended state. Then there is in In the formula, Joint angle vector; This refers to the joint driving torque; This is the equivalent total disturbance; The nonlinear extended state observer is designed as follows: Where the observation error is In the formula, They are respectively Observed values; This is due to the error in joint angle observation; The observable gain matrix is ​​typically taken in diagonal form: when hour, ;when hour, ; Nonlinear injection function Defined by element in In the formula, For nonlinear gain coefficients; For exponential parameters.

7. The method for collaborative planning and robust control of a pneumatic humanoid dexterous hand based on digital twins according to claim 6, characterized in that, The adaptive backstepping super-torsional sliding mode control design in step six is ​​as follows: In the formula, This refers to the joint driving torque; The super-torsional sliding mode control term is used to improve the system's robustness to disturbances and uncertainties and reduce chattering; To estimate the compensation term for tendon-chord coupling perturbation; This is the compensation term for the estimation of unmodeled dynamics and external disturbances; For super-torsional gain parameters; It is an integral sliding surface.

8. The method for collaborative planning and robust control of a pneumatic humanoid dexterous hand based on digital twins according to claim 7, characterized in that, Step seven includes: Construct the Lyapunov function as follows In the formula, The backstepping error variable; For integral sliding surfaces; For internal variables related to torsion; Error in estimating unknown parameters; It is a positive definite adaptive gain matrix; is the non-negative function corresponding to the hyper-torsion algorithm; Constructed from virtual control laws Substituting the control law from step six and the disturbance estimate from step five into the closed-loop system, we obtain... In the formula, For feedback gain; This represents the error in perturbation estimation. This indicates element-wise multiplication; The friction regression vector; The derivative of the Lyapunov function is in The bounded terms introduced by the perturbation residuals and observation errors are summarized; an adaptive law is constructed using the projection operator, and the key inequality is obtained. This cancels out the parameter coupling terms; further, by using Young's inequality and Cauchy–Schwarz inequality to scale the cross terms, and combining this with the external perturbation suppression property of the super-torsion term, we obtain... In the formula, It is a constant related to the control gain; This is a constant related to the upper bound of the disturbance residual; by selecting a sufficiently large control gain, To be positive, and make The closed-loop system error can be suppressed within a given range. and sliding surfaces Global consistency is eventually bounded.

Images