A method and system for human-machine mapping and dexterous manipulation based on virtual hand driving

Abstract

The application discloses a kind of man-machine mapping and deft operation method and system based on virtual hand drive, form complete closed loop from hand perception to deft hand control.It includes: image is collected with monocular RGB camera, after normalization etc., 21 three-dimensional coordinates of joint of hand are predicted through multi-task deep neural network, then data is smoothed through dynamic filtering;Particle swarm optimization algorithm is used to fit the shape parameters of MANO hand model;Based on weighted singular value decomposition, the rotation matrix of root joint is calculated, and the global rotation matrix of non-fingertip joint is calculated recursively combined with hand topology structure;Local coordinate system of hand is established, and angle mapping function is designed to convert joint angle into deft hand control quantity;Through double buffering mechanism interpolation processing low frame rate perception result, hybrid control is realized by fusing current data.The scheme improves precision, robustness, adaptability and safety, and provides technical support for efficient and reliable human-computer interaction.

Description

Technical Field

[0001] This invention belongs to the field of human-computer collaboration technology, and in particular relates to a human-computer mapping and dexterous operation method and system based on virtual hand-driven human-computer interaction. Background Technology

[0002] In recent years, with the development of human-machine collaboration technology, dexterous hands have played an increasingly important role in complex tasks such as industrial assembly, telemedicine, and service robots. Especially in remote control or teleoperation systems, guiding dexterous hands to complete fine tasks using the natural movement trajectory of the human hand has become an important research direction. Traditional human-dexterous hand mapping methods mainly rely on devices such as data gloves, inertial measurement units (IMUs), or depth cameras to capture hand posture, but these methods have significant limitations: data gloves require complex sensor arrays, limiting the naturalness of operation and incurring high costs; depth camera-based solutions are limited by device size and light sensitivity, making them difficult to widely apply in mobile scenarios. While hand posture estimation technology based on monocular RGB images has made some progress in recent years, it still faces multiple challenges in terms of real-time performance, 3D reconstruction accuracy, and motion mapping adaptability.

[0003] Existing monocular vision methods mostly employ end-to-end neural networks to directly predict the 3D coordinates of hand joints, but they suffer from two key drawbacks: First, they ignore hand bone length constraints and biomechanical motion characteristics, leading to reconstructed results prone to physically illogical states such as joint misalignment and bone penetration; second, the prediction results lack temporal smoothness, and joint jitter directly affects the stability of dexterity hand control. Regarding mapping algorithms, existing research often uses linear scaling or simple inverse kinematics solutions, failing to consider the structural differences between the human hand and the dexterity hand in terms of degree of freedom distribution and joint range of motion, resulting in deformed grasping movements or end effector oscillations. Furthermore, existing systems generally lack force feedback mechanisms; when the dexterity hand contacts an object, rigid control can easily lead to uncontrolled grasping force, making stable gripping difficult. Therefore, this paper proposes a human-machine mapping and dexterity operation method based on virtual hand-driven approaches. This method achieves accurate 3D human hand reconstruction and precise dexterity hand mapping by integrating hand bone length constraints, biomechanical motion characteristics, and structural differences between the human hand and the dexterity hand. Summary of the Invention

[0004] To address the above technical problems, this invention provides a method and system for human-machine mapping and dexterous operation based on virtual hand-driven technology.

[0005] The technical solution adopted by this invention to solve its technical problem is:

[0006] A human-computer mapping and dexterous operation method based on virtual hand-driven operation, the method comprising the following steps:

[0007] S100: Normalizes and resizes hand images acquired by a monocular RGB camera, inputs the resized images into a multi-task deep neural network, and simultaneously predicts the three-dimensional coordinate information of 21 joints of the human hand; a dynamic filtering mechanism is used to smooth the joint time series data.

[0008] S200: The shape parameters of the MANO hand model are fitted using the particle swarm optimization algorithm, with the objective function being the minimization of the normalized bone length error, combined with regularization constraints to suppress parameter divergence;

[0009] S300: Calculate the root joint rotation matrix based on weighted singular value decomposition: Construct a set of weighted vectors for the template joint and the target joint, generate a covariance matrix by specifying weight coefficients and perform singular value decomposition to obtain the initial rotation alignment matrix;

[0010] S400: Recursively calculates the global rotation matrix of non-finger joints using hand topology and biomechanical constraints: the predicted joint position is transformed to the local coordinate system of the parent node, the relative rotation matrix of the child node is solved by the vector rotation relationship between the template bone and the predicted bone, and the global rotation matrix is ​​generated by combining the Rodrigues transformation.

[0011] S500: Establish a local coordinate system for the hand with the wrist as the origin and the direction of the coordinate axes determined by the direction of the middle and ring fingers and their cross product. Design a finger bending angle mapping function based on the degree of freedom distribution of the dexterous hand to convert the joint angle into a dexterous hand control quantity.

[0012] S600: Through a double buffering mechanism, the low frame rate perception results are linearly interpolated and converted into a preset control frequency input. It integrates the current data of the dexterous hand end. When the current exceeds the threshold, the current position is locked; otherwise, it is controlled according to the mapping result, thus realizing the mixed control of force and position.

[0013] Preferably, in S100, the hand image acquired by the monocular RGB camera is normalized and its size adjusted, specifically as follows:

[0014] The input image is cropped and resized to the resolution required for the model input, and then subjected to channel-level pixel normalization as shown in the following formula:

[0015] ;

[0016] in, This represents the normalized image tensor. This indicates the resized image in the channel. ,Location( The pixel value at () It is the first The channel's preset mean. It is the first The preset standard deviation of the channel, The three color channels of digital image color data;

[0017] The adjusted image in S100 is input into a multi-task deep neural network to predict the 3D coordinate information of 21 joints of the human hand, including:

[0018] The processed image is used to extract image features through a ResNet50 backbone network, and a lightweight CNN is used to generate 2D heatmaps for 21 joints. Each heatmap represents the probability distribution of the corresponding joint point in the image. Based on the feature maps and 2D heatmaps, 3D position maps and skeletal orientation maps are further predicted. Finally, by combining camera intrinsic parameters and reference bone length, the absolute depth of the root joint is calculated through geometric constraints to recover the global 3D position and obtain the 3D coordinates of the 21 joints of the human hand.

[0019] S100 employs a dynamic filtering mechanism to smooth the timing data of joint positions, specifically as follows:

[0020] ;

[0021] in, Indicates time The original input signal, Indicates time The filtered output signal, Indicates time The smoothing coefficient of the filtered output signal Changes with speed:

[0022] ;

[0023] ;

[0024] in, The minimum cutoff frequency, For speed sensitivity coefficient, The absolute value of the instantaneous velocity of the input signal. The time interval between two signals. It is an intermediate variable that is dynamically adjusted with speed.

[0025] Preferably, S200 includes:

[0026] S210: Let the number of particle swarm iterations be... Population size is Each particle represents a 10-dimensional chiral parameter vector. Its value range is The initial position of each particle is The initial velocity is And generate the position matrix of the current group. With velocity matrix :

[0027] ;

[0028] S220: Input the predicted joint coordinates of the target hand into the MANO model to obtain the reference target bone length. The reference target bone length vector is arranged according to a specific bone connection order, including 15 bone segments extending from the wrist towards each finger joint; to eliminate individual differences and the influence of dimensions, the shape parameters corresponding to each candidate particle are... Input the data into the MANO shape generator to generate the particle's three-dimensional joint coordinates. ;

[0029] S230: Let the parent-child node numbers of each bone be... , Represents the skeletal coordinates of the parent node. Representing the skeletal coordinates of the child node, then the first... The bone length vector of each particle The calculation is as follows:

[0030] ;

[0031] The goal is to normalize the skeletal structure vector, therefore, from the target skeletal structure vector... A specific bone segment length is selected as the reference length. In each round of optimization, the bone length vector generated from the candidate particles Extract its corresponding segment length Used to normalize the structure of the current particle:

[0032] ;

[0033] in, This represents the normalized current bone vector. Represents the normalized target skeleton vector;

[0034] To guide the particles to converge to a chiral shape that matches the target structure, the following fitness function is designed. :

[0035] ;

[0036] in, This is due to skeletal structure matching error. The regularization term is used to suppress shape parameter divergence. The weights are regularized; the fitness function values ​​of the entire population are calculated in batches using a vectorized method, and the optimal positions of individuals and the global optimum are selected for the next iteration;

[0037] S240: The standard particle swarm optimization formula is used for search iteration. In each iteration, the... The position and velocity of each particle are updated according to the following formula:

[0038] ;

[0039] ;

[0040] in, Inertia factor There are two learning factors. For independently sampled random vectors, Indicates uniform distribution. Represents particles The location of the optimal solution found in each iteration is its individual historical best record; It represents the smallest of the current best values ​​of all individuals in the population, and represents the global optimal solution of the particle swarm.

[0041] After each iteration, the individual optimal position and fitness of the particles are updated. If the global optimal solution does not show a significant improvement compared to the previous round, the search is terminated early. The final optimal particle position is obtained. This refers to the optimized human hand shape parameters, which are then used to generate the bone length. With target structure Perform error alignment analysis to quantify the fitting accuracy:

[0042] ;

[0043] in, Indicates the optimal particle position Length of generated bones With target structure The L2 norm error between them.

[0044] Preferably, S300 includes:

[0045] S310: Based on the input template joint data and target joint data Construct vector sets from the root joint to each finger base joint in both the template joint data and the target joint data. and :

[0046] ；

[0047] ；

[0048] in, These represent the three-dimensional coordinates of joints 0, 1, 5, 9, 13, and 17 in the template joint, respectively. These represent the three-dimensional coordinates of joints 0, 1, 5, 9, 13, and 17 in the target joint, respectively.

[0049] S320: Construct the weighted covariance matrix :

[0050] ；

[0051] in, These are the weighting coefficients for the thumb, index finger, middle finger, ring finger, and little finger, respectively. Represents a diagonal matrix;

[0052] S330: For the matrix Perform singular value decomposition:

[0053] ;

[0054] in, Describes a left singular vector matrix. Represents a singular value diagonal matrix. Represents a right singular vector matrix;

[0055] S340: Based on the left singular vector matrix and right singular vector matrix Calculate the rotation matrix of the root joint. This is to achieve initial alignment between the template joint and the target joint.

[0056] .

[0057] Preferably, S400 includes:

[0058] S410: Let the three-dimensional position of the template hand joint be... The predicted joint position is ,in This indicates the joint number, with 0 representing the root node of the wrist; for each joint... The global rotation matrix and position of its parent node are respectively represented as: , Then, local coordinate transformation and bone alignment are performed to convert the predicted joint position into its position relative to the parent node in the parent node's local coordinate system, as shown below:

[0059] ;

[0060] in, The relative position after conversion It is the inverse of the global rotation matrix of the parent node. For the prediction results, the first The position coordinates of each joint;

[0061] At the same time, the bone vector in the template space The calculation is as follows:

[0062] ;

[0063] in, For joints Location coordinates, The coordinates of its parent joint;

[0064] S420: Subsequently, the rotation axis and rotation angle are calculated. The template bone is rotated to the predicted bone using a two-vector rotation method to obtain its rotation axis. and rotation angle They are respectively:

[0065] ;

[0066] ;

[0067] in, It is an infinitesimal quantity to prevent division by zero;

[0068] S430: Construct a bone alignment rotation matrix based on the rotation axis and rotation angle. Further combine this with the matrix of rotation around the bone direction, and multiply the two to obtain the rotation matrix of the child node relative to the parent node. for:

[0069] ;

[0070] Here, Rodrigues represents the Rodrigues transformation, which converts a description in the form of rotation axis–rotation angle into a rotation matrix. Indicates the direction around the template skeleton The additional rotation angle, ultimately, is determined based on the parent node's global rotation matrix. This yields the rotation matrix of the child nodes in global coordinates. for:

[0071] ;

[0072] It can also be based on the global position of the parent node. The global position of the child nodes is calculated recursively. :

[0073] ;

[0074] S440: After completing the rotation calculations for all joints except the fingertips, construct the pose rotation matrix set. The optimized shape parameters are input into the MANO model to obtain the complete pose information of 21 points of the hand.

[0075] Preferably, S500 includes:

[0076] S510: Establish a local coordinate system for the human hand, selecting the wrist as the origin, and choosing the normalized vectors of the vectors formed by the wrist origin, the middle finger origin (3), and the ring finger origin (4). The direction is obtained by taking the cross product of the vector from the wrist base point and the middle finger base point 3 with the vector from the wrist base point and the ring finger base point 4, and then normalizing it. The direction is then determined by the right-hand screw rule, which gives the direction of lateral movement of the palm. The direction, specifically:

[0077] ;

[0078] in, This represents the vector formed by the wrist base point and the middle finger base point 3. This represents the vector formed by the wrist base point pointing to the ring finger base point 4;

[0079] Then all manpower data is converted to the local coordinate system of the manpower in real time:

[0080] ;

[0081] in, This represents the position in the transformed local coordinate system. Represents the original global coordinates. Represents the global coordinates of the wrist pivot point;

[0082] S520: Because the dexterous hand has a total of 6 degrees of freedom, and the four fingers other than the thumb can only bend and not lateralize, meaning that only the degree of finger bending can be controlled, therefore, the degree of freedom for bending of each finger other than the thumb is... The design calculation formula is as follows:

[0083] ;

[0084] in, The three-dimensional vector representing the fingertip relative to the corresponding finger base point. The projection onto a plane, because all fingers except the thumb can only move in a plane perpendicular to the x-axis. This indicates the projection direction; simultaneously, in order to convert it into the input data format required by the time-dependent dexterous hand, a control variable conversion was performed to obtain the final dexterous hand control variable. :

[0085] ;

[0086] For the bending degree of freedom of the thumb, the projection vector is obtained by projecting the three-dimensional vector of the thumb tip relative to the base point onto the plane formed by the vector from the base point to the thumb's base point 6 and the z-axis. Then, the angle between the projection vector and the vector from the base point to the thumb's base point 6 is calculated to obtain the bending degree of freedom of the thumb. :

[0087] ;

[0088] in, This represents the projection vector obtained above. The vector represents the distance from the base point to the base point of the thumb (point 6). For the lateral swing degree of freedom of the thumb, the projection vector is obtained by projecting the three-dimensional vector of the thumb tip relative to the base point onto the xy plane, and then calculating the projection vector. With reference vector The included angle between them gives the lateral freedom of the thumb. :

[0089] ;

[0090] Similarly, these control values ​​are converted into the input data format of the dexterous hand:

[0091] ;

[0092] .

[0093] Preferably, S600 includes:

[0094] S610: Uses non-blocking data transmission for adjacent time intervals. Data Linear interpolation between them is performed to obtain Control quantity of interpolation time t between :

[0095] ;

[0096] When a sudden change in motion is detected, that is... When the value is obtained, the value from the previous moment is automatically used as the current value.

[0097] S620: Employs a force-position hybrid control strategy, maintaining the current bending angle of the dexterous hand's fingers once the current value representing finger force reaches the current threshold, and performing normal movements when the current value is less than the threshold, thus achieving safe and stable grasping operations.

[0098] ;

[0099] in, This indicates the input angle obtained through manual recognition and calculation. This indicates the current actual bending state of the dexterous hand. This represents the final input value for dexterous hand control. Represents the proportionality coefficient. Indicates the current current value. Indicates the current threshold;

[0100] S630: After each grabbing operation, the system determines whether it is a real contact by comparing the time window between the target current value and the trigger threshold. If a short-term contact-release-re-contact oscillation occurs, the system will automatically enter the damping mode to maintain the current state until the next operation intention is generated, ensuring the continuity and safety of the operation.

[0101] A human-computer mapping and dexterous operating system based on virtual hand-driven operation, comprising:

[0102] The hand 3D joint perception module is used to normalize and resize hand images acquired by a monocular RGB camera, and input the resized images into a multi-task deep neural network to simultaneously predict the 3D coordinate information of 21 joints of the human hand; a dynamic filtering mechanism is used to smooth the joint time series data.

[0103] The virtual hand model parameter fitting module is used to fit the shape parameters of the MANO hand model using the particle swarm optimization algorithm, with the objective function being the minimization of the normalized bone length error, combined with regularization constraints to suppress parameter divergence.

[0104] The root joint rotation matrix calculation module is used to calculate the root joint rotation matrix based on weighted singular value decomposition: construct a set of weighted vectors for the template joint and the target joint, generate a covariance matrix by specifying weight coefficients and perform singular value decomposition to obtain the initial rotation alignment matrix;

[0105] The global rotation matrix calculation module for non-finger joints is used to recursively calculate the global rotation matrix of non-finger joints using the topology of the hand and biomechanical constraints: the predicted joint position is transformed to the local coordinate system of the parent node, the relative rotation matrix of the child node is solved by the vector rotation relationship between the template bone and the predicted bone, and the global rotation matrix is ​​generated by combining the Rodrigues transformation.

[0106] The Joint Angle-Dexterity Hand Control Quantity Mapping Module is used to establish a local coordinate system for the hand with the wrist as the origin and the coordinate axes determined by the middle and ring finger directions and cross product. Based on the degree of freedom distribution of the dexterity hand, a finger bending angle mapping function is designed to convert joint angles into dexterity hand control quantities.

[0107] The dexterous hand control signal processing and execution module is used to linearly interpolate the low frame rate perception results through a double buffering mechanism, convert them into a preset control frequency input, and integrate the dexterous hand end current data. When the current exceeds the threshold, the current position is locked; otherwise, the control is based on the mapping result, thus realizing a hybrid control of force and position.

[0108] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of a human-computer mapping and dexterity operation method based on virtual hand-driven operation.

[0109] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a human-computer mapping and dexterity operation method based on a virtual hand-driven approach.

[0110] A method and system for human-machine mapping and dexterous manipulation based on virtual hand-driven operation is presented. It achieves low-cost hand perception through a monocular RGB camera, accurately predicts the 3D coordinates of 21 key points using a multi-task deep neural network, and ensures the smoothness of time-series data through a dynamic filtering mechanism. Particle swarm optimization is used to fit the shape parameters of the MANO model, combined with weighted singular value decomposition and recursive calculation to achieve precise mapping between the human hand and the dexterous hand. The system adapts to the degrees of freedom of the dexterous hand by establishing a local coordinate system for the hand and designing an angle mapping function. Hybrid control using double-buffered interpolation and current fusion balances real-time performance and operational safety. Overall, it achieves low-cost, high-precision human-machine mapping and dexterous manipulation, improving interaction stability and reliability. Attached Figure Description

[0111] Figure 1 This is a flowchart of a human-computer mapping and dexterous operation method based on virtual hand-driven operation in one embodiment of the present invention;

[0112] Figure 2 This is a schematic diagram of hand joint recognition and local coordinate system establishment in one embodiment of the present invention;

[0113] Figure 3 This is an example of real-time 3D human hand pose reconstruction and dexterous hand mapping based on virtual hand driving in one embodiment of the present invention; wherein, (a) represents the real-time input human hand, (b) represents the virtual hand inferred in real time based on the human hand pose, and (c) represents the result mapped to the dexterous hand in real time;

[0114] Figure 4This is an example of real-time 3D human hand pose reconstruction and dexterous hand mapping effect based on virtual hand driving in another embodiment of the present invention; wherein, (a) represents the human hand input in real time, (b) represents the virtual hand inferred in real time based on the human hand pose, and (c) represents the result mapped to the dexterous hand in real time;

[0115] Figure 5 This invention demonstrates the effect of the change in the degree of freedom of the index finger bending when a dexterous hand grasps an object, as shown in one embodiment of the invention.

[0116] Figure 6 This invention demonstrates the effect of the change in the degree of freedom of the middle finger bending when a dexterous hand grasps an object, as shown in one embodiment of the invention.

[0117] Figure 7 This invention demonstrates the effect of the change in the degree of freedom of the ring finger of a dexterous hand when a dexterous hand grasps an object, as shown in one embodiment of the invention.

[0118] Figure 8 This invention demonstrates the change in the degree of freedom of the little finger bending of a dexterous hand when a dexterous hand grasps an object, as shown in one embodiment of the invention.

[0119] Figure 9 This invention demonstrates the change in the degree of freedom of the thumb bending of a dexterous hand when a dexterous hand grasps an object, as shown in one embodiment of the invention.

[0120] Figure 10 This invention demonstrates the effect of the change in the degree of freedom of the thumb's lateral swing when a dexterous hand grasps an object, as shown in one embodiment of the invention. Detailed Implementation

[0121] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0122] In one embodiment, such as Figure 1 As shown, a human-computer mapping and dexterous operation method based on virtual hand-driven operation is described, the method comprising the following steps:

[0123] S100: Normalizes and resizes hand images acquired by a monocular RGB camera, inputs the resized images into a multi-task deep neural network, and simultaneously predicts the three-dimensional coordinate information of 21 joints of the human hand; a dynamic filtering mechanism is used to smooth the joint time series data.

[0124] S200: The shape parameters of the MANO (Model Articulated Non-Origid Shapes) hand model are fitted using the particle swarm optimization algorithm, with the objective function being the minimization of the normalized bone length error, combined with regularization constraints to suppress parameter divergence;

[0125] S300: Calculate the root joint rotation matrix based on weighted singular value decomposition: Construct a set of weighted vectors for the template joint and the target joint, generate a covariance matrix by specifying weight coefficients and perform singular value decomposition to obtain the initial rotation alignment matrix;

[0126] S400: Recursively calculates the global rotation matrix of non-finger joints using hand topology and biomechanical constraints: the predicted joint position is transformed to the local coordinate system of the parent node, the relative rotation matrix of the child node is solved by the vector rotation relationship between the template bone and the predicted bone, and the global rotation matrix is ​​generated by combining the Rodrigues transformation.

[0127] S500: Establish a local coordinate system for the hand with the wrist as the origin and the direction of the coordinate axes determined by the direction of the middle and ring fingers and their cross product. Design a finger bending angle mapping function based on the degree of freedom distribution of the dexterous hand to convert the joint angle into a dexterous hand control quantity.

[0128] S600: Through a double buffering mechanism, the low frame rate perception results are linearly interpolated and converted into a preset control frequency input. It integrates the current data of the dexterous hand end. When the current exceeds the threshold, the current position is locked; otherwise, it is controlled according to the mapping result, thus realizing the mixed control of force and position.

[0129] Specifically, this invention aims to solve the problems of poor mapping effect, unstable reconstruction, and non-robust grasping control when using human hand to guide dexterous hand movement. It designs a human-machine mapping and dexterous operation method based on virtual hand-driven operation, which features low cost, high precision, high real-time performance, and generalizability. The main contents of this method are as follows:

[0130] First, images from a monocular RGB camera are normalized and resized, then input into a deep neural network based on multi-task learning. The network structure can simultaneously predict 2D heatmaps, 3D position maps, and skeletal orientation maps of 21 hand joints, achieving spatial localization of the joints. To improve prediction stability, a dynamic filtering mechanism is introduced to smooth the joint positions in the time series, effectively suppressing interference caused by image input jitter and prediction abrupt changes. 2. After obtaining the initial joint positions, a particle swarm optimization algorithm is used to optimize the hand shape parameters, ensuring that the prediction results are consistent with biological skeletal length constraints. Subsequently, an adaptive inverse kinematics module based on weighted singular value decomposition is proposed. Combining the hand's topological structure and biomechanical constraints, the global rotation matrix of joints other than the fingertips is recursively calculated, thereby achieving high-precision, drift-free 3D pose modeling of the human hand. 3. A stable local coordinate system for the human hand is constructed based on the positions of 21 key points. The coordinate axis directions are defined using a vector construction method based on key point combinations. A mapping function between finger movements and dexterous hand control quantities is designed, fully considering the differences between the human hand and the dexterous hand in terms of degree of freedom distribution, motion constraints, and posture space, to achieve a natural and continuous motion mapping process. 4. To achieve high response speed and control continuity, a non-blocking transmission mechanism and time axis interpolation technology are used to convert low frame rate perception results into a 60Hz control frequency input, alleviating jitter caused by communication delays. Simultaneously, dexterous hand end-effector current data is integrated, and a force threshold judgment mechanism is introduced to dynamically switch control strategies when touching or grasping objects, achieving adaptive stable grasping control based on a combination of position-driven and force-adjusted approaches. 5. Finally, the method of this invention is integrated into a time-dependent dexterous hand control system with 6 degrees of freedom. Based on actual human hand operation demonstrations, the system's performance in target recognition, posture reconstruction, motion mapping, and stable grasping is verified. Experimental results show that the method has good real-time performance, accuracy, and grasping stability, and is applicable to various scenarios such as remote collaboration, virtual reality control, and human-computer interaction.

[0131] Furthermore, a schematic diagram of human hand joint recognition and local coordinate system establishment is shown below. Figure 2 As shown, for the 21 key points obtained through key point feature extraction and position and pose calculation of the human hand, the wrist base point 0 is taken as the origin of the coordinate system, the sum of the normalized vectors from base point 0 to the middle finger root joint 3 and from base point 0 to the unnamed root joint 4 is taken as the y direction of the coordinate system, the cross product normalized vector from base point 0 to the middle finger root joint 3 and from base point 0 to the unnamed root joint 4 is taken as the z direction of the coordinate system, and the x direction is obtained by cross product of the y direction and the z direction.

[0132] In one embodiment, S100 normalizes and resizes the hand image acquired by the monocular RGB camera, specifically as follows:

[0133] First, the RealSense D435 camera image is displayed using the windowing functionality of OpenCV (Open Source Computer Vision Library) to position the hand appropriately within the camera's viewpoint. Then, the input image is cropped and resized to the required 128x128 resolution for the model input, followed by channel-level pixel normalization as shown in the following formula:

[0134] ;

[0135] in, This represents the normalized image tensor. This indicates the resized image in the channel. ,Location( The pixel value at () It is the first The channel's preset mean. It is the first The preset standard deviation of the channel, The three color channels of digital image color data;

[0136] The adjusted image in S100 is input into a multi-task deep neural network to predict the 3D coordinate information of 21 joints of the human hand, including:

[0137] The processed image is used to extract image features via a ResNet50 backbone network, and a lightweight CNN is used to generate 2D heatmaps for 21 joints. Each heatmap represents the probability distribution of the corresponding joint point in the image. Based on the feature maps and 2D heatmaps, 3D position maps and skeletal orientation maps are further predicted. Finally, by combining camera intrinsic parameters and reference bone length, the absolute depth of the root joint is calculated through geometric constraints to recover the global 3D position, obtaining the 3D coordinates of the 21 joints of the human hand. This invention is compatible with various feature extraction backbone network structures, such as ResNet50, HRNet, and MobileNetV2, allowing for the selection of an appropriate network based on the deployment hardware platform and real-time requirements. The model can be pre-trained on public datasets and quickly adapt to different user hand shapes and operating habits through a small amount of transfer learning data, improving the system's generalization ability.

[0138] S100 employs a dynamic filtering mechanism to smooth the timing data of joint positions, specifically as follows:

[0139] ;

[0140] in, Indicates time The original input signal, Indicates time The filtered output signal, Indicates time The smoothing coefficient of the filtered output signal Changes with speed:

[0141] ;

[0142] ;

[0143] in, The minimum cutoff frequency, For speed sensitivity coefficient, The absolute value of the instantaneous velocity of the input signal. The time interval between two signals. It is an intermediate variable that is dynamically adjusted with speed.

[0144] Furthermore, to avoid keypoint prediction biases caused by input image quality degradation or occlusion, a skeleton consistency check module was implemented in the pose estimation stage. This module sets a threshold based on the physiological length range between adjacent joints. When a significant deviation is detected, the previous reliable pose is automatically retrieved for prediction correction, ensuring the stability and continuity of pose estimation. Specifically, an adaptive fitting method for human hand shape parameters based on particle swarm optimization was used. This method uses joint bone length as a constraint and continuously optimizes the shape parameters in the MANO hand model to ensure that the reconstructed result is structurally consistent with the predicted result, thereby improving the accuracy and physical consistency of subsequent pose estimation.

[0145] In one embodiment, S200 includes:

[0146] S210: Let the number of particle swarm iterations be... Population size is Each particle represents a 10-dimensional chiral parameter vector. Its value range is The initial position of each particle is The initial velocity is And generate the position matrix of the current group. With velocity matrix :

[0147] ;

[0148] S220: Input the predicted joint coordinates of the target hand into the MANO model to obtain the reference target bone length. The reference target bone length vector is arranged according to a specific bone connection order, including 15 bone segments extending from the wrist towards each finger joint; to eliminate individual differences and the influence of dimensions, the shape parameters corresponding to each candidate particle are... Input the data into the MANO shape generator to generate the particle's three-dimensional joint coordinates. ;

[0149] S230: Let the parent-child node numbers of each bone be... , Represents the skeletal coordinates of the parent node. Representing the skeletal coordinates of the child node, then the first... The bone length vector of each particle The calculation is as follows:

[0150] ;

[0151] The goal is to normalize the skeletal structure vector, therefore, from the target skeletal structure vector... A specific bone segment length is selected as the reference length. In each round of optimization, the bone length vector generated from the candidate particles Extract its corresponding segment length Used to normalize the structure of the current particle:

[0152] ;

[0153] in, This represents the normalized current bone vector. Represents the normalized target skeleton vector;

[0154] This avoids the influence of variations in the operator's overall hand size, focusing solely on fitting the hand shape's structural proportions. Furthermore, to guide particles to converge to a hand shape that matches the target structure, the following fitness function is designed. :

[0155] ;

[0156] in, This is due to skeletal structure matching error. The regularization term is used to suppress shape parameter divergence. The weights are regularized; the fitness function values ​​of the entire population are calculated in batches using a vectorized method, and the optimal positions of individuals and the global optimum are selected for the next iteration;

[0157] S240: The standard particle swarm optimization formula is used for search iteration. In each iteration, the... The position and velocity of each particle are updated according to the following formula:

[0158] ;

[0159] ;

[0160] in, Inertia factor There are two learning factors. For independently sampled random vectors, Indicates uniform distribution. Represents particles The location of the optimal solution found in each iteration is its individual historical best record; It represents the smallest of the current best values ​​of all individuals in the population, and represents the global optimal solution of the particle swarm.

[0161] After each iteration, the individual optimal position and fitness of the particles are updated. If the global optimal solution does not show a significant improvement compared to the previous round, the search is terminated early. The final optimal particle position is obtained. This refers to the optimized human hand shape parameters, which are then used to generate the bone length. With target structure Perform error alignment analysis to quantify the fitting accuracy:

[0162] ;

[0163] in, Indicates the optimal particle position Length of generated bones With target structure The L2 norm error between them.

[0164] This method can converge quickly with fewer iterations and significantly improve the structural accuracy of the 3D hand while maintaining model compactness.

[0165] In one embodiment, S300 includes:

[0166] S310: After optimizing the shape parameters of the human hand, the position parameters of the human hand can be optimized. First, based on the input template joint data... and target joint data Construct vector sets from the root joint to each finger base joint in both the template joint data and the target joint data. and :

[0167] ；

[0168] ；

[0169] in, These represent the three-dimensional coordinates of joints 0, 1, 5, 9, 13, and 17 in the template joint, respectively. These represent the three-dimensional coordinates of joints 0, 1, 5, 9, 13, and 17 in the target joint, respectively.

[0170] S320: Construct the weighted covariance matrix :

[0171] ；

[0172] in, These are the weighting coefficients for the thumb, index finger, middle finger, ring finger, and little finger, respectively. This represents a diagonal matrix; in this embodiment, since the coordinate system is subsequently established based on the middle and ring fingers, the middle and ring fingers are assigned the highest weights. Furthermore, since the thumb involves the calculation of two degrees of freedom, it is also assigned a high weight. In summary, this invention is designed... The values ​​are 0.9, 0.8, 1.0, 1.0, and 0.8, respectively.

[0173] S330: For the matrix Perform singular value decomposition:

[0174] ;

[0175] in, Describes a left singular vector matrix. Represents a singular value diagonal matrix. Represents a right singular vector matrix;

[0176] S340: Based on the left singular vector matrix and right singular vector matrix Calculate the rotation matrix of the root joint. This is to achieve initial alignment between the template joint and the target joint.

[0177] .

[0178] Furthermore, based on the global rotation of the root node, the kinematic topology of the human hand is further utilized to calculate the rotation matrix of each joint in the global coordinate system step by step from the root node, so as to achieve complete human hand posture modeling.

[0179] In one embodiment, S400 includes:

[0180] S410: Let the three-dimensional position of the template hand joint be... The predicted joint position is ,in This indicates the joint number, with 0 representing the root node of the wrist; for each joint... The global rotation matrix and position of its parent node are respectively represented as: , Then, local coordinate transformation and bone alignment are performed to convert the predicted joint position into its position relative to the parent node in the parent node's local coordinate system, as shown below:

[0181] ;

[0182] in, The relative position after conversion It is the inverse of the global rotation matrix of the parent node. For the prediction results, the first The position coordinates of each joint;

[0183] At the same time, the bone vector in the template space The calculation is as follows:

[0184] ;

[0185] in, For joints Location coordinates, The coordinates of its parent joint;

[0186] S420: Subsequently, the rotation axis and rotation angle are calculated. The template bone is rotated to the predicted bone using a two-vector rotation method to obtain its rotation axis. and rotation angle They are respectively:

[0187] ;

[0188] ;

[0189] in, It is an infinitesimal quantity to prevent division by zero;

[0190] S430: Construct a bone alignment rotation matrix based on the rotation axis and rotation angle. Further combine this with the matrix of rotation around the bone direction, and multiply the two to obtain the rotation matrix of the child node relative to the parent node. for:

[0191] ;

[0192] Here, Rodrigues represents the Rodrigues transformation, which converts a description in the form of rotation axis–rotation angle into a rotation matrix. Indicates the direction around the template skeleton The additional rotation angle, ultimately, is determined based on the parent node's global rotation matrix. This yields the rotation matrix of the child nodes in global coordinates. for:

[0193] ;

[0194] It can also be based on the global position of the parent node. The global position of the child nodes is calculated recursively. :

[0195] ;

[0196] S440: After completing the rotation calculations for all joints except the fingertips, construct the pose rotation matrix set. The optimized shape parameters are input into the MANO model to obtain the complete pose information of 21 points of the hand.

[0197] Since the MANO model hand contains rich gesture information, this method can further reduce the impact of misidentification and data mutation by combining it with the actual motion constraints of the human hand itself.

[0198] In one embodiment, S500 includes:

[0199] S510: Establish a local coordinate system for the human hand, selecting the wrist as the origin, and choosing the normalized vectors of the vectors formed by the wrist origin, the middle finger origin (3), and the ring finger origin (4). The direction is obtained by taking the cross product of the vector from the wrist base point and the middle finger base point 3 with the vector from the wrist base point and the ring finger base point 4, and then normalizing it. The direction is then determined by the right-hand screw rule, which gives the direction of lateral movement of the palm. The direction, specifically:

[0200] ;

[0201] in, This represents the vector formed by the wrist base point and the middle finger base point 3. This represents the vector formed by the wrist base point pointing to the ring finger base point 4;

[0202] Then all manpower data is converted to the local coordinate system of the manpower in real time:

[0203] ;

[0204] in, This represents the position in the transformed local coordinate system. Represents the original global coordinates. Represents the global coordinates of the wrist pivot point;

[0205] S520: Because the dexterous hand has a total of 6 degrees of freedom, and the four fingers other than the thumb can only bend and not lateralize, meaning that only the degree of finger bending can be controlled, therefore, the degree of freedom for bending of each finger other than the thumb is... The design calculation formula is as follows:

[0206] ;

[0207] in, The three-dimensional vector representing the fingertip relative to the corresponding finger base point. The projection onto a plane, because all fingers except the thumb can only move in a plane perpendicular to the x-axis. This indicates the projection direction; simultaneously, in order to convert it into the input data format required by the time-dependent dexterous hand, a control variable conversion was performed to obtain the final dexterous hand control variable. :

[0208] ;

[0209] For the bending degree of freedom of the thumb, the projection vector is obtained by projecting the three-dimensional vector of the thumb tip relative to the base point onto the plane formed by the vector from the base point to the thumb's base point 6 and the z-axis. Then, the angle between the projection vector and the vector from the base point to the thumb's base point 6 is calculated to obtain the bending degree of freedom of the thumb. :

[0210] ;

[0211] in, This represents the projection vector obtained above. The vector represents the distance from the base point to the base point of the thumb (point 6). For the lateral swing degree of freedom of the thumb, the projection vector is obtained by projecting the three-dimensional vector of the thumb tip relative to the base point onto the xy plane, and then calculating the projection vector. With reference vector The included angle between them gives the lateral freedom of the thumb. :

[0212] ;

[0213] Similarly, these control values ​​are converted into the input data format of the dexterous hand:

[0214] ;

[0215] .

[0216] Specifically, the mapping module of this invention employs parametric modeling design, enabling automatic adaptation to the degree-of-freedom distribution and initial posture of different dexterous hands through a single offline calibration. For example, for dexterous hands with multi-joint structures (such as Shadow Hand and DART), a customized mapping function can be constructed based on their control interface and mechanical limit parameters, thereby supporting rapid replacement of dexterous hand models without retraining the network or adjusting the system framework. The real-time 3D human hand posture reconstruction and dexterous hand mapping effect based on virtual hand-driven technology is shown below. Figure 3 and Figure 4 The image shows a frame captured during real-time operation. (a) represents the real-time input of the human hand, (b) represents the virtual hand inferred in real-time based on the human hand's posture, and (c) represents the result mapped onto the dexterous hand in real-time. The image illustrates the changes in the various degrees of freedom of the dexterous hand when it grasps an object. Figure 5-10 The figures show the effects of varying degrees of freedom in index finger bending, middle finger bending, ring finger bending, little finger bending, thumb bending, and thumb lateral swinging. The vertical axis represents the dexterity control value obtained through human hand posture mapping, ranging from 0 to 1000, without a specific unit.

[0217] Furthermore, in the overall system implementation, a dual-thread structure is adopted for task separation. The image perception thread independently runs joint recognition and posture calculation, and transmits control data to the main control thread through a shared memory caching mechanism. The main control thread completes motion interpolation and encapsulates lower-level machine control commands, and further achieves real-time communication with the dexterous hand controller via a serial port. Specifically, non-blocking data transmission is used first to avoid low control data frequency caused by the low frequency of the camera itself, which could lead to stuttering movements of the dexterous hand. A double buffering mechanism is used to achieve 60Hz control.

[0218] In one embodiment, S600 includes:

[0219] S610: Uses non-blocking data transmission for adjacent time intervals. Data Linear interpolation between them is performed to obtain Control quantity of interpolation time t between :

[0220] ;

[0221] When a sudden change in motion is detected, that is... When the value is obtained, the value from the previous moment is automatically used as the current value.

[0222] S620: Due to the unavoidable structural differences between the human hand and the dexterous hand, it is essential to improve the safe control of the dexterous hand during teleoperation demonstrations. This invention achieves this goal by introducing force feedback control, preventing the downward bending control data of the dexterous hand from decreasing after it has stably grasped the target object. The following force-position hybrid control strategy is adopted: once the current value representing the finger force reaches a current threshold, the current bending angle of the dexterous hand's fingers is maintained; when the current value is less than the threshold, normal movement is performed to achieve safe and stable grasping operations.

[0223] ;

[0224] in, This indicates the input angle obtained through manual recognition and calculation. This indicates the current actual bending state of the dexterous hand. This represents the final input value for dexterous hand control. Represents the proportionality coefficient. Indicates the current current value. Indicates the current threshold;

[0225] S630: To address failed operations or accidental contact in complex grasping scenarios, a grasping fault-tolerance strategy based on feedback verification is designed. After each grasping operation, the consistency between the target current value and the trigger threshold within a time window is compared to determine whether it is a genuine contact. If a short-term contact-release-re-contact oscillation occurs, the system will automatically enter a damped mode, maintaining the current state until the next operation intention is generated, ensuring operational continuity and safety.

[0226] The aforementioned human-computer mapping and dexterity operation method based on virtual hand-driven approach forms a complete closed loop from hand perception to dexterity hand control. Image normalization combined with a multi-task network improves the accuracy and robustness of joint prediction; dynamic filtering and particle swarm optimization shape parameter fitting adapt to individual hand differences and reduce the influence of dimensions; weighted singular value decomposition and recursive rotation matrix calculation ensure the accuracy of hand posture mapping; a local hand coordinate system and a targeted angle mapping function accurately match the dexterity hand's degree of freedom distribution; double-buffered interpolation solves the low frame rate problem; current threshold control enables mixed adjustment of force and position to avoid excessive force operation; and oscillation damping mode ensures operational continuity. This solution balances low cost, high precision, strong adaptability, and safety, providing efficient and reliable technical support for human-computer interaction.

[0227] In one embodiment, a human-computer mapping and dexterous operating system based on a virtual hand is also provided, comprising:

[0228] The hand 3D joint perception module is used to normalize and resize hand images acquired by a monocular RGB camera, and input the resized images into a multi-task deep neural network to simultaneously predict the 3D coordinate information of 21 joints of the human hand; a dynamic filtering mechanism is used to smooth the joint time series data.

[0229] The virtual hand model parameter fitting module is used to fit the shape parameters of the MANO hand model using the particle swarm optimization algorithm, with the objective function being the minimization of the normalized bone length error, combined with regularization constraints to suppress parameter divergence.

[0230] The root joint rotation matrix calculation module is used to calculate the root joint rotation matrix based on weighted singular value decomposition: construct a set of weighted vectors for the template joint and the target joint, generate a covariance matrix by specifying weight coefficients and perform singular value decomposition to obtain the initial rotation alignment matrix;

[0231] The global rotation matrix calculation module for non-finger joints is used to recursively calculate the global rotation matrix of non-finger joints using the topology of the hand and biomechanical constraints: the predicted joint position is transformed to the local coordinate system of the parent node, the relative rotation matrix of the child node is solved by the vector rotation relationship between the template bone and the predicted bone, and the global rotation matrix is ​​generated by combining the Rodrigues transformation.

[0232] The Joint Angle-Dexterity Hand Control Quantity Mapping Module is used to establish a local coordinate system for the hand with the wrist as the origin and the coordinate axes determined by the middle and ring finger directions and cross product. Based on the degree of freedom distribution of the dexterity hand, a finger bending angle mapping function is designed to convert joint angles into dexterity hand control quantities.

[0233] The dexterous hand control signal processing and execution module is used to linearly interpolate the low frame rate perception results through a double buffering mechanism, convert them into a preset control frequency input, and integrate the dexterous hand end current data. When the current exceeds the threshold, the current position is locked; otherwise, the control is based on the mapping result, thus realizing a hybrid control of force and position.

[0234] For specific limitations regarding a human-computer mapping and dexterous operating system based on virtual hand-driven operation, please refer to the limitations of a human-computer mapping and dexterous operation method based on virtual hand-driven operation described above, which will not be repeated here. The various modules in the aforementioned human-computer mapping and dexterous operating system based on virtual hand-driven operation can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independent of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0235] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of a human-computer mapping and dexterity operation method based on virtual hand-driven operation.

[0236] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a human-computer mapping and dexterity operation method based on a virtual hand-driven approach.

[0237] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0238] The present invention has provided a detailed description of a human-computer mapping and dexterous operation method and system based on virtual hand-driven operation. Specific examples have been used to illustrate the principles and implementation methods of the invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the core ideas of the invention. It should be noted that those skilled in the art can make various improvements and modifications to the invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. A human-computer mapping and dexterous operation method based on virtual hand-driven operation, characterized in that, The method includes the following steps: S100: Normalizes and resizes hand images acquired by a monocular RGB camera, inputs the resized images into a multi-task deep neural network, and simultaneously predicts the three-dimensional coordinate information of 21 joints of the human hand; a dynamic filtering mechanism is used to smooth the joint time series data. S200: Fits the shape parameters of the MANO hand model using a particle swarm optimization algorithm, with the objective function being the minimization of normalized bone length error, combined with regularization constraints to suppress parameter divergence; S200 includes: S210: Let the number of particle swarm iterations be... Population size is Each particle represents a 10-dimensional chiral parameter vector. Its value range is The initial position of each particle is The initial velocity is And generate the position matrix of the current group. With velocity matrix : ； S220: Input the predicted joint coordinates of the target hand into the MANO model to obtain the reference target bone length. The reference target bone length vector is arranged according to a specific bone connection order, including 15 bone segments extending from the wrist towards each finger joint; to eliminate individual differences and the influence of dimensions, the shape parameters corresponding to each candidate particle are... Input the data into the MANO shape generator to generate the particle's three-dimensional joint coordinates. ; S230: Let the parent-child node numbers of each bone be... , Represents the skeletal coordinates of the parent node. Representing the skeletal coordinates of the child node, then the first... The bone length vector of each particle The calculation is as follows: ； The goal is to normalize the skeletal structure vector, therefore, from the target skeletal structure vector... A specific bone segment length is selected as the reference length. In each round of optimization, the bone length vector generated from the candidate particles Extract its corresponding segment length Used to normalize the structure of the current particle: ； in, This represents the normalized current bone vector. Represents the normalized target skeleton vector; To guide the particles to converge to a chiral shape that matches the target structure, the following fitness function is designed. : ； in, This is due to skeletal structure matching error. The regularization term is used to suppress shape parameter divergence. The weights are regularized; the fitness function values ​​of the entire population are calculated in batches using a vectorized method, and the optimal positions of individuals and the global optimum are selected for the next iteration; S240: The standard particle swarm optimization formula is used for search iteration. In each iteration, the... The position and velocity of each particle are updated according to the following formula: ； ； in, Inertia factor There are two learning factors. For independently sampled random vectors, Indicates uniform distribution. Represents particles The location of the optimal solution found in each iteration is its individual historical best record; It represents the smallest of the current best values ​​of all individuals in the population, and represents the global optimal solution of the particle swarm. After each iteration, the individual optimal position and fitness of the particles are updated. If the global optimal solution does not show a significant improvement compared to the previous round, the search is terminated early. The final optimal particle position is obtained. This refers to the optimized human hand shape parameters, which are then used to generate the bone length. With target structure Perform error alignment analysis to quantify the fitting accuracy: in, Indicates the optimal particle position Length of generated bones With target structure The L2 norm error between them; S300: Calculate the root joint rotation matrix based on weighted singular value decomposition: Construct a set of weighted vectors for the template joint and the target joint, generate a covariance matrix by specifying weight coefficients and perform singular value decomposition to obtain the initial rotation alignment matrix; S400: Recursively calculates the global rotation matrix of non-finger joints using hand topology and biomechanical constraints: the predicted joint position is transformed to the local coordinate system of the parent node, the relative rotation matrix of the child node is solved by the vector rotation relationship between the template bone and the predicted bone, and the global rotation matrix is ​​generated by combining the Rodrigues transformation. S500: Establishes a local coordinate system for the hand with the wrist as the origin and the coordinate axes determined by the cross product of the middle and ring fingers. Based on the degree-of-freedom distribution of the dexterous hand, it designs a finger bending angle mapping function to convert joint angles into dexterous hand control quantities. S500 includes: S510: Establish a local coordinate system for the human hand, selecting the wrist as the origin, and selecting the normalized vectors of the vectors formed by the wrist origin, the middle finger root joint 3, and the ring finger root joint 4, respectively, as... The direction is obtained by taking the cross product of the vector from the wrist base point and the middle finger base point 3 with the vector from the wrist base point and the ring finger base point 4, and then normalizing it. The direction is then determined by the right-hand screw rule, which gives the direction of lateral movement of the palm. The direction, specifically: ； in, This represents the vector formed by the wrist base point and the middle finger base point 3. This represents the vector formed by the wrist base point pointing to the ring finger base point 4; Then all manpower data is converted to the local coordinate system of the manpower in real time: ； in, This represents the position in the transformed local coordinate system. Represents the original global coordinates. Represents the global coordinates of the wrist pivot point; S520: Since a dexterous hand has a total of 6 degrees of freedom, and the four fingers other than the thumb can only bend and not lateralize, meaning control can only be exercised over the degree of finger bending, the degree of freedom for bending each finger other than the thumb is... The design calculation formula is as follows: ； in, The three-dimensional vector representing the fingertip relative to the corresponding finger base point. The projection onto a plane, because all fingers except the thumb can only move in a plane perpendicular to the x-axis. This indicates the projection direction; simultaneously, in order to convert it into the input data format required by the dexterous hand, the control quantity was converted to obtain the final dexterous hand control quantity. : ； For the bending degree of freedom of the thumb, the projection vector is obtained by projecting the three-dimensional vector of the thumb tip relative to the base point onto the plane formed by the vector from the base point to the thumb's base point 6 and the z-axis. Then, the angle between the projection vector and the vector from the base point to the thumb's base point 6 is calculated to obtain the bending degree of freedom of the thumb. : ； in, This represents the projection vector obtained above. The vector represents the distance from the base point to the base point of the thumb (point 6). For the lateral swing degree of freedom of the thumb, the projection vector is obtained by projecting the three-dimensional vector of the thumb tip relative to the base point onto the xy plane, and then calculating the projection vector. With reference vector The included angle between them gives the lateral freedom of the thumb. : ； Similarly, these control values ​​are converted into the input data format of the dexterous hand: ； ； S600: Through a double buffering mechanism, the low frame rate perception results are linearly interpolated and converted into a preset control frequency input. It integrates the current data of the dexterous hand end. When the current exceeds the threshold, the current position is locked; otherwise, it is controlled according to the mapping result, thus realizing the mixed control of force and position.

2. The method according to claim 1, characterized in that, In the S100, the hand images captured by the monocular RGB camera are normalized and resized, specifically as follows: The input image is cropped and resized to the resolution required by the model, and then pixel normalization at the channel level is performed as shown in the following formula: ； in, This represents the normalized image tensor. This indicates the resized image in the channel. ,Location( The pixel value at () It is the first The channel's preset mean. It is the first The preset standard deviation of the channel, The three color channels of digital image color data; The adjusted image in S100 is input into a multi-task deep neural network to predict the 3D coordinate information of 21 joints of the human hand, including: The processed image is used to extract image features through a ResNet50 backbone network, and a lightweight CNN is used to generate 2D heatmaps for 21 joints. Each heatmap represents the probability distribution of the corresponding joint point in the image. Based on the feature maps and 2D heatmaps, 3D position maps and skeletal orientation maps are further predicted. Finally, by combining camera intrinsic parameters and reference bone length, the absolute depth of the root joint is calculated through geometric constraints to recover the global 3D position and obtain the 3D coordinates of the 21 joints of the human hand. S100 employs a dynamic filtering mechanism to smooth the timing data of joint positions, specifically as follows: ； in, Indicates time The original input signal, Indicates time The filtered output signal, Indicates time The smoothing coefficient of the filtered output signal Changes with speed: ； ； in, The minimum cutoff frequency, For speed sensitivity coefficient, The absolute value of the instantaneous velocity of the input signal. The time interval between two signals. It is an intermediate variable that is dynamically adjusted with speed.

3. The method according to claim 2, characterized in that, The S300 includes: S310: Based on the input template joint data and target joint data Construct vector sets from the root joint to each finger base joint in both the template joint data and the target joint data. and : ； ； in, These represent the three-dimensional coordinates of joints 0, 1, 5, 9, 13, and 17 in the template joint, respectively. These represent the three-dimensional coordinates of joints 0, 1, 5, 9, 13, and 17 in the target joint, respectively. S320: Construct the weighted covariance matrix : ； in, These are the weighting coefficients for the thumb, index finger, middle finger, ring finger, and little finger, respectively. Represents a diagonal matrix; S330: For the matrix Perform singular value decomposition: ； in, Describes a left singular vector matrix. Represents a singular value diagonal matrix. Represents a right singular vector matrix; S340: Based on the left singular vector matrix and right singular vector matrix Calculate the rotation matrix of the root joint. This is to achieve initial alignment between the template joint and the target joint. 。 4. The method according to claim 3, characterized in that, The S400 includes: S410: Let the three-dimensional position of the template hand joint be... The predicted joint position is ,in This indicates the joint number, with 0 representing the root node of the wrist; for each joint... The global rotation matrix and position of its parent node are respectively represented as: , Then, local coordinate transformation and bone alignment are performed to convert the predicted joint position into its position relative to the parent node in the parent node's local coordinate system, as shown below: ； in, The relative position after conversion It is the inverse of the global rotation matrix of the parent node. For the prediction results, the first The position coordinates of each joint; At the same time, the bone vector in the template space The calculation is as follows: ； in, For joints Location coordinates, The coordinates of its parent joint; S420: Subsequently, the rotation axis and rotation angle are calculated. The template bone is rotated to the predicted bone using a two-vector rotation method to obtain its rotation axis. and rotation angle They are respectively: ； ； in, It is an infinitesimal quantity to prevent division by zero; S430: Construct a bone alignment rotation matrix based on the rotation axis and rotation angle. Further combine this with the matrix of rotation around the bone direction, and multiply the two to obtain the rotation matrix of the child node relative to the parent node. for: ； Here, Rodrigues represents the Rodrigues transformation, which converts a description in the form of rotation axis–rotation angle into a rotation matrix. Indicates the direction around the template skeleton The additional rotation angle, ultimately, is determined based on the parent node's global rotation matrix. This yields the rotation matrix of the child nodes in global coordinates. for: ； It can also be based on the global position of the parent node. The global position of the child nodes is calculated recursively. : ； S440: After completing the rotation calculations for all joints except the fingertips, construct the pose rotation matrix set. The optimized shape parameters are input into the MANO model to obtain the complete pose information of 21 points of the hand.

5. The method according to claim 4, characterized in that, The S600 includes: S610: Uses non-blocking data transmission for adjacent time intervals. Data Linear interpolation between them is performed to obtain Control quantity of interpolation time t between : ； When a sudden change in motion is detected, i.e. When the value is obtained, the value from the previous moment is automatically used as the current value. S620: Employs a force-position hybrid control strategy, maintaining the current bending angle of the dexterous hand's fingers once the current value representing finger force reaches the current threshold, and performing normal movements when the current value is less than the threshold, thus achieving safe and stable grasping operations. ； in, This indicates the input angle obtained through manual recognition and calculation. This indicates the current actual bending state of the dexterous hand. This represents the final dexterous hand control input value. Represents the proportionality coefficient. Indicates the current current value. Indicates the current threshold; S630: After each grabbing operation, the system determines whether it is a real contact by comparing the time window between the target current value and the trigger threshold. If a short-term contact-release-re-contact oscillation occurs, the system will automatically enter the damping mode to maintain the current state until the next operation intention is generated, ensuring the continuity and safety of the operation.

6. A human-computer mapping and dexterous operating system based on virtual hand-driven operation, performing the method as described in any one of claims 1 to 5, characterized in that, include: The hand 3D joint perception module is used to normalize and resize hand images acquired by a monocular RGB camera, and input the resized images into a multi-task deep neural network to simultaneously predict the 3D coordinate information of 21 joints of the human hand; a dynamic filtering mechanism is used to smooth the joint time series data. The virtual hand model parameter fitting module is used to fit the shape parameters of the MANO hand model using the particle swarm optimization algorithm, with the objective function being the minimization of the normalized bone length error, combined with regularization constraints to suppress parameter divergence. The root joint rotation matrix calculation module is used to calculate the root joint rotation matrix based on weighted singular value decomposition: construct a set of weighted vectors for the template joint and the target joint, generate a covariance matrix by specifying weight coefficients and perform singular value decomposition to obtain the initial rotation alignment matrix; The global rotation matrix calculation module for non-finger joints is used to recursively calculate the global rotation matrix of non-finger joints using the topology of the hand and biomechanical constraints: the predicted joint position is transformed to the local coordinate system of the parent node, the relative rotation matrix of the child node is solved by the vector rotation relationship between the template bone and the predicted bone, and the global rotation matrix is ​​generated by combining the Rodrigues transformation. The Joint Angle-Dexterity Hand Control Quantity Mapping Module is used to establish a local coordinate system for the hand with the wrist as the origin and the direction of the coordinate axes determined by the index and cross product of the middle and ring fingers. Based on the degree of freedom distribution of the dexterity hand, a finger bending angle mapping function is designed to convert the joint angle into dexterity hand control quantities. The dexterous hand control signal processing and execution module is used to linearly interpolate the low frame rate perception results through a double buffering mechanism, convert them into a preset control frequency input, and integrate the dexterous hand end current data. When the current exceeds the threshold, the current position is locked; otherwise, the control is based on the mapping result, thus realizing a hybrid control of force and position.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images