General dexterous hand gesture control method based on biological perspective proxy

Abstract

The invention discloses a general dexterous hand gesture control method based on biological perspective agency, and aims to realize decoupling of a control strategy and a specific hand skeleton by introducing a flexion angle and an abduction angle with definite biological semantics as an intermediate agent layer. The same set of control algorithm can adaptively drive dexterous hand models with different forms and different sizes; meanwhile, by utilizing a real-time state fitting technology based on a virtual sensor, a natural dexterous hand grabbing gesture which accords with a biological movement rule and is free of mold penetration can be quickly generated under the condition that long time sequence dependence is not needed, so that gesture control with high universality, high real-time performance and good interpretability is realized.

Description

Technical Field

[0001] This invention relates to a universal dexterity hand gesture control method based on a biological perspective agent. Background Technology

[0002] With the rapid development of virtual reality, digital humans, robotic dexterous hands, and human-computer interaction technologies, gesture-based fine manipulation and grasping control have become one of the key issues in current research and applications. Existing gesture generation and dexterous hand control methods typically rely on the parametric representation of specific hand models, such as mesh models or joint parameter models based on fixed topology. These methods are strongly bound to specific hand shapes, making it difficult to transfer between different hand models and resulting in poor versatility.

[0003] Meanwhile, many existing gesture generation methods directly use the joint angles or mesh shapes of specific hand models as control targets, lacking a unified intermediate representation layer. This means that when the size, skeletal structure, or number of joints of the hand model changes, the control strategy needs to be redesigned or the model needs to be retrained, which seriously restricts the promotion and application of dexterous hand technology in multi-hand models and multiple application scenarios.

[0004] In addition, existing gesture representation methods mostly use Euler angles, rotation matrices or high-dimensional mesh parameters, which have unintuitive physical meanings and poor interpretability. They are difficult to directly map to the drive joints of the robot's dexterous hand, and are not conducive to achieving natural, stable gestures that conform to human biological constraints. They are prone to problems such as unnatural bending and joint crossing, which affect the reliability and realism of dexterous hand operations.

[0005] In the prior art, the closest approach to this invention is a gesture generation method based on a parametric hand model. This type of method describes joint angles and gesture morphology using hand model parameters with a fixed topological structure, and generates grasping postures using deep learning or optimization algorithms. However, this type of method typically relies on a specific hand model, making it difficult to directly transfer the generated gestures to other hand models. It exhibits strong model dependency, requiring retraining or parameter adjustment when the hand model structure changes. This results in poor versatility, and the parameter representations lack clear biological meaning, hindering unified control of dexterous hands across devices or robots.

[0006] Patent CN202511872051.8, "Robot Dexterous Hand Control Method, Device, and Medium Based on Interference Detection," proposes a dexterous hand motion planning method based on an interference detection model (binary classification network). This method detects whether mechanical collisions occur during joint state vector transitions and uses an optimization algorithm to regenerate the motion path, achieving interference-free control with low computational power. However, this method heavily relies on the geometric parameters of a specific robotic hand structure for training, lacking compatibility and universality for hand models of different sizes and topologies. The algorithm fails once the hand model changes. Furthermore, its control method based on numerical joint vectors lacks biological semantic interpretability, failing to achieve natural transfer and manipulation of cross-morphological gestures through a unified biological proxy.

[0007] Patent CN202511674180.6 discloses a general dexterous hand action redirection method based on modular residual reinforcement learning. This patent proposes a reinforcement learning-based action redirection framework that receives structured decomposed MANO model parameters, utilizes multiple finger sub-policy networks to output initial action predictions, and combines a residual coordination network to globally correct local predictions, thereby driving the dexterous hand to complete actions. While this method can achieve action mapping from human hand to robot, its core relies heavily on a specific human parametric model (MANO) as the input benchmark, making it difficult to directly control non-standard human hand topologies (such as cartoon hands or irregularly shaped hands). Furthermore, its end-to-end reinforcement learning strategy lacks an intermediate representation layer with clear physical meaning, resulting in poor interpretability of the control logic. Moreover, when changing the target hand model, the strategy usually needs to be retrained, failing to achieve universal adaptation and instant transfer across hand models.

[0008] Patent CN202511473556.7, "Dexterous Hand Adaptive Grasping Method, Dexterous Hand Control System, and Storage Medium," proposes a multimodal adaptive grasping method combining visual point clouds and tactile perception. It extracts global features of the object's point cloud through a PointNet network and fuses simulated signals from tactile patches. A candidate grasping gesture set is generated using Glow-CVAE (Conditional Variational Autoencoder), and the target grasping posture is finally selected based on contact matching and stability indices. While this method improves the grasping stability of irregular or flexible objects through multimodal fusion, its generation model relies on a pre-set specific gesture training template. The generated output directly corresponds to the joint posture parameters of a specific hand model, resulting in strong coupling between the algorithm and the kinematic structure of a specific dexterous hand. This makes it impossible to directly transfer the algorithm to other hand models with significantly different skeletal structures (such as different fingerings or different degrees of freedom).

[0009] Current mainstream dexterous hand grasping and gesture generation technologies typically rely heavily on specific human parametric models (such as the MANO model) or fixed skeletal topologies. Because this control strategy is highly coupled with the geometry and number of joints of the specific hand model, the original control algorithms cannot be directly adapted when the application scenario switches to virtual hands of different sizes, cartoon hands with non-standard topologies, or dexterous hands of physical robots. Data must be re-acquired and the network retrained, significantly limiting the versatility and transferability of gesture control technology. Furthermore, existing gesture representation methods often employ high-dimensional rotation matrices, quaternions, or simple 3D joint coordinates. Since these representations lack intuitive biological and physical meaning and are difficult to apply anatomically consistent motion constraints, the generated grasping gestures are prone to unnatural phenomena such as joint reversal, finger clipping, or skeletal distortion, lacking interpretability and difficult to directly map to robot motor control signals. Simultaneously, to ensure the continuity of movements, many deep learning-based generative models require long historical frames as contextual information. Because the model relies on long-term time-series data, it causes significant delays during inference, making it difficult to meet the real-time requirements of interactive scenarios such as virtual reality or robot teleoperation. Summary of the Invention

[0010] To address the aforementioned issues of poor versatility, lack of biological constraints, and insufficient real-time performance, this invention proposes a universal dexterity hand gesture control method based on a biologically derived proxy. By introducing "flexion" and "abduction," which possess clear biological semantics, as intermediate proxy layers, the aim is to decouple the control strategy from the specific hand skeleton, enabling the same control algorithm to adaptively drive dexterity hand models of different shapes and sizes. Simultaneously, utilizing real-time state fitting technology based on virtual sensors, this invention can rapidly generate natural and seamless dexterity hand grasping gestures that conform to biological motion principles without long temporal dependencies, thus achieving highly versatile, real-time, and interpretable gesture control.

[0011] This invention can be achieved through the following technical solutions: A universal dexterous hand gesture control method based on a biological agent includes the following steps: Step S1: Data preprocessing and agent construction. Define the local coordinate system of the hand skeleton, decouple the hand posture into two biological angles: flexion angle and abduction angle, as a general control agent. The flexion angle mainly represents the bending movement of the fingers. Calculate the current skeleton vector in... The projection vector on the plane, which is related to The angle between axes is the flexion angle; the abduction angle mainly represents the opening and closing movements of the fingers. The abduction angle is calculated by the angle between the current skeletal vector and the above projection vector. Step S2: Virtual sensor perception, constructing environmental sensors, joint sensors and joint gap sensors on the current hand skeleton to collect geometric interaction features between the hand and objects; Step S3: Two-stage network inference, Phase 1: Based on the current sensor status, predict the "ideal state" suitable for capturing the target; Second stage: Based on the difference between the current state and the ideal state, generate the biological angle increment for the next frame; Step S4: Motion redirection and actuation, applying the generated biological perspective to the skeleton of the target hand model, and combining it with positive kinematics to reconstruct the final 3D grasping posture.

[0012] Furthermore, the construction of the local coordinate system in step S1: for any joint point of the hand With its parent node Construct a local coordinate system with the origin: Y-axis: defined as a unit vector along the direction of the parent node's bone; Z-axis: defined as the normal direction of the palm plane; X-axis: obtained by the cross product of the Y-axis and Z-axis, conforming to the right-hand rule.

[0013] Furthermore, the environmental sensors are arranged such that a sphere with a radius of [missing information] is constructed, centered on the base joint of the middle finger or other central joint of the hand model. The detection sphere; the sampling strategy is to generate the sample using a hexagonal dense stacking algorithm inside the sphere. Given a set of uniformly distributed base points, calculate the shortest Euclidean distance from each base point to the surface of the target object.

[0014] Furthermore, the joint sensor sets sampling points at each joint position of the hand model and at the middle position of each finger bone to calculate the shortest distance from these points on the bones to the surface of the object.

[0015] Furthermore, the joint gap sensor is connected to the same level joint of adjacent fingers to calculate the length of their connection.

[0016] Further, in step S4) From the biological perspective of each frame obtained through reasoning ( After that, universal control over different hand models is achieved through the following steps: Obtain target model parameters: Read the length of the bone segments of the target hand model that needs to be driven. ; Positive kinematic reconstruction: Starting from the root node (wrist), calculate downwards level by level; Using the reverse process defined in this invention, from a biological perspective ( Combining the Rodriguez rotation formula, it is transformed into a rotation matrix relative to the parent coordinate system; Combined with the actual bone length of the target model Calculate the coordinate position of the sub-joint in three-dimensional space; Grip tightness adjustment: For hand models with large knuckles, increasing the infill distance can prevent the fingers from sinking into the object; for hand models with slender knuckles, decreasing the infill distance can ensure close contact. Biological formula: in This is the index of the hand joint node currently being calculated; The current node The parent node index, for example, if fingertips, then It is the joint above the finger. Represents the local coordinate system of the parent joint; in and These are the X-axis and Y-axis unit vectors in the local coordinate system of the parent joint; the Y-axis usually points in the direction of the bone, from the parent node to the child node, and the X-axis is obtained by the cross product of the Y-axis and the Z-axis; in The current node The skeletal vector, i.e., from the parent node Point to the current node ; in It is the current skeleton vector The projection vector on the YZ plane of the parent joint local coordinate system is used to calculate the flexion angle, eliminating the influence of bone rotation about its own axis; in This is the buckling angle sign determination value, calculated using a scalar triple product. It is used to determine which side of the Y-axis the projection vector is located on, thereby determining the buckling angle. Positive and negative directions; in This is the abduction angle sign determination value, used to determine which side of the projection plane the actual bone vector is located on, thereby determining the abduction angle. Positive and negative directions, It is a sign function; it returns 1 if the input value is greater than 0, and -1 if it is less than 0. in It is the flexion angle, representing the flexion / extension movement of the joint, calculated by the parent joint's Y-axis and the projection vector. The angle between them; in It is the abduction angle, representing the left and right swing of the joint, and is calculated using the projection vector. Compared to real skeletal vectors The angle between them.

[0017] Beneficial effects This invention innovatively proposes an intermediate proxy representation method based on a biological perspective, and, combined with a three-dimensional virtual sensor system encompassing the environment, joints, and gaps, establishes a standardized mapping from the geometric features of an object to the biological motion features of the hand. Through a two-stage inference network constructed using a conditional variational autoencoder, the system can predict the ideal grasping posture in real time based on the current state and generate angle increments, thereby eliminating dependence on specific hand model topology and long-term temporal context data.

[0018] Therefore, this invention enables universal control of various dexterous hands, including human hands, robotic hands, and irregularly shaped hands, using a single control strategy without retraining the model. This significantly reduces the migration costs for cross-platform applications and solves the problem of poor versatility in existing technologies. Furthermore, the angle generation method based on biological constraints effectively avoids joint reverse bending and clipping phenomena. While ensuring the naturalness and physical interpretability of movements, it achieves low-latency real-time response, providing a novel solution with high versatility and real-time performance for virtual reality interaction and robotic dexterous hand control. Attached Figure Description

[0019] Figure 1 Flowchart of a general dexterous hand gesture control method from a biological perspective; Figure 2 A schematic diagram of the geometric definition principle for a biological proxy; Figure 3 This is a schematic diagram of the virtual sensor distribution and sampling principle. Detailed Implementation

[0020] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification.

[0021] This invention provides a general dexterous hand gesture control method based on a biological-angled proxy. This method does not rely on a specific hand mesh model or a single skeletal topology, but instead proposes an intermediate proxy representation layer called "Bio-Angle". By constructing a composite sensing system including environmental sensors, joint sensors, and joint gap sensors, and combining it with a Conditional Variational Autoencoder (cVAE) network, real-time mapping from object geometric features to hand biological motion features is achieved. The specific embodiments of this invention are described in detail below with reference to the accompanying drawings: The method of the present invention can be run in a computer device or an embedded robot controller that includes a processor (CPU), a graphics processor (GPU) and memory.

[0022] Input module: used to acquire 3D point cloud data or depth image data of the object to be grasped, as well as the skeletal parameters (bone length, joint hierarchy) of the current dexterous hand model.

[0023] Computation module: Executes the core algorithm of this invention, including the construction of virtual sensors, sampling, network inference, and kinematic reconstruction.

[0024] Output / Execution Module: Outputs a general biological angle control signal, which can directly drive a digital human hand model in a virtual environment, or drive the servo motors of a physical robot's dexterous hand after mapping.

[0025] Overall process description (in conjunction with) Figure 1 ) Figure 1 This is a flowchart illustrating the overall implementation of the method of the present invention. (For example...) Figure 1 As shown, the present invention mainly includes the following steps: Step S1: Data Preprocessing and Agent Construction. Define the local coordinate system of the hand skeleton and decouple the hand pose into two biological angles: flexion angle and abduction angle, which serve as a general control agent.

[0026] Step S2: Virtual sensor perception. Environmental sensors, joint sensors, and joint gap sensors are constructed on the current hand skeleton to collect geometric interaction features between the hand and objects.

[0027] Step S3: Two-stage network inference.

[0028] Phase 1 (FitNet): Based on the current sensor status, predict the "ideal state" suitable for grasping the target.

[0029] The second stage (GrabNet) generates the biological angle increment for the next frame based on the difference between the current state and the ideal state.

[0030] Step S4: Motion Redirection and Actuation. The generated biological perspective is applied to the skeleton of the target hand model (whether human, animal, or robotic), and the final 3D grasping posture is reconstructed using forward kinematics.

[0031] Detailed technical implementation scheme Figure 2 This invention presents a geometric definition diagram for a bio-Angle proxy. To achieve universal control across hand models, it abandons traditional Euler angles or quaternion representations and adopts a bio-Angle that is independent of bone length. The specific implementation is as follows: Local coordinate system construction: for any joint of the hand With its parent node Construct a local coordinate system with the origin.

[0032] Y-axis: Defined as a unit vector along the direction of the parent node's skeleton.

[0033] Z-axis: defined as the normal direction of the palm plane (determined by the cross product of the root vectors of the index finger and the middle finger).

[0034] X-axis: Obtained by the cross product of Y-axis and Z-axis, conforming to the right-hand rule.

[0035] Angular decoupling: Flexion ): Primarily represents the bending motion of the fingers. Calculates the current skeletal vector in... The projection vector on the plane, which is related to The angle between the axes is the buckling angle.

[0036] Abduction ( The abduction angle is mainly used to represent the opening and closing movements of the fingers. The angle between the current skeletal vector and the aforementioned projection vector is calculated.

[0037] Technical advantages: This representation method eliminates redundant degrees of freedom caused by the rotation of bones around their own axis (Twist), which not only conforms to the movement restrictions of human hand anatomy (avoiding reverse joint movements), but is also a set of dimensionless scalars that can be directly applied to skeletons of different sizes.

[0038] Construction of a composite virtual sensor system (combined) Figure 3 To perceive the spatial relationship between the hand and the object being grasped, this invention designs three virtual sensors: Ambient Sensor: Arrangement method: Using the base joint of the middle finger (or other central joint) of the hand model as the center, construct a sphere with a radius of... A probe sphere (e.g., 12cm).

[0039] Sampling strategy: The hexagonal close packing algorithm is used to generate samples inside the sphere. 1011 (e.g., 1011) uniformly distributed base points.

[0040] Perceptual data: Calculate the shortest Euclidean distance from each base point to the surface of the target object. This data encodes the global geometric contour and rough position information of the object relative to the palm.

[0041] Joints Sensor: Arrangement method: Sampling points are set at each joint of the hand model and at the middle of each finger bone.

[0042] Perceptual data: Calculates the shortest distance from points on these bones to the object's surface. This data is used to guide the fingertips and pads to precisely conform to the object's surface.

[0043] Joint gap sensor: Arrangement method: Connect the same joints of adjacent fingers (e.g., the first joint of the index finger and the first joint of the middle finger) and calculate the length of the line connecting them.

[0044] Function: This data serves as a physical constraint to prevent fingers from interlocking in the generated gestures.

[0045] All sensor data is normalized to eliminate the influence of the absolute size of the object on the network input.

[0046] Motion reconstruction and generalized control. From a biological perspective, each frame is obtained through inference (…). Then, the present invention achieves universal control over different hand models through the following steps: Obtain target model parameters: Read the skeletal segment lengths of the target hand model to be driven (e.g., a robotic hand with only three fingers). .

[0047] Positive kinematic reconstruction: Starting from the root node (wrist), calculate downwards level by level.

[0048] Using the reverse process defined in this invention, from a biological perspective ( Combining the Rodrigues' Rotation Formula, it is transformed into a rotation matrix relative to the parent coordinate system.

[0049] Combined with the actual bone length of the target model Calculate the coordinate position of the sub-joint in three-dimensional space.

[0050] Grip tightness adjustment: This invention also introduces an adjustable parameter—the fill distance. When calculating the sensor distance, the tightness of the grip can be controlled by manually increasing or decreasing an offset value. For example, for a hand model with large knuckles, increasing the fill distance prevents the fingers from sinking into the object; for a hand model with slender knuckles, decreasing the fill distance ensures close contact.

[0051] Biological formula (combined) Figure 2 illustrate): in This is the index of the hand joint node currently being calculated.

[0052] The current node The parent node index. For example, if fingertips, then It is the joint above that finger. Represents the local coordinate system of the parent joint.

[0053] in and These are the unit vectors for the X-axis and Y-axis in the local coordinate system of the parent joint. The Y-axis typically points in the direction of the bone (from the parent node to the child node), and the X-axis is obtained by the cross product of the Y-axis and the Z-axis.

[0054] in The current node The skeletal vector. That is, from the parent node. Point to the current node The vector.

[0055] in It is the current skeleton vector The projection vector onto the YZ plane of the parent joint's local coordinate system. This vector is used to calculate the flexion angle, eliminating the influence of bone rotation about its own axis.

[0056] in This is the buckling angle sign determination value. Calculated using a scalar triple product, it is used to determine which side of the Y-axis the projection vector is located on, thereby determining the buckling angle. The positive and negative directions.

[0057] in This is the abduction angle sign determination value. It is used to determine which side of the projection plane the actual bone vector is located on, thereby determining the abduction angle. The positive and negative directions. It is a sign function. It returns 1 if the input value is greater than 0, and -1 if the input value is less than 0.

[0058] in It is the flexion angle. It characterizes the flexion / extension movement of the joint. It is calculated by the parent joint's Y-axis and the projection vector. The angle between them.

[0059] in It is the abduction angle. It represents the lateral movement of the joint. It is calculated using the projection vector. Compared to real skeletal vectors The angle between them.

[0060] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A universal dexterous hand gesture control method based on biological agents, characterized in that, Includes the following steps: Step S1: Data preprocessing and agent construction. Define the local coordinate system of the hand skeleton and decouple the hand posture into two biological angles: flexion angle and abduction angle, which serve as a general control agent. The flexion angle mainly represents the bending movement of the fingers. Calculate the projection vector of the current skeleton vector on the plane. The angle between this projection vector and the axis is the flexion angle. The abduction angle mainly represents the opening and closing movements of the fingers. Calculate the angle between the current skeleton vector and the above projection vector. Step S2: Virtual sensor perception, constructing environmental sensors, joint sensors and joint gap sensors on the current hand skeleton to collect geometric interaction features between the hand and objects; Step S3: Two-stage network inference, Phase 1: Based on the current sensor status, predict the "ideal state" suitable for capturing the target; Second stage: Based on the difference between the current state and the ideal state, generate the biological angle increment for the next frame; Step S4: Motion redirection and actuation, applying the generated biological perspective to the skeleton of the target hand model, and combining it with positive kinematics to reconstruct the final 3D grasping posture.

2. The universal dexterous hand gesture control method based on biological perspective agent according to claim 1, characterized in that, The construction of the local coordinate system in step S1: for any joint point of the hand With its parent node Construct a local coordinate system for the origin: Y-axis: defined as a unit vector along the direction of the parent node's bone; Z-axis: Defined as the normal direction of the palm plane; X-axis: Obtained by the cross product of Y-axis and Z-axis, conforming to the right-hand rule.

3. The universal dexterous hand gesture control method based on biological perspective agent according to claim 1, characterized in that, The environmental sensors are arranged with the base joint of the middle finger or other central joint of the hand model as the center, constructing a sphere with a radius of [missing information]. The detection sphere; the sampling strategy is to generate the sample using a hexagonal dense stacking algorithm inside the sphere. Given a set of uniformly distributed base points, calculate the shortest Euclidean distance from each base point to the surface of the target object.

4. The universal dexterous hand gesture control method based on biological perspective agent according to claim 1, characterized in that, The joint sensor sets sampling points at each joint position of the hand model and at the middle position of each finger bone, and calculates the shortest distance from these points on the bones to the surface of the object.

5. The universal dexterous hand gesture control method based on biological perspective agent according to claim 1, characterized in that, The joint gap sensor is connected to the same level joint of adjacent fingers to calculate the length of the connection.

6. The universal dexterous hand gesture control method based on biological perspective agent according to claim 1, characterized in that, In step S4) From the biological perspective of each frame obtained through reasoning ( After that, universal control over different hand models is achieved through the following steps: Obtain target model parameters: Read the length of the bone segments of the target hand model that needs to be driven. ; Positive kinematic reconstruction: Starting from the root node, calculate downwards level by level; Using the reverse process defined in this invention, from a biological perspective ( Combining the Rodriguez rotation formula, it is transformed into a rotation matrix relative to the parent coordinate system; Combined with the actual bone length of the target model Calculate the coordinate position of the sub-joint in three-dimensional space; Grip tightness adjustment: For hand models with large knuckles, increasing the infill distance can prevent the fingers from sinking into the object; for hand models with slender knuckles, decreasing the infill distance can ensure close contact. Biological formula: in This is the index of the hand joint node currently being calculated; The current node The parent node index, for example, if fingertips, then It is the joint above the finger. Represents the local coordinate system of the parent joint; in and These are the X-axis and Y-axis unit vectors in the local coordinate system of the parent joint; the Y-axis usually points in the direction of the bone, from the parent node to the child node, and the X-axis is obtained by the cross product of the Y-axis and the Z-axis; in The current node The skeletal vector, i.e., from the parent node Point to the current node ; in It is the current skeleton vector The projection vector on the YZ plane of the parent joint local coordinate system is used to calculate the flexion angle, eliminating the influence of bone rotation about its own axis; in This is the buckling angle sign determination value, calculated using a scalar triple product. It is used to determine which side of the Y-axis the projection vector is located on, thereby determining the buckling angle. Positive and negative directions; in This is the abduction angle sign determination value, used to determine which side of the projection plane the actual bone vector is located on, thereby determining the abduction angle. Positive and negative directions, It is a sign function; it returns 1 if the input value is greater than 0, and -1 if it is less than 0. in It is the flexion angle, representing the flexion / extension movement of the joint, calculated by the parent joint's Y-axis and the projection vector. The angle between them; in It is the abduction angle, representing the left and right swing of the joint, and is calculated using the projection vector. Compared to real skeletal vectors The angle between them.

Images