Thumb posture estimation method based on multistage optimization and kinematics constraint and computing equipment

Abstract

The invention relates to a multi-stage optimization and kinematics constraint-based thumb posture estimation method and computing equipment. The method comprises the following steps of: acquiring original hand data, preprocessing the original hand data, and initializing parameters; constructing a multi-objective optimization function with multiple constraints, obtaining an initial estimation thumb posture based on the preprocessed data, taking the initial estimation thumb posture as an initial value, and performing recursive calculation by using a kinematics chain to obtain an accurate angle of each joint of the thumb; integrating the accurate thumb joint angle to complete hand posture parameters and outputting the parameters; a computing device is implemented based on the method. Complete and accurate thumb kinematic chain recovery under incomplete input is achieved, it is guaranteed that all estimated thumb joint angles conform to the human anatomy activity range, and high-fidelity basic data are provided for downstream application; the generated thumb motion sequence is continuous and smooth in the time dimension, and the visual experience in real-time interaction application is greatly improved; in an interactive application, a more natural, accurate and jitter-free gesture control experience can be realized.

Description

Technical Field

[0001] This invention relates to the technical field of general image data processing or generation, and in particular to a thumb pose estimation method and computing device based on multi-level optimization and kinematic constraints. Background Technology

[0002] In the field of augmented reality (AR) and virtual reality (VR) gesture interaction technology, building a high-precision, highly realistic hand posture training dataset is the cornerstone of achieving natural and immersive interaction. Current mainstream technologies heavily rely on vision-based data acquisition systems, such as depth cameras and optical motion capture. However, these systems face a fundamental technical bottleneck when processing complex hand movements, especially the fine movements of the thumb: precise multidimensional motion information of the thumb's root joint—the carpometacarpal joint (CMC)—is generally lacking in existing data streams. This deficiency directly leads to problems across the entire chain from data acquisition and model building to algorithm training, specifically manifested as follows:

[0003] (1) Traditional depth cameras and optical motion capture systems have significant blind spots when capturing the complex structure of the thumb with a unique kinematic chain. The carpal metacarpophalangeal joint (CMC) is a typical saddle joint with two main degrees of freedom: flexion and extension, and adduction and extension. However, existing equipment is difficult to stably and accurately calculate the spatial transformation matrix of the joint relative to the wrist base when faced with hand self-occlusion, rapid movement or limited field of view. This results in the loss or serious distortion of the most critical root degree of freedom information in the thumb kinematic chain, which poses a hidden danger for subsequent high-precision reconstruction.

[0004] (2) Due to the lack of accurate CMC joint information in the original data, the existing hand skeleton model often oversimplifies or idealizes the thumb kinematic chain, ignoring the independent and flexible movement ability of the first metacarpal bone. This results in deviations in the spatial relative position relationship between the reconstructed thumb and the palm and other fingers. The movement trajectory of the thumb when performing fine operations such as palm opposition and pinching cannot be realistically restored, which seriously restricts the naturalness and operation accuracy of gesture mapping in AR / VR applications.

[0005] (3) Due to the lack of root degrees of freedom in the original data, the algorithm for pose estimation or generating datasets based on it is difficult to simultaneously take into account the rationality of the global gesture and the detail of the local thumb pose. In addition, the lack of kinematic constraints makes it easy for the estimated or synthesized joint angles to exceed the range of human anatomical activity, resulting in a large number of "pseudo-poses" that violate physiological laws. The model has poor generalization ability and significantly damages the user experience and interaction credibility. Summary of the Invention

[0006] This invention solves the problems existing in the prior art and provides a thumb pose estimation method and computing device based on multi-level optimization and kinematic constraints.

[0007] The technical solution adopted in this invention is a thumb pose estimation method based on multi-level optimization and kinematic constraints. The method acquires raw hand data and performs preprocessing and parameter initialization.

[0008] A multi-objective optimization function with multiple constraints is constructed, and a preliminary estimate of thumb pose is obtained based on the preprocessed data.

[0009] Based on the initial estimated thumb posture as the initial value, the kinematic chain is used for recursive calculation to obtain the precise angles of each joint of the thumb.

[0010] The precise thumb joint angle is integrated into complete hand posture parameters and output.

[0011] Preferably, the preprocessing involves performing coordinate system unification processing on the hand model and constructing standardized single-gesture posture data.

[0012] Preferably, the multi-objective optimization function is,

[0013]

[0014] Where θ is the thumb joint angle vector, which includes 4 components. j=1,2,3,4 correspond to the two angles of the wrist-metacarpal joint, the angle of the metacarpophalangeal joint, and the angle of the interphalangeal joint, respectively.

[0015] This refers to the key point matching error term;

[0016] Joint restriction penalty item;

[0017] This is a time smoothness constraint term;

[0018] α, β, γ are weighting coefficients.

[0019] Preferably, satisfy,

[0020]

[0021] in, It is a positive kinematic function, obtained by inputting joint angles. Output the 3D coordinates of the i-th keypoint in the world coordinate system. ;

[0022] Let i be the target's three-dimensional coordinates in the world coordinate system for the i-th keypoint;

[0023] The weight coefficient for the i-th key point;

[0024] This represents the total number of key points.

[0025] Preferably, satisfy,

[0026]

[0027] in, and These are the upper and lower physiological limits of the j-th component, respectively.

[0028] Preferably, satisfy,

[0029]

[0030] Where t is the current time and t-1 is the previous time.

[0031] Preferably, the recursive calculation is performed using the kinematic chain.

[0032] The angle of the wrist-to-palm joint is decomposed into a combination of continuous rotation matrices around the vertical and horizontal axes. The angle of the metacarpophalangeal joint and the corresponding rotation matrix are calculated by vector projection. The angle of the interphalangeal joint is calculated based on the combination of continuous rotation matrices and the rotation matrix corresponding to the angle of the metacarpophalangeal joint.

[0033] Preferably, the calculated precise thumb joint angle is processed according to the preprocessing and converted into a preset standard output format.

[0034] Preferably, historical state data is updated for time smoothing constraints in the next frame.

[0035] A computing device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the thumb pose estimation method based on multi-level optimization and kinematic constraints.

[0036] This invention relates to a method and computing device for thumb posture estimation based on multi-level optimization and kinematic constraints. The method involves acquiring raw hand data and preprocessing it to initialize parameters; constructing a multi-objective optimization function with multiple constraints; obtaining a preliminary estimate of the thumb posture based on the preprocessed data as the initial value; performing recursive calculations using a kinematic chain to obtain precise angles of each thumb joint; integrating the precise thumb joint angles into complete hand posture parameters and outputting them; and implementing the computing device based on the method.

[0037] The beneficial effects of this invention are as follows:

[0038] (1) Based on multi-level optimization strategy and kinematic chain recursive calculation, the missing biaxial rotational degrees of freedom of CMC joint can be accurately compensated and reconstructed from partial observation data, effectively filling the gap in the underlying motion data and realizing complete and accurate thumb kinematic chain recovery under incomplete input.

[0039] (2) By explicitly introducing a joint angle physiological constraint penalty term into the multi-objective optimization function and embedding axial constraints in the kinematic recursion, it is fundamentally guaranteed that all estimated thumb joint angles conform to the range of human anatomical activity, avoiding the generation of "pseudo-postures" that violate physiological laws, and providing high-fidelity basic data for downstream applications;

[0040] (3) By introducing a time smoothness constraint term, the optimization results of historical frames are used as the prior of the current frame, which effectively suppresses the jitter and sudden changes in the estimation results between frames, making the generated thumb motion sequence continuous and smooth in the time dimension, greatly improving the visual experience in real-time interactive applications.

[0041] (4) By unifying the three constraints of key point spatial matching, joint physiological constraints and temporal continuity into an optimization framework, the stringent requirements on hardware systems and data acquisition costs are significantly reduced. This can directly improve the quality of hand posture training datasets, thereby helping downstream models such as gesture recognition and action prediction to obtain stronger generalization ability and robustness. In interactive applications such as AR / VR, a more natural, accurate and jitter-free gesture control experience can be achieved. Attached Figure Description

[0042] Figure 1 This is a flowchart of the method of the present invention;

[0043] Figure 2 This is a flowchart of the recursive calculation using a kinematic chain in this invention;

[0044] Figure 3 This is a schematic diagram of the missing wrist and palm joint in the application of the present invention. The left figure shows the target point at different times, and the right figure shows the rest state of the hand model. When the joints are corresponding, the missing joint is indicated by the red arrow. Detailed Implementation

[0045] The present invention will be further described in detail below with reference to embodiments, but the scope of protection of the present invention is not limited thereto.

[0046] This invention relates to a thumb pose estimation method based on multi-level optimization and kinematic constraints, employing a two-level progressive optimization strategy:

[0047] The first stage uses numerical optimization methods to initially estimate the thumb pose from incomplete observation data under multiple constraints;

[0048] The second stage performs refined recursive calculations based on the kinematic chain principle on the preliminary results;

[0049] By introducing a triple constraint mechanism of spatial matching, physiological constraints, and temporal continuity, the final generated thumb posture is ensured to be both accurate and in line with the laws of human movement.

[0050] The method includes the following steps:

[0051] (1) Obtain raw hand data and preprocess it, and initialize parameters;

[0052] (2) Construct a multi-objective optimization function with multiple constraints, and obtain a preliminary estimate of thumb pose based on the preprocessed data;

[0053] (3) Based on the preliminary estimated thumb posture as the initial value, the kinematic chain is used for recursive calculation to obtain the accurate angles of each joint of the thumb;

[0054] (4) Integrate the precise thumb joint angle into complete hand posture parameters and output them.

[0055] The method will be described below with reference to specific implementation methods.

[0056] (1) Obtain raw hand data and preprocess it, and initialize parameters;

[0057] The preprocessing involves standardizing the coordinate system of the hand model and constructing standardized single-gesture posture data.

[0058] Specifically, the coordinate system standardization process for the hand model involves converting the unit of the wrist transformation matrix from meters to millimeters to match the hand model's scale standard. The conversion formula is as follows:

[0059] T mm =1000×T m

[0060] Simultaneously, mirror symmetry processing is performed based on hand markings (left hand / right hand) to construct standardized single-gesture posture data.

[0061] Simultaneously, configure computing device parameters and initialize optimizer state.

[0062] (2) Construct a multi-objective optimization function with multiple constraints, and obtain a preliminary estimate of thumb pose based on the preprocessed data;

[0063] The multi-objective optimization function with multiple constraints here is a first-level optimization, constructed based on the observation data of the current frame and the historical state of the previous frame. It solves for the optimal joint angle θ within the physiological constraints of joint angles using a constrained nonlinear optimization algorithm. It should be noted that θ here includes a set of components. The multi-objective optimization function is...

[0064]

[0065] Where θ is the thumb joint angle vector, which includes 4 components. j=1,2,3,4 correspond to the two angles of the wrist-metacarpal joint, the angle of the metacarpophalangeal joint, and the angle of the interphalangeal joint, respectively.

[0066] This is the keypoint matching error term, used to ensure spatial consistency between the model's keypoints and the observed data;

[0067] These are joint restriction penalties used to impose physiological constraints and prevent joint angles from exceeding the normal range of the human body.

[0068] This is a temporal smoothness constraint term used to suppress jitter and maintain smooth motion between frames;

[0069] α, β, γ are weighting coefficients. In this embodiment, α=1, β=500, and γ=200.

[0070] Furthermore, the keypoint matching error term satisfy,

[0071]

[0072] in, It is a positive kinematic function, obtained by inputting joint angles. Output the 3D coordinates of the i-th keypoint in the world coordinate system. ;

[0073] The target's 3D coordinates in the world coordinate system for the i-th keypoint, provided by input data (such as multi-view images, depth maps, sensor data);

[0074] The weight coefficient for the i-th key point;

[0075] This represents the total number of key points.

[0076] Joint restriction penalty satisfy,

[0077]

[0078] in, and These are the upper and lower physiological limits of the j-th component, respectively;

[0079] This item is called "boundary violation penalty". A positive secondary penalty will only be generated when the angle value exceeds the preset range.

[0080] Time smoothness constraint satisfy,

[0081]

[0082] Where t represents the current time and t-1 represents the previous time. The thumb joint angle vector, optimized from the previous frame, is used as a historical reference for the current frame.

[0083] (3) Based on the preliminary estimated thumb posture as the initial value, the kinematic chain is used for recursive calculation to obtain the accurate angles of each joint of the thumb;

[0084] The recursive calculation using the kinematic chain here is the second-level optimization. It is based on the preliminary optimization results obtained from the first-level optimization, and uses the kinematic chain principle and axial constraints to perform recursive calculations to obtain accurate joint angles.

[0085] The recursive calculation based on the kinematic chain is as follows:

[0086] (3-1) Calculation of CMC of metacarpophalangeal joints;

[0087] The angle of the wrist-metacarpal joint is decomposed into a combination of continuous rotation matrices about the vertical and horizontal axes; by multiplying the continuous rotation matrices, the final orientation of the base of the thumb (first metacarpal bone) in space is synthesized.

[0088]

[0089] in, The parent (total) rotation matrix of the CMC joint represents the rotational transformation from the palm coordinate system to the first metacarpal coordinate system; The angle of rotation about the V-axis (such as the expansion / contraction axis) The corresponding basic rotation matrix, The angle of rotation about the H-axis (such as the flexion-extension axis). The corresponding basic rotation matrix, and The two degrees of freedom angles of the CMC joint to be solved.

[0090] (3-2) Calculation of MCP at the proximal interphalangeal joint;

[0091] Based on the known CMC joint rotation, the flexion angle of the MCP joint is solved by calculating the angle between the target direction and the rest direction of the proximal phalanx of the thumb.

[0092] The proximal phalanx vector and target direction vector after CMC rotation are calculated using the vector projection method.

[0093]

[0094]

[0095] Project the two vectors onto a plane perpendicular to the rotation axis u.

[0096]

[0097]

[0098]

[0099] in, Let be the flexion angle of the metacarpophalangeal joint to be solved. The direction vector of the proximal phalanx in its local coordinate system (usually centered at the MCP joint) when the thumb is in a static position with the thumb fully extended. Let u be the target direction vector that the proximal phalanx needs to point to in the current frame, driven by the observation data, and let u be the rotation axis.

[0100] (3-3) IP calculation of distal interphalangeal joints;

[0101] The angle of the interphalangeal joint is calculated based on the combination of the continuous rotation matrix and the rotation matrix corresponding to the angle of the metacarpophalangeal joint. That is, based on the known rotation of the CMC and MCP joints, the angle between the target direction of the distal phalanx of the thumb and its direction after two levels of rotation is calculated, and the angle of the IP joint is solved.

[0102] The bending angle of the interphalangeal joint to be solved The calculation involves calculating the distal phalanx vector after two levels of rotation and the final target direction vector.

[0103]

[0104]

[0105] Project the two vectors onto a plane perpendicular to the rotation axis u.

[0106]

[0107]

[0108]

[0109] in, For the angle of the MCP joint The generated local rotation matrix describing the rotation of the MCP joint itself. Let be the direction vector of the distal phalanx in its local coordinate system when the thumb is in a static position with full extension. The target direction vector of the distal phalanx in the current frame. The direction of the stationary distal phalanx vector is transformed into the world coordinate system by passing it through the MCP joint rotation matrix and the CMC joint rotation matrix in sequence, where u is the rotation axis.

[0110] (4) Integrate the precise thumb joint angle into complete hand posture parameters and output them.

[0111] The calculated precise thumb joint angle is processed in accordance with the preprocessing and converted into a preset standard output format. Specifically, the calculated precise thumb joint angle is integrated into the complete hand posture parameters, and the result is mirror-symmetrically restored and converted into a standard output format.

[0112] Update historical state data for time smoothing constraints in the next frame.

[0113] The present invention also relates to a computing device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the thumb pose estimation method based on multi-level optimization and kinematic constraints.

[0114] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0116] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0117] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0118] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0119] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A thumb pose estimation method based on multi-level optimization and kinematic constraints, characterized in that: The method acquires raw hand data and performs preprocessing, and initializes parameters; A multi-objective optimization function with multiple constraints is constructed, and a preliminary estimate of thumb pose is obtained based on the preprocessed data. Based on the initial estimated thumb posture as the initial value, the kinematic chain is used for recursive calculation to obtain the precise angles of each joint of the thumb. The precise thumb joint angle is integrated into complete hand posture parameters and output.

2. The thumb pose estimation method based on multi-level optimization and kinematic constraints according to claim 1, characterized in that: The preprocessing involves standardizing the coordinate system of the hand model and constructing standardized single-gesture posture data.

3. The thumb pose estimation method based on multi-level optimization and kinematic constraints according to claim 1, characterized in that: The multi-objective optimization function is, ， Where θ is the thumb joint angle vector, which includes 4 components. j=1,2,3,4 correspond to the two angles of the wrist-metacarpal joint, the angle of the metacarpophalangeal joint, and the angle of the interphalangeal joint, respectively. This refers to the key point matching error term; Joint restriction penalty item; This is a time smoothness constraint term; α, β, γ are weighting coefficients.

4. The thumb pose estimation method based on multi-level optimization and kinematic constraints according to claim 3, characterized in that: satisfy, ， in, It is a positive kinematic function, obtained by inputting joint angles. Output the 3D coordinates of the i-th keypoint in the world coordinate system. ; Let i be the target's three-dimensional coordinates in the world coordinate system for the i-th keypoint; The weight coefficient for the i-th key point; This represents the total number of key points.

5. The thumb pose estimation method based on multi-level optimization and kinematic constraints according to claim 3, characterized in that: satisfy, ， in, and These are the upper and lower physiological limits of the j-th component, respectively.

6. The thumb pose estimation method based on multi-level optimization and kinematic constraints according to claim 3, characterized in that: satisfy, ， Where t is the current time and t-1 is the previous time.

7. The thumb pose estimation method based on multi-level optimization and kinematic constraints according to claim 1, characterized in that: The recursive calculation based on the kinematic chain is as follows: The angle of the wrist-to-palm joint is decomposed into a combination of continuous rotation matrices around the vertical and horizontal axes. The angle of the metacarpophalangeal joint and the corresponding rotation matrix are calculated by vector projection. The angle of the interphalangeal joint is calculated based on the combination of continuous rotation matrices and the rotation matrix corresponding to the angle of the metacarpophalangeal joint.

8. The thumb pose estimation method based on multi-level optimization and kinematic constraints according to claim 7, characterized in that: The calculated precise thumb joint angle is processed according to the preprocessing steps and converted into a preset standard output format.

9. The thumb pose estimation method based on multi-level optimization and kinematic constraints according to claim 8, characterized in that: Update historical state data for time smoothing constraints in the next frame.

10. A computing device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the thumb pose estimation method based on multi-level optimization and kinematic constraints as described in any one of claims 1 to 9.

Images