A hand pose reconstruction method based on multi-stage joint enhancement

Abstract

The present invention discloses a method for reconstructing hand posture based on multi-stage joint point enhancement. The method of the present invention proposes a multi-stage joint enhancement network, the first stage of which consists of a network that only trains the fingertip joint positions and the base of palm joint positions; the second stage combines the input image features obtained from the feature extractor and the 12 joint points output by the first stage, and performs joint reasoning through the Transformer module and the features extracted from the image to enhance the accuracy of the prediction of the remaining 30 joint points. The present invention combines visual modalities and geometric modalities, and improves the model's prediction ability for the remaining 30 joint points through multimodal fusion, thereby enhancing the accuracy of joint point prediction.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning, and in particular relates to a hand posture reconstruction method based on multi-stage joint point enhancement. Background Art

[0002] Gesture recognition and reconstruction is a hot research topic in computer vision and human-computer interaction. It aims to understand and reconstruct hand postures and gestures through computer algorithms to enable natural human-computer interaction. With advances in deep learning, image processing, and 3D sensing technologies, gesture recognition and reconstruction techniques have gained widespread application and development in applications such as virtual reality, augmented reality, gaming, robotic control, and sign language translation. Accurate estimation is challenging due to the high degree of articulation and self-occlusion of the hand. Early gesture recognition methods relied on manually extracted image features such as edges, contours, and skin color. These methods required manually designed feature extraction algorithms and were sensitive to factors such as illumination variations and occlusion, resulting in poor robustness. Another early approach used gloves or devices equipped with sensors to capture hand motions. These sensors could detect finger bending and displacement. However, these methods are expensive, lack naturalness, and are unsuitable for large-scale application. Existing methods are based on 2D images or 3D depth maps or point clouds. However, existing methods still face some problems: First, accurately segmenting the hand area and performing pose estimation under complex backgrounds and changing lighting conditions is very challenging; second, when estimating hand pose, when fingers or palms occlude each other, the detected key points may be incomplete or inaccurate, which has a great impact on reconstruction accuracy; third, processing high-resolution depth maps or point cloud data, and using complex deep learning models for pose estimation, will bring high computational costs. Summary of the Invention

[0003] To address the shortcomings of the existing technology, the present invention provides a hand posture reconstruction method based on multi-stage joint point enhancement. The specific technical solution is as follows:

[0004] This embodiment of the present application provides a hand posture reconstruction method based on multi-stage joint point enhancement, the steps are as follows:

[0005] Dataset preprocessing: Feature learning and prediction of 12 fingertip and palmar base joints are performed through the first stage of the multi-stage joint enhancement network.

[0006] The features of the remaining 30 hand joints are learned through the second stage of the multi-stage joint enhancement network.

[0007] In the second stage of the multi-stage joint enhancement network, the Transformer module performs prediction reasoning on 30 hand joints.

[0008] The 12 fingertip and palm base joint feature outputs obtained in the first stage and the 30 joint points obtained by Transformer inference in the second stage are weighted fused and output.

[0009] Reconstruct hand pose and perform quantitative analysis.

[0010] In one possible implementation, the original data to be feature extracted is flipped and randomly cropped to a size of 256 × 256. The preprocessed image is used as the input image for the first and second stages of the multi-stage joint enhancement network.

[0011] In one possible implementation, the preprocessed image is fed into the 12-joint prediction training network structure of the first stage of a multi-stage joint augmentation network. This network structure consists of a feature extractor and EABlock. The feature extractor first extracts features from the input image. After feature extraction, the features are passed to the EABlock module to obtain adaptive interaction features. The adaptive interaction features are then concatenated with the corresponding hand features to generate the final adaptive hand features.

[0012] The adaptive hand features are then fed into the joint feature extractor and joint enhancer. The joint feature extractor first extracts rough joint features, which are then used as input to the joint enhancer to obtain the final joint features and predicted joint positions. This yields joint features and predicted joint positions for 10 fingertip joints and two metacarpal joints.

[0013] In one possible implementation, the EABlock module comprises two phases: an extraction phase and an adaptation phase. The FuseFormer module in the extraction phase uses left-hand and right-hand features to extract interaction features. The interaction features and hand features are then combined and fed into the adaptation phase as two sets of inputs. The two FuseFormer modules in the adaptation phase process these two inputs, generating adaptive left-hand interaction features and adaptive right-hand interaction features.

[0014] In a possible implementation, in the first stage, pre-processed training data is received during training, and training is performed before the start of the second stage training. The loss function used in the training is MSE.

[0015] The training batch size is set to 40 epochs. After forward propagation, the prediction results are output and the loss is calculated using the above loss function. Then, back propagation is performed using the Adam optimizer. The initial learning rate is set to , when the loss saturates, the learning rate is dropped tenfold.

[0016] In one possible implementation, the second-stage network structure consists of an EABlock module, a MANO module, and a Transformer module. In the second stage, the hand features extracted from the image are concatenated with the joint features of the 12 joints predicted in the first stage. The concatenated feature vector is then fed into the MANO module. Using the 12 joint coordinates output from the first stage as constraints, the MANO parameterized model infers the positions of the remaining 30 joints.

[0017] Similar to the first stage, the EABlock module is used to extract hand features from the input image (i.e., preprocessed training data), and joint features are extracted and enhanced for the 30 extracted hand joints. Unlike the first stage, the 30 joint features predicted in this stage do not need to output the corresponding joint point coordinates.

[0018] In one possible implementation, a multi-head self-attention mechanism is used in the second-stage Transformer module to capture the interdependencies between different joints, as well as the associations between joint geometric constraints and image features, thereby improving prediction accuracy. This multi-head self-attention mechanism captures the dependencies between the 12 joints and the 30 to-be-predicted joints, while also incorporating contextual information extracted from image features. Finally, the enhanced features of the 30 joints are output.

[0019] In one possible implementation, the input data used in the second stage of training is the preprocessed training data and the output of the first stage of the multi-stage joint enhancement network. The loss function used in training is as follows:

[0020]

[0021] in is the number of joints in the second stage, The positions of other joints except fingertips and wrists predicted in the second stage, is the corresponding true value.

[0022]

[0023] in and is a hyperparameter, and C is a constant used to make the values ​​of the two cases continuous at |x|=w.

[0024] The comprehensive loss function of the second stage is:

[0025]

[0026] in and is the weight used to balance the losses of the two stages.

[0027] In one possible implementation, the 12 joint features output from the first stage and the enhanced 30 joint features output from the second stage are weightedly spliced ​​and fused through a feature fusion module, and the final 42 hand joint point coordinates are obtained by regression through an MLP module.

[0028] In a possible implementation, the hand posture regression module is used to regress the last 42 hand joint features to reconstruct the hand posture, and the experimental results are quantitatively analyzed in combination with the MPJPE evaluation index.

[0029] The beneficial effects of the present invention are as follows:

[0030] 1. The present invention proposes that the fingertip error obtained by training the fingertips individually is lower than the fingertip error obtained by overall training, thereby proposing a new two-stage joint point enhancement network.

[0031] 2. This paper proposes a dual-modal enhancement structure, which combines geometric constraints and visual constraints to perform joint feature reasoning in the second stage, thereby enhancing the accuracy of joint point prediction.

[0032] 3. Specifically, the final effect of the present invention is better than that of existing methods. In terms of MPJPE performance, the 7.48mm of the present invention is better than 8.79mm of IntagHand, 13.48mm of Zhang et al., and 16.00mm of Moon et al. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0034] Figure 1 It is a flow chart of a method according to an embodiment of the present invention.

[0035] Figure 2 It is a schematic diagram of the overall network structure adopted in the embodiment of the present invention.

[0036] Figure 3 It is a schematic diagram of the EABlock module structure of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of this application.

[0038] The present invention proposes a unique multi-stage joint enhancement technology for determining the complete hand shape and posture from a monocular RGB image. The present invention proposes that when only 10 fingertip joints are trained, the network's task is focused on a specific subset. Since the number of joints is reduced, the network can focus on learning the features of the fingertips without having to process information from other joints such as the palm and wrist at the same time, and such training tasks are relatively simpler. Therefore, the network is more likely to achieve a lower loss value. When training all 42 joints at the same time, the network needs to learn the features of all joints and optimize the posture estimation of the entire hand. This task is more complex, and the loss function needs to consider the error distribution of all joints, which may cause the learning effect of certain joints to be interfered with by the loss of other joints. Therefore, the resulting individual loss of the fingertip joints may be higher than the loss during focused training. Based on the above, we propose a multi-stage joint enhancement network, or MSJENet. The first stage consists of a network trained only on the fingertip and base-of-palm joint positions. The second stage combines input image features from the feature extractor with the 12 joint points output from the first stage. Using a Transformer module, the network performs joint inference with features extracted from the image to enhance the accuracy of position prediction for the remaining 30 joint points. The key concept behind this design is the integration of visual modalities (image features) and geometric modalities (12 joint points). This multimodal fusion improves the model's ability to predict the positions of the remaining 30 joint points.

[0039] To achieve this, we first preprocess the input data. The preprocessed image is then fed into the first-stage fingertip joint predictor. After feature extraction using a ResNet-18 module, a single RGB image representation of the hand is generated. EABlock then processes the hand features for one or both hands to obtain predicted fingertip joints. After obtaining the 10 fingertip joints and two base-of-the-palm joints output from the first stage, these 12 predicted joints serve as one input to the second stage. The other input is the features of the remaining 15 or 30 joints learned by the EABlock module embedded in the second stage. In subsequent processing, the geometric input of the 12 joints output from the first stage is embedded into a high-dimensional space, positionally encoded, and fed into the Transformer encoder layer. These 12 joint features can then be fused with the joint features learned by EABlock in the second stage. At the same time, the 12 joints output by the first stage will be input into the subsequent Transformer for decoding, and the positions of the remaining 30 joints will be inferred. At the same time, they will receive positive supervision from the 15 or 30 joint features learned by EABlock in the second stage. Finally, the 30 predicted joints output by the second stage and the 12 predicted joints output by the first stage are weighted and fused to obtain the final 42 joint outputs. The network structure used in this method is as follows: Figure 2 shown.

[0040] In one possible implementation, Figure 1 As shown, the present invention applies to a hand posture reconstruction method based on multi-stage joint point enhancement, the steps are as follows:

[0041] Step 1: Dataset preprocessing.

[0042] Download the Interhand2.6M dataset, which contains 2.6M images. For feature extraction, preprocess the raw data by flipping it and randomly cropping it to 256×256. Use these preprocessed images as input to the first and second stages of the multi-stage joint augmentation network.

[0043] Step 2: Feature learning and prediction of 12 fingertip and palm base joints are performed through the first stage of the multi-stage joint enhancement network.

[0044] The pre-processed image is input into the 12 joint point prediction training network structure of the first stage of the multi-stage joint enhancement network. The network structure consists of a feature extractor and EABlock. First, the feature extractor extracts the features of the input image to obtain the left hand features. and right hand features The feature extractor uses a standard ReaNet-18. This adaptive structure enables the network to obtain higher resolution feature maps in earlier layers, but also reduces the overall receptive field. After feature extraction, the features are passed to the EABlock module to obtain adaptive interactive features. The EABlock module also consists of two stages, namely the extraction stage and the adaptation stage. The FuseFormer module in the extraction stage first uses the left hand features to extract the left hand features. and right hand features Extracting interactive features Then, the interaction features and hand features or After the combination, they are respectively fed into the adaptation stage as two sets of inputs. The two sets of inputs are processed by the FuseFormer modules of the two branches of the adaptation stage to generate the interactive features of the adaptive left hand. Interaction features with adaptive right hand Afterwards, the adaptive interaction features are concatenated with the corresponding hand features to generate the final adaptive hand features ( or ). Interaction features The extracted formula is expressed as follows:

[0045] (1)

[0046] in Represents the learnable weights of FuseFormer in the extraction phase.

[0047] During the adaptation phase, EABlock (e.g. Figure 3 As shown in Figure 2, two additional Fuseformers are used to extract the interaction features. Adapted to each hand. Although the interaction features obtained from the extraction stage are helpful for understanding how the two hands interact, directly using them for 3D hand mesh recovery of each hand may not be optimal. Therefore, the interaction features are fused with the features of each hand to achieve two goals: 1) preserve the interaction information between the two hands; 2) obtain information unique to the left and right hands. Taking the left hand adaptation as an example, the left hand features are fused to the hand mesh of the hand. and interaction features Pass them to a FuseFormer module, fuse them through the FuseFormer module, and output the interaction features adapted to the left hand The same method is used for the adaptation of the right hand. and interaction features Passed to another FuseFormer module, the fusion output adapts to the interaction features of the right hand .

[0048] (2)

[0049] (3)

[0050] in and denote the learnable weights in the Fuseformer for left and right hand adaptation, respectively. Finally, for each hand, the adaptive interaction features ( or ) and its corresponding hand features ( or ) are connected as the final adaptive hand features of each hand, and and express.

[0051] Afterwards, and The data is fed into the joint feature extractor and the self-joint transformer. The joint feature extractor first extracts rough joint features, which are then used as input to the self-joint transformer to obtain the final joint features and predicted joint positions. Ultimately, joint features and predicted joint positions are obtained for 10 fingertip joints and 2 metacarpal joints.

[0052] In the first stage, the pre-processed training data obtained in step 1 is received during training, and training is performed before the start of the second stage training. The loss function used during training is MSE:

[0053] (4)

[0054] in is the number of joints in the first stage (including fingertips and wrist), The positions of the fingertips and wrist joints predicted in the first stage, is the corresponding true value.

[0055] In the first stage, the training data is the preprocessed data set in step 1, and the batch size is set to 40 epochs. After the prediction results are output through forward propagation, the loss is calculated using the above loss function, and then backpropagation is performed. The adam optimizer is used and the initial learning rate is set to , when the loss saturates, the learning rate is dropped tenfold.

[0056] Step 3: Perform feature learning on the remaining 30 hand joints through the second stage of the multi-stage joint enhancement network.

[0057] The network structure of the second stage consists of an EABlock module, a MANO module, and a Transformer module. In the second stage, the goal is to use the features extracted from the image , splice it with the 12 joint points predicted in the first stage, input the spliced ​​feature vector into the MANO module, use the 12 joint coordinates output in the first stage as constraints, and infer the remaining 30 joint positions through the MANO parameterized model.

[0058] Similar to the first stage, the EABlock module is used to extract hand features from the input image, which is the training data preprocessed in step 1. Joint features are then extracted and enhanced for the 30 extracted hand joints. However, unlike the first stage, the 30 joint features predicted in this stage do not need to output the corresponding joint point coordinates.

[0059] Step 4: In the second stage of the multi-stage joint enhancement network, the Transformer module performs prediction inference on 30 hand joints.

[0060] The 12 joint features output from the first stage are fused with the 30 joint features obtained in step three through the Transformer module. This step realizes the fusion of the geometric constraints obtained in the first stage, i.e., the 12 joint features, and the image constraints obtained in the second stage, i.e., the 30 joint features, to achieve the purpose of dual-modal enhancement. In the second-stage Transformer module, through the multi-head self-attention mechanism, the model can capture the mutual dependence between different joints, as well as the association between the joint geometric constraints and image features, thereby improving the prediction accuracy. For the multi-head self-attention mechanism, first, perform a linear transformation on the input: given the input matrix , where n is the length of the sequence, is the dimension of the input feature. The input obtained by feature fusion is linearly transformed into query (Query), key (Key), and value (Value):

[0061] (5)

[0062] in, 、 、 Is a trainable parameter matrix with dimensions of ,in Is the feature dimension of each head. Then, calculate the self-attention: For each head, calculate the attention score:

[0063] (6)

[0064] in, is a scaling factor used to prevent the dot product value from being too large, which would make the gradient of the softmax function too small. Secondly, multi-head calculation is performed: the multi-head self-attention mechanism divides the input into h different subspaces and calculates multiple attention heads in parallel. Each head uses different 、 、 Parameter matrix. The outputs of each head will be spliced ​​together:

[0065] (7)

[0066] (8)

[0067] in, is the concatenated linear transformation matrix, which is used to map the output back to the original feature dimension. They are respectively from The sub-matrix split out from is used to calculate the output of the i-th attention head.

[0068] Finally, the residual connection and layer normalization are performed: the output of the multi-head self-attention is residually connected to the input, and the layer is normalized:

[0069] (9)

[0070] The multi-head self-attention mechanism captures the dependencies between the 12 joints and the 30 joints to be predicted, while also combining the contextual information extracted from image features. Finally, the enhanced features of the 30 joints are output.

[0071] The input data used for training in this stage is the preprocessed training data in step 1 and the output of the first stage of the multi-stage joint enhancement network. The loss function used in training is as follows:

[0072] (10)

[0073] in is the number of joints in the second stage, The positions of other joints except fingertips and wrists predicted in the second stage, is the corresponding true value.

[0074] (11)

[0075] in and is a hyperparameter, and C is a constant used to make the values ​​of the two cases continuous at |x|=w.

[0076] The comprehensive loss function of the second stage is:

[0077] (12)

[0078] in and is the weight used to balance the losses of the two stages.

[0079] Step 5: Perform weighted fusion and output of the 12 fingertip and palm base joint feature outputs obtained in the first stage and the 30 joint points obtained by Transformer inference in the second stage.

[0080] The feature fusion module performs weighted splicing and fusion of the 12 joint features output from the first stage and the enhanced 30 joint features output from the second stage, and regresses through the MLP module to obtain the final 42 hand joint point coordinates. The feature fusion formula for this step is as follows:

[0081] (13)

[0082] in is a learnable weight matrix, is the bias term, and [⋅,⋅] represents the concatenation operation.

[0083] Step 6: Reconstruct hand posture and perform quantitative analysis.

[0084] The 42 hand joint features obtained in step 5 are regressed through the hand posture regression module (such as the fully connected layer) to reconstruct the hand posture. The experimental results are quantitatively analyzed in combination with the MPJPE evaluation index, and compared with existing excellent methods to measure the accuracy and effectiveness of this method.

[0085] The present invention uses MPJPE as the evaluation index, and the formula is as follows:

[0086] (14)

[0087] Where: N is the total number of joint points, is the predicted position of the i-th joint point, is the true position of the i-th joint, Represents the Euclidean distance between the predicted position of the i-th joint point and the true position (usually a three-dimensional coordinate).

[0088] Neural network details: In the PyTorch framework, the optimizer is Adam, and the learning rate is initially set to , when the loss is saturated, the learning rate is reduced by ten times. The input image size is set to 256×256 and the batch size is set to 32. The experimental data is as follows:

[0089] Method Interhand2.6M(MPJPE / mm) IntagHand 8.79 Zhang et al. 13.48 Moon et al. 16.00 The present invention 7.48

[0090] Compared with other existing mainstream methods, the proposed method achieves better performance when trained in the same experimental environment and tested on the public dataset InterHand2.6M dataset, which is better than IntagHand's 8.79.

[0091] The above description is a further detailed description of the present invention in conjunction with specific / preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art of the present invention may make various substitutions or modifications to the described embodiments without departing from the scope of the present invention, and such substitutions or modifications should be considered to fall within the scope of protection of the present invention.

[0092] Parts of the present invention that are not described in detail belong to the common knowledge of those skilled in the art.

Claims

1. A hand posture reconstruction method based on multi-stage joint point enhancement, characterized in that: Here are the steps: Dataset preprocessing: Feature learning and prediction of 12 fingertip and palmar base joints are performed through the first stage of the multi-stage joint enhancement network; The preprocessed image is input into the 12-joint prediction training network structure of the first stage of the multi-stage joint enhancement network. The network structure consists of a feature extractor and EABlock. The feature extractor first extracts features from the input image. After feature extraction, the features are passed to the EABlock module to obtain adaptive interaction features. The adaptive interaction features are then concatenated with the corresponding hand features to generate the final adaptive hand features. Subsequently, the adaptive hand features are fed into the joint feature extractor and joint enhancer, respectively. The joint feature extractor first extracts rough joint features, which are then used as input to the joint enhancer to obtain the final joint features and predicted joint positions. This results in joint features and predicted joint positions for 10 fingertip joints and 2 metacarpal joints. The remaining 30 hand joints are trained using the second stage of the multi-stage joint enhancement network. The network structure of the second stage consists of an EABlock module, a MANO module, and a Transformer module. In the second stage, the hand features extracted from the image are concatenated with the joint features of the 12 joint points predicted in the first stage. The concatenated feature vector is input into the MANO module. Using the 12 joint coordinates output from the first stage as constraints, the positions of the remaining 30 joints are inferred using the MANO parameterized model. Similar to the first stage, the EABlock module is used to extract hand features from the input image, and joint feature extraction and joint feature enhancement are performed on the extracted 30 hand joint points. Unlike the first stage, the 30 joint features predicted in this stage do not need to output the corresponding joint point coordinates; In the second stage of the multi-stage joint enhancement network, the Transformer module performs prediction reasoning on 30 hand joints; The 12 fingertip and palm base joint feature outputs obtained in the first stage and the 30 joint points obtained by Transformer inference in the second stage are weighted fused and output; Reconstruct hand pose and perform quantitative analysis.

2. The hand posture reconstruction method based on multi-stage joint point enhancement according to claim 1 is characterized in that: For the original data that needs to be extracted, it is flipped and randomly cropped to a size of 256×256; the preprocessed image is used as the input image of the first stage part and the second stage part of the multi-stage joint enhancement network.

3. A hand posture reconstruction method based on multi-stage joint point enhancement according to claim 1 or 2, characterized in that: The EABlock module includes two stages, namely the extraction stage and the adaptation stage. The FuseFormer module in the extraction stage uses the left-hand features and the right-hand features to extract the interaction features. Then, the interaction features and the hand features are combined and passed into the adaptation stage as two sets of inputs. The two sets of inputs are processed by the FuseFormer modules of the two branches of the adaptation stage to generate adaptive left-hand interaction features and adaptive right-hand interaction features.

4. A hand posture reconstruction method based on multi-stage joint point enhancement according to claim 1 or 2, characterized in that: In the first stage, pre-processed training data is received during training, and training is performed before the start of the second stage training. The loss function used in training is MSE.

5. The hand posture reconstruction method based on multi-stage joint point enhancement according to claim 1, characterized in that: In the second-stage Transformer module, the multi-head self-attention mechanism is used to capture the interdependencies between different joints, as well as the associations between the geometric constraints of the joints and image features, thereby improving prediction accuracy. The multi-head self-attention mechanism captures the dependencies between 12 joints and 30 joints to be predicted, while combining the contextual information extracted from image features. Finally, the enhanced 30 joint features are output.

6. The hand posture reconstruction method based on multi-stage joint point enhancement according to claim 5, characterized in that: The input data used in the second stage of training is the preprocessed training data in step 1 and the output of the first stage of the multi-stage joint enhancement network; the loss function used in training is as follows: ； in is the number of joints in the second stage, The positions of other joints except fingertips and wrists predicted in the second stage, is the corresponding true value; ； in and is a hyperparameter, and C is a constant used to make the values ​​of the two cases continuous at |x|=w; The comprehensive loss function of the second stage is: ； in and is the weight used to balance the losses of the two stages.

7. The hand posture reconstruction method based on multi-stage joint point enhancement according to claim 5, characterized in that: The specific method of weighted fusion is as follows: Through the feature fusion module, the 12 joint features output from the first stage and the enhanced 30 joint features output from the second stage are weightedly spliced ​​and fused, and the final 42 hand joint point coordinates are obtained by regression through the MLP module.

8. The hand posture reconstruction method based on multi-stage joint point enhancement according to claim 7, characterized in that: Reconstruct hand posture and perform quantitative analysis, including: The hand posture regression module is used to regress the last 42 hand joint features to reconstruct the hand posture, and the experimental results are quantitatively analyzed in combination with the MPJPE evaluation index.

Images