Hand model reconstruction method based on feature interaction

Abstract

The application discloses a hand model reconstruction method based on feature interaction, introduces a processing mode of feature interaction in a hand posture estimation stage, so that hand features and skeleton features can effectively learn from each other, so that in the case of occlusion, the hand gesture recognition result can also have higher accuracy. At the same time, in the aspect of feature extraction, the application of the subnetwork also reduces the parameters of the network and accelerates the convergence speed of the model. Finally, the output of the network is a hand shape containing surface texture information, compared with the traditional hand posture estimation result which only presents a hand shape composed of points and lines, the hand reconstruction model obtained by the application is more in line with the hand shape of a person, and when human-computer interaction is realized, a more real interactive experience can be realized, and the operation accuracy can be improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a hand model reconstruction method based on feature interaction. Background Technology

[0002] Hand model reconstruction refers to the process of reconstructing a three-dimensional hand model from an image or video. It can be used in fields such as virtual reality, augmented reality, and hand pose estimation. Hand model reconstruction is generally divided into two stages: hand pose estimation and three-dimensional model reconstruction.

[0003] In the hand pose estimation stage, deep learning techniques can be used to predict hand pose by extracting features from hand images using convolutional neural networks (CNNs). This method can improve accuracy by increasing training data and refining the network architecture. Furthermore, multi-task learning methods can be used to simultaneously predict hand pose and hand position to improve prediction accuracy. However, occlusion is a major challenge in current hand pose estimation. During hand interaction with objects, parts of the hand are often occluded, leading to problems such as joint distortion and incorrect estimated position in the final hand pose estimation. Simply extracting features or increasing training data without utilizing the contextual information provided by the image will not effectively address the occlusion problem. Summary of the Invention

[0004] The purpose of this invention is to provide a hand model reconstruction method based on feature interaction, which solves the problem in the prior art that the hand pose estimation is inaccurate due to the hand being occluded by other objects, thus affecting the accuracy of hand model reconstruction.

[0005] The technical solution adopted in this invention is a hand model reconstruction method based on feature interaction, which is implemented according to the following steps:

[0006] Step 1: Obtain a dataset consisting of hand gesture training images;

[0007] Step 2: Preprocess the data obtained in Step 1 to obtain a preprocessed hand image;

[0008] Step 3: Construct a network model structure for hand pose estimation, and combine it with the preprocessed hand image to predict the 3D hand pose position;

[0009] Step 4: Reconstruct the initial hand model based on the predicted 3D hand pose position;

[0010] Step 5: Set the network parameters for hand pose estimation in Step 3, input the preprocessed hand image into the hand pose estimation network model for training, and save the trained hand pose estimation network model as output.

[0011] Step 6: Input the preprocessed hand image into the trained hand pose estimation network model, input the output result into the reconstructed initial hand model for training, and save the trained model to obtain the reconstructed hand model.

[0012] The invention is further characterized by:

[0013] Step 2 is as follows:

[0014] The hand pose training images are cropped to 256×256 pixels and then normalized to obtain the preprocessed hand images.

[0015] In the data preprocessing stage, the joint at the wrist is selected as the root joint, and the bone from the root joint to the base joint of the middle finger is defined as the reference bone.

[0016] Step 3 is as follows:

[0017] The network model structure for constructing hand pose estimation includes a 2D hand pose estimation network and a 3D hand pose estimation network;

[0018] The 2D hand pose estimation network comprises a ResNet-50 residual network, a softmax layer, and sub-networks. The ResNet-50 residual network extracts features from the preprocessed hand image input to the network, outputting pose-aware feature encodings that include joint-aware and skeletal-aware features. It also represents the predicted keypoint and bone positions using heatmaps and skeletal maps, respectively, generating joint heatmaps predicting the hand joint distribution and skeletal heatmaps predicting the bone distribution. The softmax layer enhances the contrast of the joint-aware and skeletal-aware features in the pose-aware features, fusing them with the corresponding joint and skeletal heatmaps to obtain joint feature maps of 21 joints and skeletal feature maps containing 20 bones. The sub-network structure includes two convolutional layers and one MLP layer, used for channel and feature reduction of the joint and skeletal feature maps, respectively, to obtain reduced joint features f. j And Joan's skeletal features f s ;

[0019] Between the 2D and 3D hand pose estimation networks, a feature interaction module is included. This module contains semantic graph convolutional layers, self-attention layers, MLP layers, and the qkv mechanism in the transformer. The semantic graph convolutional layers and self-attention layers are used to update the reduced joint features; the qkv mechanism is used to implement the updated reduced joint features f. j And Joan's skeletal features f sThe interaction, the joint features and bone features after the interaction are processed through a multi-head self-attention mechanism and an MLP layer to output the final joint features and the final bone features.

[0020] The 3D hand pose estimation network contains multiple fully connected layers, and the predicted 3D hand pose position is obtained by regressing the final joint features and the final skeletal features.

[0021] The specific process of feature extraction from the preprocessed hand image input into the network is as follows:

[0022] For each joint i in the image, a heatmap is generated using the standard deviation σ in the x and y spatial directions, denoted as:

[0023]

[0024] where u=[u x ,u y ] represents a 2D joint index in the pixel plane; H i (u) represents the generated heatmap of the i-th joint; P i 2d This represents the predicted coordinates of the i-th joint in the pixel plane;

[0025] For each bone j, connect two endpoints. and Calculate the position of the intermediate joint v = [v x ,v y And the angles θ of these endpoints, then using the 2D rotation matrix R θ Convert u and v to in

[0026] Based on this, the resulting bone heatmap is represented as follows:

[0027]

[0028] Where, σ x and σ y It is the standard deviation of the Gaussian distribution along the bone direction in 2D space.

[0029] The specific process of updating the reduced joint features using semantic graph convolutional layers and self-attention layers is as follows:

[0030] Reduced joint features f j Input semantic graph convolutional neural network generates semantic graph convolution updated joint features f J The update expression is:

[0031]

[0032] Where || denotes channel cascading, σ denotes the nonlinear activation function ReLU, and ω d Let ρi and M be the d-th row of the learnable matrix A. d These represent the Softmax nonlinearity and the weighting matrix, respectively.

[0033] Reduced joint features f j Through the self-attention layer, based on the complementary matrix Update joint feature maps and complement matrices. A 21x21 matrix represents the connections between 21 joints. If a joint is not correctly connected, the corresponding row and column are set to 1, indicating an incorrect connection between the two joints. The fully connected layer first converts the reduced joint features f... j Mapped to vectors q, k, and v of the same shape, vectors q, k, and v are responsible for calculating the attention mapping designed in the Transformer module, and the calculation expression is:

[0034]

[0035] Where σ represents the Softmax function layer, Θ represents element-wise multiplication, and ω represents the learnable weight parameters of the self-attention layer;

[0036] Reduced joint features f j After semantic graph convolution and self-attention layer, the final updated joint features are obtained as follows:

[0037] f J =αf J '+βf J "

[0038] Here, α and β are the weight parameters that are set.

[0039] The specific process for outputting the final joint features and the final bone features is as follows:

[0040] For the final updated joint feature f J and skeletal features f S First, use the MLP layer to f J Convert to q J k J and v J Skeletal features f S It is fed into the MLP layer and outputs q. S k S v S The feature interaction module utilizes two attention matrices a JS and a SJ Calculate joint features after processing by the qkv mechanism and skeletal features

[0041] Two attention matrices a JS and a SJ Their corresponding eigenvectors v J and v S Perform element-wise multiplication;

[0042] Joint features after QKV mechanism processing and skeletal features The residual connections are used to update the feature vector v. J and v s In addition, the two combined layers employ a multi-head self-attention mechanism and an MLP layer to act on the joint features processed by the qkv mechanism. and skeletal features Finally, the expression for retaining non-local content is:

[0043]

[0044]

[0045] in, Indicates the final joint features. Indicates the final skeletal features. Provides matrix multiplication, ω J and ω S These are learnable weight parameters.

[0046] Step 4 is as follows:

[0047] Step 4.1: Use the IKnet neural network to encode and train the joint positions in the predicted 3D hand pose position, and finally output the rotation angle Q of each joint.

[0048] Step 4.2: Select the MANO hand model and estimate the shape parameters of the MANO model by using the joint positions in the predicted 3D hand pose. During training, calculate the shape parameter β of the model, expressed as:

[0049]

[0050] The first item ensures the bone length l of the deformable hand model. b (β) and predicted 3D bone length The lengths match. From the 3D prediction of hand pose estimation, l ref (β) is the reference bone of the MANO model, the second term is the L2 regularizer of the shape parameter, and is passed through λ β Weighting;

[0051] Calculate the shape parameter β of the MANO model that minimizes E(β) according to formula (7);

[0052] Step 4.3: Combine the shape parameter β of the MANO model and the joint rotation angle Q to reconstruct the initial hand model.

[0053] Step 4.1 The specific process is as follows:

[0054] Step 4.1.1: Construct the IKnet neural network. The IKnet neural network is a batch-normalized 7-layer fully connected neural network. Except for the last layer which uses linear activation, the remaining layers use sigmoid as the activation function.

[0055] Step 4.1.2: Encode the three-dimensional joint positions in the 3D hand pose as I = [X, D, X ref D ref ], where the encoded input is given to the IKnet neural network and the output is the global rotation angle Q of each joint.

[0056] The network parameters set in step 3 include batch size, learning rate, training epochs, method for initializing model parameters, and optimization method.

[0057] The beneficial effects of this invention are:

[0058] The present invention introduces the qkv mechanism of transformer for hand model reconstruction based on feature interaction, realizing the interaction between hand joint information and bone information, and effectively solving the problem of low hand pose estimation accuracy caused by occlusion.

[0059] Compared to traditional hand pose estimation methods, which simply deepen the network layer, the hand model reconstruction method based on feature interaction of this invention can make full use of the contextual information provided by the image.

[0060] The present invention provides a hand model reconstruction method based on feature interaction. In addition to the traditional hand pose estimation which only provides the position of the hand joints, it also introduces hand model reconstruction, which makes the hand texture information richer and more specific.

[0061] In the hand model reconstruction method based on feature interaction, the present invention reduces the number of model parameters by using identical and concurrent sub-networks to perform channel and feature reduction on joint features and skeletal features during the hand pose estimation stage. Attached Figure Description

[0062] Figure 1 This is a flowchart of a hand model reconstruction method based on feature interaction according to the present invention;

[0063] Figure 2 This is a schematic diagram of a hand skeleton sample in a hand model reconstruction method based on feature interaction according to the present invention;

[0064] Figure 3 This is a schematic diagram of the internal structure of the feature interaction module in the hand model reconstruction method based on feature interaction of the present invention;

[0065] Figure 4 This is a schematic diagram of the feature reduction subnetwork in the hand model reconstruction method based on feature interaction of the present invention;

[0066] Figure 5 This is a schematic diagram of the MANO hand reconstruction model used in the hand model reconstruction method based on feature interaction of the present invention.

[0067] Figure 6 This is a schematic diagram of the 3D hand pose estimation result and hand reconstruction model in Embodiment 2 of the present invention;

[0068] Figure 7 This is a schematic diagram of the accuracy results of the hand pose estimation network after 75 training rounds in Embodiment 2 of the present invention. Detailed Implementation

[0069] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0070] This invention relates to a hand model reconstruction method based on feature interaction, such as... Figure 1 As shown, please follow these steps:

[0071] Step 1: Obtain a dataset consisting of hand gesture training images;

[0072] Step 2: Preprocess the data obtained in Step 1 to obtain a preprocessed hand image;

[0073] The hand pose training images are cropped to 256×256 pixels and then normalized to obtain preprocessed hand images. Normalization of the images ensures that the image input size of the hand pose estimation network is consistent, while also ensuring that the network training can converge faster.

[0074] Since there may be more than one 3D pose corresponding to a 2D hand pose, to mitigate the depth scale blur caused by a monocular camera, the predicted coordinates need to be relative to the root joint and standardized by the length of a reference hand bone. In the data preprocessing stage, the wrist joint is selected as the root joint, and the bone from this joint to the base of the middle finger joint is defined as the reference bone.

[0075] Step 3: Construct a network model structure for hand pose estimation, and predict the 3D hand pose position by combining the preprocessed hand image; the specific process is as follows:

[0076] The network model structure for constructing hand pose estimation includes a 2D hand pose estimation network and a 3D hand pose estimation network;

[0077] The 2D hand pose estimation network includes a residual network ResNet_50, a softmax layer, and subnetworks. The residual network ResNet_50 is used to extract features from the preprocessed hand image input into the network. The specific extraction process is as follows:

[0078] For each joint i in the image, a heatmap is generated using the standard deviation σ in the x and y spatial directions, denoted as:

[0079]

[0080] where u=[u x ,u y ] represents a 2D joint index in the pixel plane; H i (u) represents the generated heatmap of the i-th joint; P i 2d This represents the predicted coordinates of the i-th joint in the pixel plane;

[0081] For each bone j, connect two endpoints. and Calculate the position of the intermediate joint v = [v x ,v y And the angles θ of these endpoints, then using the 2D rotation matrix R θ Convert u and v to in

[0082] Based on this, the resulting bone heatmap is represented as follows:

[0083]

[0084] Where, σ x and σ y It is the standard deviation of the Gaussian distribution along the bone direction in 2D space.

[0085] After feature extraction from the preprocessed hand image input into the network, the output is a pose-aware feature encoding containing joint-aware and skeletal-aware features. The predicted keypoint and bone locations are represented using heatmaps and skeletal maps, respectively, generating joint heatmaps predicting the hand joint distribution and skeletal heatmaps predicting the bone distribution. A softmax layer enhances the contrast of the joint-aware and skeletal-aware features in the pose-aware features. The joint-aware and skeletal-aware features are then fused with the corresponding joint and skeletal heatmaps to obtain a joint feature map with 21 joint points and a skeletal feature map containing 20 bones. The sub-network structure includes two convolutional layers and one MLP layer, used for channel and feature reduction of the joint and skeletal feature maps, respectively, to obtain the reduced joint features f. j And Joan's skeletal features f s ;

[0086] Between the 2D and 3D hand pose estimation networks, a feature interaction module is included. This module contains semantic graph convolutional layers, self-attention layers, MLP layers, and the QKV mechanism in the transformer. The semantic graph convolutional layers and self-attention layers are used to update the reduced joint features; the specific update process is as follows:

[0087] Reduced joint features f j Input semantic graph convolutional neural network generates semantic graph convolution updated joint features f J The update expression is:

[0088]

[0089] Where || denotes channel cascading, σ denotes the nonlinear activation function ReLU, and ω d Let ρi and M be the d-th row of the learnable matrix A. d These represent the Softmax nonlinearity and the weighting matrix, respectively.

[0090] Reduced joint features f j Through the self-attention layer, based on the complementary matrix Update joint feature maps and complement matrices. A 21x21 matrix represents the connections between 21 joints. If a joint is not correctly connected, the corresponding row and column are set to 1, indicating an incorrect connection between the two joints. The fully connected layer first converts the reduced joint features f... j Mapped to vectors q, k, and v of the same shape, vectors q, k, and v are responsible for calculating the attention mapping designed in the Transformer module, and the calculation expression is:

[0091]

[0092] Where σ represents the Softmax function layer, Θ represents element-wise multiplication, and ω represents the learnable weight parameters of the self-attention layer;

[0093] Reduced joint features f j After semantic graph convolution and self-attention layer, the final updated joint features are obtained as follows:

[0094] f J =αf J '+βf J "

[0095] Here, α and β are the weight parameters that are set.

[0096] The qkv mechanism is used to implement the updated reduced joint features f j And Joan's skeletal features f s The interaction process, followed by the processing of joint and skeletal features, utilizes a multi-head self-attention mechanism and an MLP layer to output the final joint and skeletal features. The specific process is as follows:

[0097] For the final updated joint feature f J and skeletal features f S First, use the MLP layer to f J Convert to q J k J and v J Skeletal features f S It is fed into the MLP layer and outputs q. S k S v S The feature interaction module utilizes two attention matrices a JS and a SJ Calculate joint features after processing by the qkv mechanism and skeletal features

[0098] Two attention matrices a JS and a SJ Their corresponding eigenvectors v J and v S Perform element-wise multiplication;

[0099] Joint features after QKV mechanism processing and skeletal features The residual connections are used to update the feature vector v. J and v s In addition, the two combined layers employ a multi-head self-attention mechanism and an MLP layer to act on the joint features processed by the qkv mechanism. and skeletal features Finally, the expression for retaining non-local content is:

[0100]

[0101]

[0102] in, Indicates the final joint features. Indicates the final skeletal features. Provides matrix multiplication, ω J and ω S These are learnable weight parameters.

[0103] The 3D hand pose estimation network contains multiple fully connected layers, and the predicted 3D hand pose position is obtained by regressing the final joint features and the final skeletal features.

[0104] Step 4: Reconstruct the initial hand model based on the predicted 3D hand pose position; the specific process is as follows:

[0105] Step 4.1: Use the IKnet neural network to encode and train the joint positions in the predicted 3D hand pose, and finally output the rotation angle Q of each joint; the specific process is as follows:

[0106] Step 4.1.1: Construct the IKnet neural network. The IKnet neural network is a batch-normalized 7-layer fully connected neural network. Except for the last layer which uses linear activation, the remaining layers use sigmoid as the activation function.

[0107] Step 4.1.2: Encode the three-dimensional joint positions in the 3D hand pose as I = [X, D, X ref D ref ], where the encoded input is given to the IKnet neural network and the output is the global rotation angle Q of each joint.

[0108] Step 4.2: Select the MANO hand model and estimate the shape parameters of the MANO model by using the joint positions in the predicted 3D hand pose. During training, calculate the shape parameter β of the model, expressed as:

[0109]

[0110] The first item ensures the bone length l of the deformable hand model. b (β) and predicted 3D bone length The lengths match. From the 3D prediction of hand pose estimation, l ref (β) is the reference bone of the MANO model, the second term is the L2 regularizer of the shape parameter, and is passed through λ β Weighting;

[0111] Calculate the shape parameter β of the MANO model that minimizes E(β) according to formula (7);

[0112] Step 4.3: Combine the shape parameter β of the MANO model and the joint rotation angle Q to reconstruct the initial hand model.

[0113] Step 5: Set the network parameters for hand pose estimation in Step 3, including batch size, learning rate, epochs, initialization method for model parameters, and optimization method. Input the preprocessed hand image into the hand pose estimation network model for training, and save the trained hand pose estimation network model as output.

[0114] Step 6: Input the preprocessed hand image into the trained hand pose estimation network model, input the output result into the reconstructed initial hand model for training, and save the trained model to obtain the reconstructed hand model.

[0115] Example 1

[0116] To obtain hand pose training datasets, commonly used datasets include: the Stereo Hand Pose TrackingBenchmark (STB) dataset, which contains two subsets: subset 1, captured by the Bumblebee2 stereo camera, and subset 2, captured by the active depth camera; the Rendered Hand Dataset (RHD) dataset, containing 41,258 training samples and 2,728 test samples; and the FreiHAND dataset, containing 130,240 training samples and 3,960 evaluation samples. The hand pose training data is cropped to 256*256 pixels. Since there may be more than one 3D pose corresponding to a 2D hand pose, to mitigate the depth scale blur caused by a monocular camera, the predicted coordinates need to be relative to the root joint and standardized by the length of a reference hand bone. In the data preprocessing stage, the wrist joint is selected as the root joint, and the bones from this joint to the base of the middle finger joint are defined as reference bones. Hand joints and bones are represented using points and lines, as shown below. Figure 2 As shown.

[0117] The hand pose estimation network is divided into a 2D hand pose estimation stage and a 3D hand pose estimation stage. The 2D hand pose estimation stage mainly uses a residual network ResNet_50 for feature extraction, predicting the distribution of joints and bones using heatmaps and bone maps, respectively. Simultaneously, a softmax layer is used after the ResNet_50 to enhance the contrast of joint-aware and bone-aware features, and these features are concatenated with the corresponding heatmaps and bone maps, such as... Figure 1 As shown.

[0118] Following the 2D hand pose estimation in the previous stage, joint features and skeletal features are each input into a Feature Interaction Module (FCB) to enable feature interaction between joint and skeletal features, thereby obtaining refined joint and skeletal features. The internal structure of the Feature Interaction Module is as follows: Figure 3 As shown. These two types of features, after regression, output the final 3D hand pose position.

[0119] In this embodiment, the subnetwork includes a convolutional layer with a kernel size of 3*3 and a stride of 2*2, named Reso-r1; a convolutional layer with a kernel size of 1*1 and a stride of 1*1, named Reso-r2; and a final MLP layer named Chanr, where r represents the J of a joint or the S of a bone. Figure 4 Detailed dimensionality reduction methods for joint and skeletal features are presented. Combining the methods in steps 3 and 4 of this invention, the parameters of the 2D hand pose estimation network are set as follows: the batch size of the FreiHAND dataset is set to 32, and the STB dataset is set to 64. During training, the model parameters are first initialized using Kaiming, and an Adam optimizer with a mini-batch size of 48 is applied; the learning rate is 0.0005; and the training epochs are 120. The 2D hand pose estimation network parameters are fixed, while the 3D hand pose estimation network uses a fixed step size decay strategy during training, with the learning rate initialized to 0.0001 and the epochs set to 160. After training, the trained model is saved and output. The parameters of the hand pose estimation network are fixed, and the RHD dataset used for the hand reconstruction model network is set to a batch size of 64, the learning rate initialized to 0.0006, and the training epochs set to 120. The final hand model is obtained, as shown below. Figure 5 As shown.

[0120] Example 2

[0121] The training data used is shown in Table 1. The 3D hand pose estimation results and hand reconstruction model are shown in Table 1. Figure 6 As shown.

[0122] Table 1

[0123] Training rounds: 65 75 100 3D hand pose estimation accuracy: 81.63％ 87.44％ 84.48％

[0124] The accuracy results of the hand pose estimation network after 75 training rounds are as follows: Figure 7 As shown.

[0125] According to Table 1 and Figure 6 , Figure 7It can be seen that by introducing a feature interaction module into the hand pose estimation network, information interaction between joint features and skeletal features can be achieved, better modeling the structural relationship of hand pose, thereby improving the accuracy of hand pose estimation. Experiments show that as the number of training rounds increases, the accuracy of 3D hand pose estimation of the designed network model reaches 87.44% after 75 training rounds. As shown in the results figure, in the process of real-time hand pose estimation and hand model reconstruction, the trained model can estimate 3D hand pose with high accuracy and reconstruct the hand model based on the MANO hand model. The results prove that the hand reconstruction model obtained by the method of this invention is more in line with the human hand shape, which can achieve a more realistic interactive experience in human-computer interaction, and at the same time improve the accuracy of operation.

[0126] Through the above methods, this invention's hand model reconstruction method based on feature interaction introduces feature interaction processing in the hand pose estimation stage, enabling hand features and skeletal features to learn from each other more effectively, resulting in high accuracy in gesture recognition even under occlusion. Simultaneously, the application of sub-networks in feature extraction reduces network parameters and accelerates model convergence. Ultimately, the network output presents a hand shape containing surface texture information. Compared to traditional hand pose estimation results that only present a hand shape composed of points and lines, the hand reconstruction model in this invention more closely resembles the human hand shape, achieving a more realistic interactive experience during human-computer interaction and improving operational accuracy.

Claims

1. A hand model reconstruction method based on feature interaction, characterized in that, The specific steps are as follows: Step 1: Obtain a dataset consisting of hand gesture training images; Step 2: Preprocess the dataset obtained in Step 1 to obtain the preprocessed hand image; Step 3: Construct a hand pose estimation network model structure and predict the 3D hand pose position by combining the preprocessed hand image; The structure of the hand pose estimation network model includes a 2D hand pose estimation network and a 3D hand pose estimation network. The 2D hand pose estimation network comprises a ResNet-50 residual network, a softmax layer, and a sub-network. The ResNet-50 residual network extracts features from the preprocessed hand image input to the network, outputting pose-aware feature encodings that include joint-aware and skeletal-aware features. It also represents the predicted keypoint and bone positions using heatmaps and skeletal maps, respectively, generating joint heatmaps predicting the hand joint distribution and skeletal heatmaps predicting the bone distribution. The softmax layer enhances the contrast of the joint-aware and skeletal-aware features in the pose-aware features, fusing them with the corresponding joint and skeletal heatmaps to obtain a joint feature map with 21 joint points and a skeletal feature map containing 20 bones. The sub-network structure includes two convolutional layers and one MLP layer, used for channel and feature reduction of the joint and skeletal feature maps, respectively, to obtain reduced joint features. Skeletal features of John ; Between the 2D and 3D hand pose estimation networks, a feature interaction module is included. This module comprises semantic graph convolutional layers, self-attention layers, MLP layers, and the qkv mechanism in the transformer. The semantic graph convolutional layers and self-attention layers are used to update the reduced joint features; the qkv mechanism is used to implement the updated reduced joint features. Skeletal features of John The interaction, the joint features and bone features after the interaction are processed through a multi-head self-attention mechanism and an MLP layer to output the final joint features and the final bone features. The 3D hand pose estimation network contains multiple fully connected layers, and the predicted 3D hand pose position is obtained by regressing the final joint features and the final skeletal features. Step 4: Reconstruct the initial hand model based on the predicted 3D hand pose position; Step 5: Set the network parameters for hand pose estimation in Step 3, input the preprocessed hand image into the hand pose estimation network model for training, and save the trained hand pose estimation network model as output. Step 6: Input the preprocessed hand image into the trained hand pose estimation network model, input the output result into the reconstructed initial hand model for training, and save the trained model to obtain the reconstructed hand model.

2. The hand model reconstruction method based on feature interaction according to claim 1, characterized in that, Step 2 is as follows: Crop the hand gesture training images to 256. The image is 256 pixels in size, and then normalized to obtain the preprocessed hand image. In the data preprocessing stage, the joint at the wrist is selected as the root joint, and the bone from the root joint to the base joint of the middle finger is defined as the reference bone.

3. The hand model reconstruction method based on feature interaction according to claim 1, characterized in that, The specific process of feature extraction from the preprocessed hand image input into the network is as follows: For each joint in the image i ,use x and y Standard deviation in spatial direction σ To generate a heatmap, denoted as: (1) in u =[ u x ,u y ] represents a 2D joint index in the pixel plane; H i ( u ) represents the generated heatmap of the i-th joint; P i 2d This represents the predicted coordinates of the i-th joint in the pixel plane; For each bone j Connect two endpoints and Calculate the position of the intermediate joint v =[ v x ,v y and the angles of these endpoints θ Then use a 2D rotation matrix R θ Will u , v Convert to ,in ; Based on this, the resulting bone heatmap is represented as follows: (2) in, σ x and σ y It is the standard deviation of the Gaussian distribution along the bone direction in 2D space.

4. The hand model reconstruction method based on feature interaction according to claim 1, characterized in that, The semantic graph convolutional layer and self-attention layer are used to update the reduced joint features. The specific process is as follows: Reduced joint features f j Input semantic graph convolutional neural network generates semantic graph convolution updated joint features The update expression is: (3) Where || represents channel cascading, ReLU represents the nonlinear activation function. ω d Let ρ be the d-th row of the learnable matrix A. i and M d These represent the Softmax nonlinearity and the weighting matrix, respectively. Reduced joint features Through the self-attention layer, based on the complementary matrix Update the joint feature map, the complementary matrix For 21 A matrix of size 21 represents the connections between 21 joints. If a joint is not correctly connected, the corresponding row and column position is set to 1, indicating an incorrect connection between the two joints. The fully connected layer first converts the reduced joint features... Mapped to vectors q, k, and v of the same shape, vectors q, k, and v are responsible for calculating the attention mapping designed in the Transformer module, and the calculation expression is: (4) in σ Represents the Softmax function layer. For element-wise multiplication, ω These are the learnable weight parameters for the self-attention layer; Reduced joint features After semantic graph convolution and self-attention layer, the final updated joint features are obtained as follows: in, and These are the weight parameters that are set.

5. The hand model reconstruction method based on feature interaction according to claim 4, characterized in that, The specific process for outputting the final joint features and final bone features is as follows: For the final updated joint features f J and skeletal features f S First use the MLP layer to f J Convert to q J , k J and v J Skeletal features f S It is fed into the MLP layer and outputs q S , k S , v S The feature interaction module utilizes two attention matrices. a JS and a SJ Calculate joint features after processing by the qkv mechanism and skeletal features ; Two attention matrices a JS and a SJ Their corresponding eigenvectors v J and v S Perform element-wise multiplication; Joint features after QKV mechanism processing and skeletal features The residual connections are used to update the feature vectors respectively. v J and v s In addition, the two combined layers employ a multi-head self-attention mechanism and an MLP layer to act on the joint features processed by the qkv mechanism. and skeletal features Finally, the non-local content is retained, and the expression is: (5) (6) in, Indicates the final joint features. Indicates the final skeletal features. Provides matrix multiplication. ω J and ω S These are learnable weight parameters.

6. The hand model reconstruction method based on feature interaction according to claim 1, characterized in that, Step 4 is as follows: Step 4.1: Use the IKnet neural network to encode and train the joint positions in the predicted 3D hand pose position, and finally output the rotation angle Q of each joint. Step 4.2: Select the MANO hand model and estimate the shape parameters of the MANO model by using the joint positions in the predicted 3D hand pose. During training, calculate the shape parameters of the model. The expression is: (7) The first item ensures the bone length of the deformable hand model. Compared with the predicted 3D bone length The lengths match. Obtained from 3D predictions based on hand pose estimation. As the reference bone of the MANO model, the second term serves as the L2 regularizer for the shape parameter, and is passed through Weighting; Calculate E( ) according to formula (7) When the shape parameters of the corresponding MANO model are minimized, ; Step 4.3: Combine the shape parameters of the MANO model The initial hand model was reconstructed by determining the joint rotation angle Q.

7. The hand model reconstruction method based on feature interaction according to claim 1, characterized in that, Step 4.1 The specific process is as follows: Step 4.1.1: Construct the IKnet neural network, which is a batch-normalized 7-layer fully connected neural network. Except for the last layer which uses linear activation, the remaining layers use sigmoid as the activation function. Step 4.1.2: Encode the three-dimensional joint positions in the 3D hand pose as I = [X, D, X]. ref D ref ], where the encoded input is given to the IKnet neural network and the output is the global rotation angle Q of each joint.

8. The hand model reconstruction method based on feature interaction according to claim 1, characterized in that, The network parameters in step 3 include the batch size, learning rate, training epochs, method for initializing model parameters, and optimization method.

Images