A 3D hand pose estimation method based on multi-level topology structure

Abstract

The application provides a 3D hand posture estimation method based on a multi-level topological structure, improves the robustness of joint feature representation and the robustness of the model, and can obtain a robust 3D hand posture. The method comprises the following steps: step 1: inputting a monocular RGB image into a multi-level topological structure network, performing feature extraction, feature splicing, node coordinate estimation and feature interaction operation, and then outputting a 3D hand posture from the multi-level topological structure network; step 2: inputting the 3D hand posture obtained in step 1 and a real hand posture into a loss function of a 3D hand posture estimation network model, training the 3D hand posture estimation network model by using a mean square error loss function, and obtaining model parameters; and step 3: loading the model parameters in step 2, inputting a new monocular RGB image into the trained 3D hand posture estimation network model, and obtaining a corresponding 3D hand posture.

Description

Technical Field

[0001] This invention belongs to the field of pose estimation technology in the field of computer vision and artificial intelligence, and specifically relates to a 3D hand pose estimation method based on a multi-level topology. Background Technology

[0002] 3D hand pose estimation is an important research topic in computer vision and human-computer interaction. Its goal is to infer the three-dimensional pose of the hand, i.e., the three-dimensional position and orientation of each joint of the hand, from an input image or video. This technology plays a crucial role in many applications, including virtual reality, augmented reality, gesture recognition, human-computer interaction, and medical rehabilitation.

[0003] However, 3D hand pose estimation currently faces many challenges. First, due to the high degree of freedom of the hand (21 joints per hand), the pose space is complex, and possible pose variations are diverse. Second, recovering 3D information from 2D images is a difficult task due to factors such as occlusion, lighting variations, and image noise. Furthermore, differences in hand size, shape, and color among individuals further complicate the problem. Meanwhile, to meet the requirements of ease of dissemination and deployment, researchers have focused on estimating 3D hand pose from monocular RGB images. Due to device limitations, 3D hand pose estimation based on monocular RGB images suffers more severely from self-occlusion and depth blur problems. Therefore, researchers in 3D hand pose estimation based on monocular RGB images utilize skeletal physical constraints and kinematic chains of the hand structure to assist in constructing 3D hand poses, and on this basis, employ graph convolutional networks (GCNs) capable of constructing hand topology structures to build the hand topology. For example, the Bihand model employs a three-stage network model to estimate 3D hand pose: SeedNet, which obtains 2D heatmaps and contour maps from monocular RGB images; LiftNet, which upscales the 2D heatmaps and contour maps to 3D heatmaps and depth maps; and SIKNet, which estimates the 3D hand pose and kinematic chain from the 3D heatmap. Estimating the 3D heatmap first serves as an interim stage, reducing the difficulty of directly estimating the 3D pose. Simultaneously, inverse kinematic constraints are used in the loss function to constrain the estimated 3D hand pose. Subsequently, SemGCN and HGCN models construct the physical topology of the skeleton, fusing feature information from first-order and higher-order neighbors, respectively.

[0004] While the aforementioned studies have all achieved excellent results in their respective fields, they either involve complex network models or only consider a single hand topology, which negatively impacts the accuracy of the models on the test set, and the robustness of the estimated 3D poses needs to be improved. Summary of the Invention

[0005] To address the aforementioned problems in existing technologies, this invention proposes a 3D hand pose estimation method based on a multi-level topology, which improves the robustness of joint feature representation and the robustness of the model, and can obtain robust 3D hand poses.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] A 3D hand pose estimation method based on a multi-level topology includes the following steps:

[0008] Step 1: Input the monocular RGB image into a multi-level topology network, perform feature extraction, feature stitching, node coordinate estimation, and feature interaction operations, and then output the 3D hand pose of the multi-level topology network.

[0009] Step 2: Input the 3D hand pose obtained in Step 1 and the actual hand pose into the loss function of the 3D hand pose estimation network model, and use the mean squared error loss function to train the 3D hand pose estimation network model to obtain the model parameters.

[0010] Step 3: Load the model parameters from Step 2, input the new monocular RGB image into the trained 3D hand pose estimation network model, and obtain the corresponding 3D hand pose.

[0011] Furthermore, step 1 specifically includes:

[0012] Step 1.1: First, for the RHD dataset, use the Hourglass network to extract features from the input monocular RGB image, and then input it into the Head module to obtain the feature map of the hand joints;

[0013] Step 1.2: The feature maps of hand joints are processed through graph convolutional network (GCN) layers and convolutional layers to form local hand features and global hand features, respectively. The local hand features and global hand features are then concatenated with the feature maps of hand joints to form new joint feature maps.

[0014] Step 1.3: Divide the new joint feature map formed in Step 1.2 into two branches with similar architecture, which are used to estimate the uv coordinates and d coordinates of the joints respectively;

[0015] Step 1.4: Input the new joint feature map into the TNTPose module. The Transformer inside the TNTPose module performs information interaction on different features of the joint. In order for the internal Transformer to complete the information interaction task of multiple features, this invention designs a multi-feature cross-attention module to perform information interaction on multiple features.

[0016] Step 1.5: The joint features processed by the internal Transformer are then interacted with by the external Transformer to output the processed results. These processed results are divided into the processed results of the branch that estimates the uv coordinates and the processed results of the branch that estimates the d coordinates.

[0017] Step 1.6: The processed results of the branches estimating uv coordinates and d coordinates are concatenated to finally output the 3D pose of the hand.

[0018] In this process, the multi-level topological network passes the feature map output of the Hourglass network to the Head module to obtain the hand joint feature vector V. j ∈R n×d Then, the network uses a GCN layer to set up an adjacency matrix based on prior knowledge of the hand, and constructs a local hand feature vector V from the hand joint feature vectors. p ∈R n×d Simultaneously, a global hand feature vector V is constructed from the hand joint feature vectors through convolution with a kernel size of 1. h ∈R n×d Therefore, a joint is represented by the hand joint feature vector, the hand local feature vector, and the hand global feature vector. Then, the TNTPose module of the network uses an internal Transformer to facilitate the interaction between features at different levels. In order to enable the three parts of features to interact effectively, this invention proposes a new multi-feature cross-attention module (MCA) to enhance the internal Transformer. The process is represented by equations (1)-(5):

[0019] X = Concat(V) j V p V h ),#(1)

[0020] V j,l =MLP(MCA(LN(V) j,l-1 ),LB(V p,l-1 ),LN(V h,l-1 ))),#(2)

[0021] V p,l =MLP(MCA(LN(V) p,l-1 ),LN(V j,l-1 ),LN(V h,l-1 ))),#(3)

[0022] V h,l =MLP(MAC(LN(V h,l-1 ),LN(V j,l-1 ),LN(V p,l-1))),#(4)

[0023] X l =X l-1 +Concat(V j,l V p,l V h,l ).#(5)

[0024] Where X represents the new hand representation obtained by concatenating features from different levels; Concat(*) is the concatenation operation; V j V is the feature vector of the hand joints; p V is a local feature vector of the hand. h V is the global feature vector of the hand. *,l It is the output of the l-th internal Transformer; MCA(*) is the multi-feature cross-attention module; MLP(*) is the multilayer perceptron mechanism; LN(*) is the normalization operation; V *,l-1 Represents the features of the (l-1)th layer; X l This represents the hand gesture at level l; X l-1 This represents the hand symbol in the (l-1)th layer.

[0025] The TNTPose module uses a standard Transformer as an external Transformer to construct global relationships between joints and interact with different joint features, as expressed by formulas (6)-(8):

[0026] χ=[X0,X1,…,X n ],χ∈R n×3d #(6)

[0027] χ′ l =χ l-1 +MSA(LN(χ l-1 )),#(7)

[0028] χ l =χ l +MLP(LN(χ′ l )).#(8)

[0029] Where, χ∈R n×3d It is a joint feature composed of n different feature vectors of length 3d each. X * Represents the characteristics of each joint; χ' l χ represents the joint features obtained in layer l after MCA processing; l-1 Indicates the joint features of layer l-1; χ l The l-th layer represents the final joint features obtained; MCA(*) is a multi-feature cross-attention module; MLP(*) is a multi-layer perception mechanism.

[0030] The Multi-Feature Cross-Attention Module (MCA) has four inputs: Query, Key_0, Key_1, and Value, representing the joint feature query, joint feature key 0, joint feature key 1, and joint feature value, respectively. First, Query is used to calculate attention scores with Key_0 and Key_1, and then the two attention scores are added together to obtain the final attention score. Finally, Value is used to calculate the final attention score to obtain the final output result. The Multi-Feature Cross-Attention Module can be represented by formulas (9)-(12):

[0031]

[0032] X = MLP(Scores*Value).#(12)

[0033] Where Linear(*) represents a linear layer; T represents the matrix transpose operation; Score0 represents the calculated attention score of 0; Score1 represents the calculated attention score of 1; Scores represents the overall attention score; and Softmax represents the softmax activation function. denoted by the square root of the feature dimension; X represents the joint feature.

[0034] Furthermore, step 2 specifically includes:

[0035] The estimated 3D hand pose obtained in step 1 and the actual 3D pose are input into the loss function to train the network. This invention proposes to use the Euclidean distance of the joint positions and the mean square error of the finger bone length as loss functions to train the network model. These two loss functions can be expressed by formulas (13)-(14):

[0036]

[0037] In the formula, These are the 3D coordinates of the joints estimated by the model, with the subscript i representing the joint number; The actual 3D coordinates of the joints; The estimated finger bone length is calculated from the 3D coordinates obtained through model estimation. The actual finger bone length is calculated using real joint coordinates; J represents the number of joints, J = 21.

[0038] Furthermore, step 3 specifically includes:

[0039] Load the model parameters best_state.pth trained in step 2, input the monocular RGB image into the model, and output the 3D coordinates of the hand joints and the corresponding evaluation metrics.

[0040] The beneficial effects of this invention include:

[0041] By using a multi-level topological network, hand features at different scales and levels in the image are captured and integrated to more accurately estimate 3D hand pose, improving the accuracy and robustness of the estimation. The 3D hand pose estimation network model is trained using the mean squared error loss function to learn the mapping relationship from the input image to the output 3D hand pose, enhancing the model's generalization ability to handle complex hand poses and scenes.

[0042] Specifically, this invention constructs a multi-level topological structure for the hand, builds new hand joint representations based on this structure, and uses a Transformer-in-Transformer architecture to simultaneously interact with different joint features, thereby improving the robustness of the joint feature representation and the robustness of the model. This allows the model to construct a more robust hand structure when estimating 3D hand pose. Furthermore, to enable the internal Transformer to effectively establish information interaction between multiple features, this invention designs a multi-feature cross-attention block, which can extract feature information at different scales. In addition, this invention combines the Euclidean distance of the joint 3D coordinates and the mean square error of the finger bone length to increase the penalty for incorrect estimation, thus ensuring the accuracy of the estimated 3D hand pose. Finally, a high-precision 3D hand pose is obtained through an end-to-end approach. Attached Figure Description

[0043] Figure 1 This is a framework diagram of the 3D hand pose estimation model based on a multi-level topology structure of the present invention. Detailed Implementation

[0044] The embodiments of the present invention are implemented under the premise of the technical solution of the present invention, and detailed implementation methods and specific operation processes are given. However, the protection scope of the present invention is not limited to the following embodiments.

[0045] Example 1

[0046] This embodiment uses Windows as the development environment, PyCharm as the development platform, and Python as the development language. It employs the 3D hand pose estimation model based on multi-level topology of this invention to perform 3D hand pose estimation on monocular RGB images.

[0047] In this embodiment, the estimation method for a 3D hand pose estimation model based on a multi-level topology includes the following steps:

[0048] Step 1: Input the monocular RGB images from the training set into a program such as... Figure 1 The 3D hand pose estimation network model shown is used for training.

[0049] Step 2: Using the required monocular RGB image as input, load the model trained and saved in Step 1 to obtain the corresponding 3D hand pose and corresponding evaluation metrics. This invention uses the average error of each joint, the percentage of correct joints at different thresholds, and the region under the 3D PCK curve as evaluation metrics, namely MPJPE, 3D PCK, and AUC. The calculation method of MPJPE can be expressed by formula (15):

[0050]

[0051] Where N is the number of joints; J^{pred} is the predicted 3D coordinates of the joints; and J^{gt} is the actual 3D coordinates of the joints. The calculation method of 3D PCK can be expressed by formula (16):

[0052]

[0053] In the formula, the subscript i represents the joint number; k is the k-th threshold T k ;p represents the p-th hand gesture; d pi The Euclidean distance between the predicted and actual values ​​of the joint with pose number i for the p-th pose; T is the scale factor for the p-th pose; k As the threshold, T k =0mm-50mm.

[0054] Table 1 compares the model's performance on the RHD and STB datasets.

[0055]

[0056] Based on the above steps, this invention is compared with models proposed by Zimmermann & Brox, Moon et al., Spurr et al., Cai et al., Ge et al., Boukhayma et al., Yang et al., Stergioulas et al., Li et al., Yang et al., Zhou et al., Lin et al., Theodoridis et al., and Ivashechkin et al. As can be seen from Table 1, the method proposed in this invention generally outperforms other methods in terms of accuracy on two common datasets.

[0057] The foregoing description of specific exemplary embodiments of the invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A 3D hand pose estimation method based on a multi-level topology, characterized in that, The method includes the following steps: Step 1: Input the monocular RGB image into a multi-level topology network, perform feature extraction, feature stitching, node coordinate estimation, and feature interaction operations, and output the 3D hand pose. Step 2: Input the 3D hand pose obtained in Step 1 and the actual hand pose into the loss function of the 3D hand pose estimation network model, and use the mean squared error loss function to train the 3D hand pose estimation network model to obtain the model parameters. Step 3: Load the model parameters from Step 2, input the new monocular RGB image into the trained 3D hand pose estimation network model, and obtain the corresponding 3D hand pose. Step 1 specifically includes: Step 1.1: First, for the RHD dataset, use the Hourglass network to extract features from the input monocular RGB image, and then input it into the Head module to obtain the feature map of the hand joints; Step 1.2: The feature maps of hand joints are processed through graph convolutional network (GCN) layers and convolutional layers to form local hand features and global hand features, respectively. The local hand features and global hand features are then concatenated with the feature maps of hand joints to form new joint feature maps. Step 1.3: Divide the new joint feature map formed in Step 1.2 into two branches with similar architecture, which are used to estimate the uv coordinates and d coordinates of the joints respectively; Step 1.4: Input the new joint feature map into the TNTPose module. The Transformer inside the TNTPose module uses the multi-feature cross-attention module to exchange information on different features of the joint. Step 1.5: The joint features processed by the internal Transformer are then interacted with by the external Transformer to output the processed results. These processed results are divided into the processed results of the branch that estimates the uv coordinates and the processed results of the branch that estimates the d coordinates. Step 1.6: After concatenating the processed results of the branch estimating uv coordinates and the branch estimating d coordinates, the final 3D hand pose is output. In step 1.4, the Transformer inside the TNTPose module uses a multi-feature cross-attention module to exchange information on different features of the key points. This process is represented by formulas (1)-(5): in, This represents a new hand representation obtained by splicing features from different levels. It's a splicing operation; This represents the feature vector of the hand joints; This represents the local feature vector of the hand. This is the global feature vector of the hand; Indicates the first The output of an internal Transformer; It is a multi-feature cross-attention module; It is a multi-layered sensing mechanism; This is a normalization operation; Indicates the first Features of the layer; Indicates the first The hand gesture representing the layer; Indicates the first The hand gesture representing the layer; In step 1.5, the TNTPose module uses the standard Transformer as the external Transformer to construct the global relationship between joints and interact with different joint features. This process is represented by formulas (6)-(8): in, It consists of n groups of 3 Joint features composed of feature vectors of different dimensional lengths; express Features of each joint; This indicates the number of lines after MCA processing. Joint features obtained from layers; Indicates the first Joint features of the layers; Indicates the first The final joint features obtained from the layers; It is a multi-feature cross-attention module; It is a multi-layered sensing mechanism; The multi-feature cross-attention module (MCA) has four inputs: Query, Key_0, Key_1, and Value, representing the joint feature query, joint feature key 0, joint feature key 1, and joint feature value, respectively. First, Query calculates attention scores with Key_0 and Key_1, and then the two attention scores are added together to obtain the final attention score. Finally, the final attention score is calculated using Value to obtain the final output result. The multi-feature cross-attention module is represented by formulas (9)-(12): in, Indicates a linear layer; This represents the matrix transpose operation; This indicates that the calculated attention score is 0; This represents the calculated attention score of 1. This represents the overall attention score; This represents the softmax activation function; This indicates taking the square root of the feature dimension; Indicates joint features; In step 2, the Euclidean distance of the joint positions and the mean square error of the finger bone length are used as loss functions to train the network model together. These two loss functions are expressed by formulas (13)-(14): In the formula, These are the 3D coordinates of the joints estimated by the model, with the subscript i representing the joint number; The actual 3D coordinates of the joints; The estimated finger bone length is calculated from the 3D coordinates obtained through model estimation. The actual finger bone length is calculated using real joint coordinates; J represents the number of joints. .

Images