Hand pose and shape estimation method based on segmentation map guidance and regularization constraints

Abstract

The present invention discloses a hand posture and shape estimation method based on segmentation map guidance and regularization constraints, which mainly solves the problem of low accuracy of estimation results in the case of hand-object occlusion in the existing technology. Its implementation scheme is: dividing the data set into a training data set and a test data set; constructing a hand posture and shape estimation network based on segmentation map guidance and regularization constraints composed of a feature extraction backbone, a regularization constraint pyramid module, a segmentation guidance module and a regression module; using the training data set to train the network; inputting the test data set into the trained network to obtain the hand estimation result. The present invention introduces a segmentation guidance module to utilize hand segmentation maps of different visibilities to assist in estimation under hand-object occlusion, and introduces a regularization constraint pyramid module to constrain network features. Compared with the existing hand posture and shape estimation methods, the present invention significantly improves the hand estimation effect and can be used for intermediate processing of hand image interpretation.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a hand posture and shape estimation method, which can be used for intermediate processing of hand image interpretation. Background Art

[0002] In recent years, computer vision has evolved from two-dimensional to three-dimensional processing, and estimating the three-dimensional pose and shape of a hand from hand images has become increasingly important. With the impressive performance of deep learning in fields such as computer vision, speech signal processing, and natural language processing, integrating deep learning methods with hand pose and shape estimation has become a hot topic. Deep learning methods employ a layer-by-layer learning approach, continuously mining the inherent attributes of training data to achieve an abstract representation of the data.

[0003] In their paper "Bihand: Recovering Hand Mesh with Multi-stage Bisected Hourglass Networks," Yang Lixin et al. proposed a method for using hand segmentation maps for hand pose and shape estimation. Based on hourglass networks, the method uses a first hourglass network to output a hand segmentation map and a 2D heatmap of hand joints. A second hourglass network then outputs a depth map and a 3D heatmap of hand joints. Finally, the resulting 3D heatmap of hand joints is used to estimate hand pose and shape. This method further regresses the hand segmentation map to obtain a depth map, thereby incorporating depth information into the 2D visual information.

[0004] In his paper "Towards Accurate Alignment in Real-Time 3D Hand-Mesh Reconstruction," Tang Xiao proposed a method for using hand segmentation maps to aid hand pose and shape estimation. This method uses the hand segmentation map as an additional task to assist network training. During network training, this method outputs the hand segmentation map and the 2D positions of the hand joints. During testing, this method crops the 2D hand joints and the network output of the hand segmentation map, making the model more efficient.

[0005] Hand images often contain hand-object interactions. The aforementioned hand pose and shape estimation methods, when using hand segmentation maps, fail to distinguish between different levels of hand visibility. Specifically, they fail to consider the application of segmentation maps when the hand is partially occluded. This lack of processing for hand segmentation maps with varying visibility levels compromises the accuracy of estimation when the hand is occluded by an object.

[0006] To further assist the hand pose and shape estimation process, some methods use a progressive optimization strategy to constrain features to achieve better estimation based on the highly nonlinear relationship between the task of estimating the 3D pose and shape of the hand from RGB images.

[0007] In the article "Interacting two-hand 3D pose and shape reconstruction from a single color image," Baowen Zhang proposed using single-scale features for progressive optimization to achieve hand pose and shape estimation. This method adds a progressive optimization module at the end of the network, which cyclically processes and constrains the hand's high-level features to obtain an optimized estimate.

[0008] In his paper "Hand image understanding via deep multi-task learning," Zhang Xiong proposed a method for hand pose and shape estimation using a progressive optimization approach. This method divides the entire process into multiple layers, outputting an estimated hand at each layer to implement feature constraints, and then outputting an optimal estimate at the final layer.

[0009] The above two methods incorporate progressive optimization into hand posture and shape estimation. However, since they only utilize single-scale features to achieve progressive optimization, they do not link multi-scale features with progressive optimization. In other words, they do not consider the multi-scale parts of the hand, such as the palm, fingers, and joints, during the progressive optimization process. Therefore, these methods lack the expression and constraints of multiple pairs of scale features in progressive optimization, and still suffer from the problem of low estimation accuracy. Summary of the Invention

[0010] The purpose of the present invention is to address the deficiencies of the above-mentioned existing technologies and propose a hand posture and shape estimation method based on segmentation map guidance and regularization constraints, so as to utilize hand segmentation maps of different visibilities and annotated constraints to assist hand posture and shape estimation, thereby improving the estimation accuracy in the case of hand-object occlusion.

[0011] To achieve the above object, the implementation scheme of the present invention includes the following steps:

[0012] (1) Construct training and test datasets:

[0013] (1a) Divide the training samples and test samples according to the given partitioning method of the dataset;

[0014] (1b) Rendering based on the label information E of d training samples to obtain the overall hand segmentation map S w ;

[0015] (1c) Construct a training dataset D based on the input images, label information, and overall hand segmentation map of d training samples:

[0016]

[0017] Among them, I i is the input image of the i-th training sample, E i is the label information of the i-th training sample, is the overall segmentation map of the hand of the i-th training sample;

[0018] (1d) Construct a test dataset M based on the input images, 3D positions of hand joints, and 3D positions of hand surface points of m test samples:

[0019]

[0020] in, is the input image of the jth test sample, is the 3D position of the hand joint point of the j-th test sample, is the 3D position of the hand surface point of the jth test sample;

[0021] (2) Construct a hand pose and shape estimation network G based on segmentation map guidance and regularization constraints:

[0022] (2a) Establishing a regularized constraint pyramid module consisting of three regularized constraint modules and an hourglass network in cascade order, which is used to constrain the multi-scale features in the feature extraction backbone to obtain hand structure features;

[0023] (2b) establishing a segmentation guidance module consisting of three visible segmentation blocks and an overall segmentation module to obtain a visible segmentation map of the hand and an overall segmentation map of the hand;

[0024] (2c) Select the known feature extraction backbone and regression module, connect the feature extraction backbone and the regularization constraint pyramid module in series, and then connect the outputs of the two to the segmentation guidance module and the regression module in cascade to form a hand posture and shape estimation network G based on segmentation map guidance and regularization constraint:

[0025] (3) Using a mini-batch gradient descent algorithm, the hand pose and shape estimation network G based on segmentation map guidance and regularization constraints is trained using the training dataset D until the network loss function is in a convergence state, and the trained hand pose and shape estimation network is obtained;

[0026] (4) Input the test data set M into the trained hand posture and shape estimation network to obtain the estimation results corresponding to the test data.

[0027] Compared with the prior art, the present invention has the following advantages:

[0028] 1) Improve the robustness of hand pose and shape estimation in scenes with object occlusion.

[0029] The present invention divides the hand segmentation map into a visible segmentation map and an overall segmentation map when in use. The visible segmentation map is used to capture the visible information of the hand, and based on the visible segmentation map of the hand, the features after regularization constraint are further used to obtain the overall segmentation map of the hand. By fusing hand segmentation maps with different visibilities, the hand-object occlusion problem in hand posture and shape estimation is alleviated, and the robustness of hand posture and shape estimation in hand-object interaction scenarios is improved.

[0030] 2) Reduce hand estimation error.

[0031] The present invention performs hand posture and shape estimation based on segmentation map guidance and regularization constraints. By introducing a segmentation guidance module to utilize hand segmentation maps of different visibilities to assist in estimation under hand-object occlusion, and by introducing a regularization constraint pyramid module to utilize progressive optimization to constrain network features, with the addition of these two modules, the hand posture and shape estimation network based on segmentation guidance and regularization constraints can achieve lower estimation errors in fused datasets and real datasets, respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is an implementation flow chart of the present invention;

[0033] Figure 2 This is a diagram of the hand posture and shape estimation network structure based on segmentation map guidance and regularization constraints in the present invention;

[0034] Figure 3 yes Figure 2 The residual block structure diagram in ;

[0035] Figure 4 yes Figure 2 The structural diagram of the regular constraint module;

[0036] Figure 5 This is a comparison chart of the hand estimation results and the true value in scenes with different degrees of occlusion on the Obman dataset;

[0037] Figure 6 This is a comparison chart of the hand estimation results of the present invention and the existing method Handocc in the real hand-object interaction scenario of the HO3D-v2 dataset. DETAILED DESCRIPTION

[0038] The embodiments and effects of the present invention are described in further detail below with reference to the accompanying drawings.

[0039] Reference Figure 1, the implementation steps of the present invention include the following:

[0040] Step 1. Build training and test datasets.

[0041] (1.1) Divide the training samples and test samples according to the given division method of the dataset;

[0042] (1.2) According to the label information of d training samples Rendering to get the whole hand segmentation map S w ,

[0043] in: is the hand visible segmentation map of the i-th training sample, is the hand joint rotation parameter of the i-th training sample, is the hand shape parameter of the i-th training sample, is the 3D position of the hand joint point of the i-th training sample, V i g is the 3D position of the hand surface point of the i-th training sample, is the 2D position of the hand joint of the i-th training sample;

[0044] (1.3) Construct a training dataset D containing d samples:

[0045]

[0046] Among them, I i is the input image of the i-th sample, E i is the label information of the i-th sample, is the overall segmentation map of the hand of the i-th sample;

[0047] (1.4) Construct the test dataset M based on the input images of m test samples, the 3D positions of the hand joint points, and the 3D positions of the hand surface points:

[0048]

[0049] in, is the input image of the jth test sample, is the 3D position of the hand joint point of the j-th test sample, is the 3D position of the hand surface point of the jth test sample.

[0050] Step 2. Build a hand pose and shape estimation network G based on segmentation map guidance and regularization constraints.

[0051] Reference Figure 2 , the specific implementation of this step is as follows:

[0052] (2.1) The existing feature extraction backbone is selected, which consists of a convolutional layer and four residual blocks cascaded in sequence to extract multi-scale features of the image. Each residual block includes a cascaded skip residual layer and a retained residual layer. Each residual layer is divided into two parallel paths, such as Figure 3 As shown, where:

[0053] The skip residual layer has a first path structure of a first convolutional layer, a first filter response normalization layer, a second convolutional layer, and a second filter response normalization layer in sequence; a second path structure of a convolutional layer and a filter response normalization layer in sequence; the output of these two paths is activated using a ReLU activation function;

[0054] The structure of the first retained residual layer is: first convolution layer → first filter response normalization layer → second convolution layer → second filter response normalization layer; the second retained residual layer is itself mapped; each filter response normalization layer is activated by a ReLU activation function;

[0055] (2.2) Construct a regular constraint pyramid module, which consists of three regular constraint modules and an hourglass network cascaded in sequence, and is used to constrain the multi-scale features in the feature extraction backbone to obtain hand structure features, such as Figure 4 As shown in Figure 2, each regular constraint module consists of a downsampling block and an upsampling block in cascade. The downsampling block consists of three downsampling layers and a hand model regressor in cascade. The upsampling block consists of three decoding upsampling layers and a linear upsampling layer in cascade, where:

[0056] The structure of each decoding upsampling layer is a linear layer, a first deconvolution layer, a first retained residual layer, a second deconvolution layer, a second retained residual layer, a third deconvolution layer, and a third retained residual layer cascaded in sequence;

[0057] The hand model regressor consists of three linear layers cascaded with a hand model layer, where the hand model layer is used to map parameters to hand joint points and hand surface points;

[0058] (2.3) Constructing a segmentation guidance module, which consists of three visible segmentation blocks and a global segmentation module, to obtain visible and global segmentation maps of the hand. The visible segmentation block consists of a first convolutional layer, a first filter response normalization layer, a second convolutional layer, and a second filter response normalization layer; the global segmentation module consists of an hourglass network cascaded with two residual layers;

[0059] (2.4) Selecting an existing regression module, which consists of three residual blocks and a hand model regressor cascaded in sequence, to obtain the final hand pose and shape estimation results;

[0060] (2.5) After connecting the feature extraction backbone and the regularization constraint pyramid module in series, their outputs are connected to the segmentation guidance module and the regression module in cascade to form a hand posture and shape estimation network G based on segmentation map guidance and regularization constraint.

[0061] Step 3. Iteratively train the hand pose and shape estimation network G based on segmentation map guidance and regularization constraints.

[0062] (3.1) Randomly select b training samples in batches from the training dataset D The input image I t Input to the feature extraction backbone, after passing through four residual blocks, four multi-scale image features are obtained in turn

[0063] (3.2) Input the four multi-scale image features obtained in (3.1) into the regularized constrained pyramid module to obtain the hand structure feature F a and the two-dimensional heat map H of the hand joint;

[0064] (3.2a) The fourth scale image feature Input to the first regular constraint module, and get the downsampled features after three downsampling layers in the downsampling block Fourth downsampling feature The hand model regressor of the downsampling block obtains the fourth-level hand joint rotation parameter θ 4 and hand shape parameter β 4 3D position of hand joint J 4 and the 3D position V of the wrist surface point 4 , the fourth down-sampled feature Input to the upsampling block, output the fourth decoded feature after three decoding upsampling layers It is related to the fourth scale image feature After splicing, it is input to the linear upsampling layer, and its output is consistent with the third-scale image feature Add together to get the third regular constraint feature

[0065] (3.2b) The third regular constraint feature Input to the second regular constraint module, and obtain the downsampled features after three downsampling layers in the downsampling block The third downsampled feature The hand model regressor of the downsampling block obtains the third-level hand joint rotation parameter θ 3 and hand shape parameter β 3 3D position of hand joint J 3 and the 3D position V of the wrist surface point 3 ; The third downsampled feature Input to the upsampling block, output the third decoded feature after three decoding upsampling layers The third decoding feature With the third regular constraint feature After splicing, it is input to the linear upsampling layer, and its output is consistent with the second scale image feature Add together to get the second regular constraint feature

[0066] (3.2c) The second regular constraint feature Input to the third regular constraint module, and obtain the second downsampling feature after three downsampling layers in the downsampling block The second downsampled feature The hand model regressor of the downsampling block obtains the second-level hand joint rotation parameter θ 2 and hand shape parameter β 2 3D position of hand joint J 2 and the 3D position V of the wrist surface point 2 ; Downsample features Input to the upsampling block, output the second decoded feature after three decoding upsampling layers The second decoding feature With the second regular constraint feature After splicing, it is input to the linear upsampling layer, and its output is consistent with the first scale image feature F1 s Add together to get the first regular constraint feature F1 y ;

[0067] (3.2d) The first regular constraint feature F1 y After passing through the hourglass network, the output is the hand structure feature F a , for the hand structure feature F a Perform linear transformation to obtain the two-dimensional heat map H of the hand joint;

[0068] (3.3) The four multi-scale image features Hand structure features F a The hand joint two-dimensional heat map H is input into the segmentation map guidance module and the fusion feature F is output. t :

[0069] (3.3a) The third scale image feature and the fourth scale image features after upsampling After splicing, input it into the first visible segmentation block to obtain the third visible segmentation feature

[0070] (3.3b) The second scale image feature and the third visible segmentation feature after upsampling After splicing, input it into the second visible segmentation block to obtain the second visible segmentation feature

[0071] (3.3c) The first scale image feature F1 s and the second visible segmentation feature after upsampling After splicing, it is input into the third visible segmentation block to obtain the predicted hand visible segmentation map

[0072] (3.3d) The hand structure feature F a and predicted hand visible segmentation map After addition, it is input into the hourglass network of the overall segmentation module, and its output is compared with the predicted hand visible segmentation map Add together to get the overall segmentation feature F w ;

[0073] (3.3e) The overall segmentation feature F w Input to the two residual layers of the overall segmentation module and output the predicted overall segmentation map of the hand

[0074] (3.3f) The hand structure feature F a , predicted hand visible segmentation map Predicted overall hand segmentation map and the hand joint two-dimensional heat map H to obtain the fusion feature F t ;

[0075] (3.4) The fusion feature F t Input to the regression module and output the first-level prediction result (θ 1 ,β 1 ,J 1 ,V 1 ), where θ 1 is the predicted first-level hand joint rotation parameter, β 1 is the predicted first-level hand shape parameter, J 1 is the predicted 3D position of the first-level hand joint point, V 1 is the predicted 3D position of the first-level hand surface point;

[0076] (3.5) Define the loss of the hand pose and shape estimation network G under regular constraints guided by the segmentation map as Loss, which is expressed as follows:

[0077]

[0078] in, is the loss of the hand visible segmentation map, is the loss of the overall hand segmentation map, L his the loss of the two-dimensional position of the hand joint, L r is the hand parameter regression loss, which are expressed as follows:

[0079]

[0080]

[0081]

[0082] L r =1.0·ME1+0.6·ME2+0.4·ME3+0.2·ME4

[0083] In the formula, N represents the number of samples, Y n Represents the predicted hand visible segmentation map of the nth training sample The flattened form of Represents the hand visible segmentation map label S of the nth training sample v The flattened form of Z n Represents the predicted hand segmentation map of the nth training sample The flattened form of Represents the overall hand segmentation label S of the nth training sample w The flattened form of ; MSE represents the mean square error function, represents the two-dimensional position of the hand joint obtained from the predicted two-dimensional heat map H of the hand joint of the n-th training sample, The label representing the two-dimensional position of the hand joint of the nth training sample; ME k is the loss of the k-th level hand model result, where k is 1, 2, 3, or 4;

[0084] In the formula and The 3D positions of hand joint points, 3D positions of hand surface points, hand model rotation parameters and hand model shape parameters output by the k-th level hand model of the n-th sample are respectively. and are the true values ​​of the 3D position of the hand joint point, the 3D position of the hand surface point, the hand model rotation parameters, and the hand model shape parameters of the nth sample;

[0085] (3.6) Update the network parameters using the gradient descent algorithm based on the loss Loss obtained in (3.5);

[0086] (3.7) Repeat (3.1) to (3.6) until the loss Loss described in (3.5) is in a converged state, and the trained hand posture and shape estimation network G based on segmentation map guidance and regularization constraints is obtained.

[0087] Step 4. Input the test data set M into the trained hand posture and shape estimation network to obtain the estimation results corresponding to the test data.

[0088] (4.1) Take c test samples in batches from the test set M in sequence The input image Input the trained hand posture and shape estimation network G based on segmentation map guidance and regularization constraints to obtain the estimation results corresponding to c test samples in The 3D position of the first-level hand joints output by the network, The 3D position of the first-level hand surface points output by the network;

[0089] (4.2) Repeat (4.1) until the test set M is completely traversed and the estimation results corresponding to all test data are obtained.

[0090] The effects of the present invention can be further illustrated by the following simulation.

[0091] 1. Simulation Data

[0092] Two public datasets are used for simulation verification in this invention, namely the synthetic dataset Obman and the real dataset HO3D-v2.

[0093] The synthetic dataset Obman has a total of approximately 141,000 training samples and 6,200 test samples. The dataset uses an automatic generation method to obtain images in hand-object interaction scenarios. The scenes and objects in this dataset are highly diverse.

[0094] The real dataset HO3D-v2 has a total of approximately 66,000 training samples and 11,000 test samples. The dataset records real hand-object interaction scenes and uses an automatic method to annotate the dataset.

[0095] 2. Simulation content

[0096] In simulation 1, the proposed method is compared with three methods that perform well on the synthetic dataset Obman. The dataset is divided into a training dataset and a test dataset using a publicly available dataset partitioning method. The estimated average error of hand joints is used as the evaluation indicator. The comparison results are shown in Table 1.

[0097] Table 1 Estimation results on the Obman dataset

[0098] method Average error of hand joints Hasson et al. 11.6 mm Grasping Field 11.3 mm Chen et al. 9.6 mm Tse et al. 9.1 mm The present invention 8.8 mm

[0099] In simulation 2, the method of the present invention is compared with three methods that perform well on the real dataset HO3D-v2. The dataset is divided into a training dataset and a test dataset using a public dataset partitioning method. The estimated average error of hand joint points, the average error of hand joint points after Prof. Roberts alignment, and the average error of hand surface points after Prof. Roberts alignment are used as evaluation indicators. The comparison results are shown in Table 2.

[0100] Table 2 Estimation results on the HO3D-v2 dataset

[0101]

[0102] Simulation 3, using the present invention to estimate hands in scenes with different degrees of occlusion on the Obman dataset, the results are as follows Figure 5 , where the a-th column is the original image, the b-th column is the true annotation of the hand, and the c-th column is the estimated result of the hand obtained by the present invention.

[0103] Simulation 4: Using the present invention and existing methods to perform hand estimation on the HO3D-v2Obman dataset in a scene of real hand-object interaction, the results are as follows: Figure 6 , where the a-th column is the original image, the b-th column is the estimation result of Handocc, a better method on the HO3D-v2 dataset, and the c-th column is the hand estimation result obtained by the present invention.

[0104] 3. Simulation effect analysis

[0105] As can be seen from Table 1, the average hand joint error of the proposed method in the test dataset of the Obman dataset is 8.8 mm, which is better than the three methods that currently perform well on this dataset.

[0106] from Figure 5 It can be seen that in the test dataset of the fused dataset Obman, the method of the present invention can achieve better hand estimation in scenes with different degrees of occlusion of the hand and object, that is, it is closer to the real annotation.

[0107] As can be seen from Table 2, the average hand joint error of the proposed method in the test data set of the HO3D-v2 dataset is 2.08 cm, the average hand joint point error after Prokhorov alignment is 0.88 cm, and the average hand surface point error after Prokhorov alignment is 0.87 cm. All three indicators are better than the three methods that currently perform well on the HO3D-v2 dataset.

[0108] from Figure 6 It can be seen that in the real dataset HO3D-v2, the method of the present invention can achieve better hand estimation in the scenario of real hand-object interaction, and can achieve more reasonable estimation compared with the current better method Handocc.

[0109] The above simulation results show that the present invention can effectively improve the accuracy of hand posture and shape estimation in the case of hand-object occlusion through the method based on the segmentation map guidance and regularization constraint network.

[0110] The above description is only a specific example of the present invention and does not constitute any limitation to the present invention. Obviously, for professionals in this field, after understanding the content and principles of the present invention, it is possible to make various modifications and changes in form and details without departing from the principles and structure of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. A hand pose and shape estimation method based on segmentation map guidance and regularization constraints, characterized by: include: (1) Construct training and test datasets: (1a) Divide the training samples and test samples according to the given partitioning method of the dataset; (1b) Rendering based on the label information E of d training samples to obtain the overall hand segmentation map S w ; (1c) Construct a training dataset D based on the input images, label information, and overall hand segmentation map of d training samples: Among them, I i is the input image of the i-th training sample, E i is the label information of the i-th training sample, is the overall segmentation map of the hand of the i-th training sample; (1d) Construct a test dataset M based on the input images, 3D positions of hand joints, and 3D positions of hand surface points of m test samples: in, is the input image of the jth test sample, is the 3D position of the hand joint point of the j-th test sample, is the 3D position of the hand surface point of the jth test sample; (2) Construct a hand pose and shape estimation network G based on segmentation map guidance and regularization constraints: (2a) Establishing a regularized constraint pyramid module consisting of three regularized constraint modules and an hourglass network in cascade order, which is used to constrain the multi-scale features in the feature extraction backbone to obtain hand structure features; (2b) establishing a segmentation guidance module consisting of three visible segmentation blocks and an overall segmentation module to obtain a visible segmentation map of the hand and an overall segmentation map of the hand; (2c) Select the known feature extraction backbone and regression module, connect the feature extraction backbone and the regularization constraint pyramid module in series, and then connect the outputs of the two to the segmentation guidance module and the regression module in cascade to form a hand posture and shape estimation network G based on segmentation map guidance and regularization constraint: (3) Using a mini-batch gradient descent algorithm, the hand pose and shape estimation network G based on segmentation map guidance and regularization constraints is trained using the training dataset D until the network loss function is in a convergence state, and the trained hand pose and shape estimation network is obtained; (4) Input the test data set M into the trained hand posture and shape estimation network to obtain the estimation results corresponding to the test data.

2. The method according to claim 1, characterized in that The label information E of d training samples in (1b) is expressed as follows: in, is the hand visible segmentation map of the i-th training sample, is the hand joint rotation parameter of the i-th training sample, is the hand shape parameter of the i-th training sample, is the 3D position of the hand joint of the i-th training sample, V i g is the 3D position of the hand surface point of the i-th training sample, is the 2D position of the hand joint of the i-th training sample.

3. The method according to claim 1, characterized in that The feature extraction backbone and regression module selected in step (2c) have the following structure and functions: The feature extraction backbone is composed of a convolutional layer and four residual blocks cascaded in sequence to extract multi-scale features of the image; The regression module is composed of three residual blocks and a hand model regressor cascaded in sequence to obtain the final hand posture and shape estimation results.

4. The method according to claim 3, characterized in that Each residual block in the feature extraction backbone and the regression module includes a cascade of skip residual layers and retained residual layers, and each residual layer is divided into two parallel paths, where: The structure of the first skip residual layer is: first convolution layer → first filter response normalization layer → second convolution layer → second filter response normalization layer; The structure of the second skip residual layer is: convolution layer → filter response normalization layer; The structure of the first residual layer is: first convolution layer → first filter response normalization layer → second convolution layer → second filter response normalization layer; The second path retains the residual layer as its own mapping; Each filter response is activated using the ReLU activation function after the normalization layer.

5. The method according to claim 1, wherein Said (2a) establishes a regular constraint module in the regular constraint pyramid module, which is composed of a downsampling block and an upsampling block in cascade. The downsampling block is composed of three downsampling layers and a hand model regressor in cascade. The upsampling block is composed of three decoding upsampling layers and a linear upsampling layer in cascade: The structure of each downsampling layer is: first skip residual layer → first pooling layer → retained residual layer → second pooling layer → second skip residual layer → third pooling layer; The structure of each decoding upsampling layer is: linear layer → first deconvolution layer → first retained residual layer → second deconvolution layer → second retained residual layer → third deconvolution layer → third retained residual layer; The hand model regressor is composed of three linear layers cascaded with a hand model layer, where the hand model layer is used to map parameters to hand joint points and hand surface points; The linear upsampling layer is composed of a cascade of a linear layer and an upsampling layer.

6. The method according to claim 1, wherein The visible segmentation blocks and the overall segmentation module structure that constitute the segmentation guidance module in step (2b) are as follows: Each visible segmentation module is composed of the first convolution layer → the first filter response normalization layer → the second convolution layer → the second filter response normalization layer in cascade order: The overall segmentation module has a structure of an hourglass network cascaded with two retained residual layers.

7. The method according to claim 1, characterized in that Step (3) uses a mini-batch gradient descent algorithm to train the hand pose and shape estimation network G based on segmentation map guidance and regularization constraints through the training dataset D, as follows: (3a) Randomly select b training samples in batches from the training dataset D The input image I t Input to the feature extraction backbone, after passing through four residual blocks, four multi-scale image features F1 are obtained in turn s ,F2 s ,F3 s , (3b) Input the four multi-scale image features obtained in (3a) into the regularized constrained pyramid module to obtain the hand structure feature F a and the two-dimensional heat map H of the hand joint; (3c) The four multi-scale image features F1 s ,F2 s ,F3 s , Hand structure features F a The hand joint two-dimensional heat map H is input into the segmentation guidance module and the fusion feature F is output. t ; (3d) Fusion feature F t Input to the regression module and output the first-level prediction result (θ 1 ,β 1 ,J 1 ,V 1 ), where θ 1 is the predicted first-level hand joint rotation parameter, β 1 is the predicted first-level hand shape parameter, J 1 is the predicted 3D position of the first-level hand joint point, V 1 is the predicted 3D position of the first-level hand surface point; (3e) Define the loss of the hand pose and shape estimation network G under regularization constraints guided by the segmentation map as: in, is the loss of the hand visible segmentation map, is the loss of the overall hand segmentation map, L h is the loss of the two-dimensional position of the hand joint, L r is the hand parameter regression loss; (3f) Update the network parameters using the gradient descent algorithm based on the loss Loss obtained in (3e); (3g) Repeat (3a) to (3e) until the loss Loss is in a converged state, and obtain the trained hand pose and shape estimation network G based on segmentation map guidance and regularization constraints.

8. The method according to claim 7, characterized in that The hand structure feature F is obtained in step (3b) a The two-dimensional heat map H of the hand joint is implemented as follows: (3b1) The fourth scale image feature Input to the first regular constraint module, and obtain the fourth down-sampled feature after three down-sampling layers in the down-sampling block The fourth downsampling feature The hand model regressor of the downsampling block obtains the fourth-level hand joint rotation parameter θ 4 and hand shape parameter β 4 3D position of hand joint J 4 and the 3D position V of the wrist surface point 4 ; The fourth downsampled feature Input to the upsampling block, output the fourth decoded feature after three decoding upsampling layers Then combine it with the fourth scale image feature After splicing, it is input to the linear upsampling layer, and its output is consistent with the third-scale image feature F3 s Add together to get the third regular constraint feature F3 y ; (3b2) The third regular constraint feature F3 y Input to the second regular constraint module, after three downsampling layers in the downsampling block, the third downsampling feature F3 is obtained x , the third down-sampled feature F3 x The hand model regressor of the downsampling block obtains the third-level hand joint rotation parameter θ 3 and hand shape parameter β 3 3D position of hand joint J 3 and the 3D position V of the wrist surface point 3 ; The third down-sampled feature F3 x Input to the upsampling block, after three decoding upsampling layers, output the third decoding feature F3 u and combine it with the third regular constraint feature F3 y After splicing, it is input to the linear upsampling layer, and its output is consistent with the second scale image feature F2 s Add together to get the second regular constraint feature F2 y ; (3b3) The second regular constraint feature F2 y Input to the third regular constraint module, and after three downsampling layers in the downsampling block, the second downsampling feature F2 is obtained x , the second down-sampled feature F2 x The hand model regressor of the downsampling block obtains the second-level hand joint rotation parameter θ 2 and hand shape parameter β 2 3D position of hand joint J 2 and the 3D position V of the wrist surface point 2 ; The second down-sampled feature F2 x Input to the upsampling block, output the second decoding feature F2 after three decoding upsampling layers u ; The second decoding feature F2 u With the second regular constraint feature F2 y After concatenation, the input is fed into the linear upsampling layer and its output is combined with the first scale image feature F1 s Add together to get the first regular constraint feature F1 y ; (3b3) The first regular constraint feature F1 y After passing through the hourglass network, the output is the hand structure feature F a , for the hand structure feature F a Perform linear transformation to obtain the two-dimensional heat map H of the hand joint.

9. The method according to claim 7, characterized in that Step (3c) combines the four multi-scale image features F1 s ,F2 s ,F3 s , Hand structure features F a The hand joint two-dimensional heat map H is input into the segmentation guidance module and the fusion feature F is output. t , implemented as follows: (3c1) The third scale image feature F3 s and the fourth scale image features after upsampling After splicing, it is input into the first visible segmentation block to obtain the third visible segmentation feature F3 a ; (3c2) The second scale image feature F2 s And the third visible segmentation feature F3 after upsampling a After splicing, it is input into the second visible segmentation block to obtain the second visible segmentation feature F2 a ; (3c3) First scale image feature F1 s and the second visible segmentation feature F2 after upsampling a After splicing, it is input into the third visible segmentation block to obtain the predicted hand visible segmentation map (3c4) The hand structure feature F a and predicted hand visible segmentation map After addition, it is input into the hourglass network of the overall segmentation module, and its output is compared with the predicted hand visible segmentation map Add together to get the overall segmentation feature F w ; (3c5) The overall segmentation feature F w Input to the two residual layers of the overall segmentation module and output the predicted overall segmentation map of the hand (3c6) The hand structure feature F a , predicted visible hand segmentation map Predicted overall hand segmentation map and the hand joint two-dimensional heat map H to obtain the fusion feature F t .

10. The method according to claim 7, characterized in that The loss of the hand visible segmentation map in step (3e) Loss of the overall hand segmentation map The loss of the 2D position of the hand joints L h With the hand regression loss L r , respectively expressed as follows: L r =1.0·ME1+0.6·ME2+0.4·ME3+0.2·ME4 Among them, N represents the number of samples, Y n Represents the predicted hand visible segmentation map of the nth training sample The flattened form, Y n g Represents the hand visible segmentation map label S of the nth training sample v The flattened form of Z n Represents the predicted hand segmentation map of the nth training sample The flattened form of Represents the overall hand segmentation label S of the nth training sample w The flattened form of ; MSE represents the mean square error function, represents the two-dimensional position of the hand joint obtained from the predicted two-dimensional heat map H of the hand joint of the n-th training sample, The label representing the two-dimensional position of the hand joint of the nth training sample; ME k is the loss of the k-th level hand model result, where k is 1, 2, 3, or 4; in, and The 3D positions of hand joint points, 3D positions of hand surface points, hand model rotation parameters and hand model shape parameters output by the k-th level hand model of the n-th sample are respectively. and are the true values ​​of the 3D position of the hand joint point, the 3D position of the hand surface point, the hand model rotation parameters, and the hand model shape parameters of the nth sample.

11. The method according to claim 1, wherein Step (4) uses the test set data M to test the trained hand posture and shape estimation network G based on segmentation map guidance and regularization constraints, as follows: (4a) Take c test samples in batches from the test set M in sequence The input image Input the trained hand posture and shape estimation network G based on segmentation map guidance and regularization constraints to obtain the estimation results corresponding to c test samples in The 3D position of the first-level hand joints output by the network, The 3D position of the first-level hand surface points output by the network; (4b) Repeat (4a) until the test set M is completely traversed and the estimation results corresponding to all test data are obtained.

Images