A 3D voxel hand gesture estimation method based on multi-task information complementarity

Abstract

A three-dimensional voxel hand gesture estimation method based on multi-task information complementarity, the steps are: training a reference point refinement network to obtain accurate reference points; training an estimation network, using the accurate reference points of the image obtained and the true values ​​of the hand joint positions for supervised training: performing probability estimation on the three-dimensional heat map output by the estimation network, determining the maximum likelihood corresponding position of each joint point, and then converting it to the real coordinate system to obtain the real 3D coordinates of the hand joint points. The present invention uses a three-dimensional voxelized method to process input and output. On the one hand, it improves the perspective distortion caused by the traditional method of representing depth images, and the high nonlinearity of regression learning. On the other hand, it realizes model parameter compression, making the model converge more quickly and easy to train. At the same time, a multi-task information complementarity strategy is introduced. This network structure that conforms to the physical characteristics of gestures is more robust while improving the estimation accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of applied activity recognition, and in particular relates to a three-dimensional voxel hand gesture estimation method based on multi-task information complementarity. Background Art

[0002] In the field of applied activity recognition, such as human-computer interaction or augmented reality, accurate 3D hand gesture estimation has always been an important demand. 3D hand gesture estimation has been studied in the computer vision community for decades. Early hand gesture estimation was mostly based on RGB images. However, due to the lack of depth information in RGB images, it is difficult to achieve accurate estimation. In recent years, due to the significant reduction in the cost of depth cameras, hand gesture estimation methods based on depth maps have become mainstream. Early methods based on depth maps mainly used hand-crafted features. This feature acquisition method is time-consuming and labor-intensive, and is not conducive to large-scale promotion and use. Thanks to the emergence of large-scale hand gesture datasets and the improvement of computing power, deep learning methods that can automatically capture features have shown great advantages in hand gesture estimation.

[0003] Most of these deep learning-based 3D hand gesture estimation methods are based on a common idea, that is, directly regressing the 3D coordinates of the key points of the hand after obtaining the 2D depth image. However, this type of method has three disadvantages: The first disadvantage is the perspective distortion problem of the 2D depth image. Most previous methods simply treat the depth map as a 2D image. In fact, the pixel value of the 2D depth map represents the physical distance from the object point to the depth camera, and the depth map is essentially 3D data. The second disadvantage is the nonlinear mapping problem between the input depth data and the output 3D coordinates. This nonlinear mapping problem is caused by the fact that each key point only needs to regress a 3D coordinate from the input. This nonlinear mapping hinders the end-to-end learning process and hinders the accurate estimation of the key point coordinates. The third disadvantage is that the difference in the degrees of freedom and flexibility of the finger joints and palm joints is ignored. From a physical intuitive point of view, the finger joints have higher degrees of freedom and flexibility than the palm joints, while most previous methods directly regard the finger joints and palm joints as a whole hand, and then regress all the joints of the hand as a whole, which will inevitably cause the accuracy of key point estimation to decrease. Therefore, hand gesture estimation requires new data representation methods based on the essential characteristics of depth images, as well as new algorithms that incorporate the real physical properties of gestures.

[0004] Zhirong Wu et al. first proposed a voxel representation method for depth images in the paper "3D Shapenets: Deep Representation of Volumetric Shapes". The main idea is to convert two-dimensional depth image data into three-dimensional voxel data. Voxel is defined as the smallest unit of digital data in three-dimensional space segmentation. To achieve voxelization, each pixel of the depth map must first be reprojected into three-dimensional space, and then the three-dimensional space must be discretized according to the predefined voxel size. In this way, the converted input data can present the appearance of real objects to a greater extent and reduce the perspective distortion problem. Then each voxel is represented as a binary random variable, and a convolutional deep belief network is used to learn the probability distribution of each voxel. Finally, the per-voxel probability of each key point is estimated, which makes this method easier to learn end-to-end than the method of estimating the nonlinear mapping of three-dimensional coordinates directly from the input data.

[0005] In deep learning, multi-task learning can retain more intrinsic information than single-task learning through information complementarity between tasks. And learning multiple tasks at the same time will make the model have better generalization ability. However, in hand posture estimation based on deep learning, multi-task learning has not received enough attention. From a physical intuitive point of view, palm joints are more closely connected than finger joints, but their flexibility and freedom are weaker than finger joints. The overall posture of the hand is mainly determined by the state of the palm joints, while the local posture of the hand is mainly reflected by the state of the finger joints. Therefore, introducing the idea of ​​multi-task information complementarity into hand gesture estimation can fit the real physical characteristics of gestures and improve the robustness of the model and the accuracy of estimation under large-scale data sets. Summary of the invention

[0006] The present invention provides a three-dimensional voxel gesture estimation method based on multi-task information complementation, the purpose of which is to construct a gesture estimation method with high precision, high robustness and high availability. The present invention uses a three-dimensional voxel method to process depth images, which can present the appearance of objects to the greatest extent and reduce the problem of perspective distortion. At the same time, a voxelized output method is used to estimate the likelihood of each voxel of each key point. Compared with the method of directly estimating the three-dimensional coordinates of the key point from the input, the above method is easier to achieve end-to-end learning. In addition, in order to meet the different physical characteristics of palm and finger flexibility and degree of freedom, a multi-task information complementation strategy is introduced, and the gesture estimation is divided into two subtasks of palm regression and finger regression. While the subtask focuses on the regression of this path, the noise generated by the task is transferred to another task by cross-connection. The noise is noise for this task but helps to refine the features of the task for another task, so that the regression result is more accurate. The three-dimensional voxelized input and output processing method combined with the multi-task information complementation strategy can improve the accuracy of gesture estimation while improving the end-to-end learning performance.

[0007] The method of the present invention comprises two modules:

[0008] (1) Train the reference point refinement network to obtain accurate reference points for locating the target object when the depth image is voxelized in three dimensions. The reference point refinement network is divided into two submodules. The first submodule is the data preprocessing part, which mainly rotates the center angle of the depth image to expand the sample, and then obtains the calculated reference point through the depth threshold processing of the hand area; the second submodule is the supervised training of the reference point refinement network. The network structure is modified to 2DRefNet as the reference point refinement network, and the three-dimensional offset from the calculated reference point to the base joint position of the middle finger of the hand after depth threshold processing is used as the label for the supervised network training. Save the trained reference point refinement network, and then apply the network to the gesture depth dataset of the training estimation network to obtain the accurate reference point of each input data and save it for locating the target object when the depth image is voxelized in the subsequent process.

[0009] (2) Train the estimation network, using the three-dimensional voxelized depth image as input data and the true value as the image label to supervise the training of the network. The estimation network is divided into four submodules. The first submodule is the voxelization part of the depth image, which mainly projects the depth image into the three-dimensional space, discretizes the three-dimensional space according to the pre-defined voxel size, and draws a cube frame around the reference point to extract the target object; the second submodule is the feature extraction and refinement module. Based on the different physical characteristics of the palm and fingers in terms of freedom and flexibility, a multi-task hierarchical model modified from CrossInfoNet is adopted to extend it from processing two-dimensional images to processing three-dimensional voxels, and feature extraction is assisted by adopting heat map constraints. The model has two branches, corresponding to the palm joint regression subtask and the finger joint regression subtask respectively. Through the cross connection between the two branches, the noise of one branch becomes the supplementary information of the other branch. The third submodule is the feature enhancement module, which is modified from the encoder-decoder structure in V2VPoseNet to fuse feature maps at different scales. The fourth submodule is the regression module, which mainly fuses the feature maps extracted by the palm regression branch and the finger regression branch to output an estimated three-dimensional heat map.

[0010] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0011] A 3D voxel hand gesture estimation method based on multi-task information complementarity, the specific steps are as follows:

[0012] Step (1) Train a reference point refinement network to obtain accurate reference points

[0013] (1.1) The original depth image of the hand is rotated at the center angle, and then the depth threshold of the hand area is processed to obtain the calculated reference point.

[0014] (1.2) Train the reference point refinement network, use the modified 2DRefNet network structure, and use the three-dimensional offset from the reference point to the base joint position of the middle finger of the hand after depth threshold processing as the label to train the reference point refinement network; the 2DRefNet network structure mainly relies on two-dimensional convolution operations, two-dimensional average pooling operations, and full connection operations. The front end of the 2DRefNet network will be connected to a two-dimensional average pooling operation after each two-dimensional convolution operation; the reason why the maximum pooling is not used here is that it is necessary to obtain the global hand area reference point, and the maximum pooling operation will lose more local features. The back end of the 2DRefNet network is connected to a Dropout after each full connection operation to avoid overfitting of the model due to excessive data generation. Finally, the three-dimensional offset is output through a full connection operation; the mean square error of the three-dimensional offset output by the network and the label is calculated as the loss function to supervise the training of the reference point refinement network. Then use the trained model to refine all the data of the gesture depth dataset to obtain the accurate reference point corresponding to each depth image.

[0015] The reference point refined network loss function is as follows:

[0016]

[0017] L r represents the loss of the reference point refined network, P com represents the three-dimensional offset of the network output, Represents the three-dimensional offset from the calculated reference point to the base joint position of the middle finger of the hand.

[0018] Step (2) trains an estimation network using the accurate reference points of the image obtained in step (1) and the true values ​​of the hand joint positions for supervised training:

[0019] (2.1) Data preprocessing: First, the original depth image is converted into a three-dimensional voxel form; each pixel of the original depth image is reprojected into three-dimensional space, and the three-dimensional space is discretized according to the pre-defined voxel size. Then, a cubic box is drawn around the reference point of the image obtained in step (1) to extract the target object. If the voxel is occupied by any projected depth point, the voxel value of the input V(i,j,k) of the estimation network is 1, otherwise it is 0; then the voxelized depth image is scaled and translated to expand the number of samples.

[0020] (2.2) The voxelized depth image is forward propagated to the feature extraction and refinement module to complete the extraction and refinement of the preliminary features of the input data, paving the way for subsequent feature enhancement. The feature extraction and refinement module is modified from the CrossInfoNet network structure and adopts a modular design. It contains two basic units, namely the volume residual block and the volume basic block, where the volume basic block consists of volume convolution, volume batch normalization, and activation function. The volume residual block is extended from the two-dimensional ResNet structure and also contains the hierarchical operations of the volume basic block. The reason for using volume batch normalization here is that the amount of data generated after the depth image is voxelized is huge, and the three-dimensional convolution operation is quite time-consuming. Therefore, adding batch normalization after each basic unit can speed up the convergence of the model and reduce the time of model training.

[0021] The formula for volume batch normalization is as follows:

[0022]

[0023] y i =γx * +δ (3)

[0024] x i represents the input data before batch processing, μ B , Represent the mean and variance of the batch data respectively, ∈ is a positive number to prevent the divisor from being 0, and γ and δ represent the scale factor and translation factor learned by the network respectively. * is the normalized data, y i is the normalized output data. In order to obtain a smoother output result, the activation function uses the RELU function. The voxelized depth image data is first sent to the front end of the module, and the initial feature extraction is completed after a volume basic block and three volume residual blocks to obtain the feature map T. At the back end of the module, the path is divided into two by introducing feature refinement constraints. The upper branch focuses on finger features, and the lower branch focuses on palm features.

[0025] The formula for feature refinement constraints is as follows:

[0026]

[0027]

[0028] L p ,L f Represent the feature refinement constraint loss of palm and fingers respectively, 3D heatmaps representing the true value joint positions of the palm and fingers, respectively, The three-dimensional heat map of the palm and finger joint positions output by the network, P and F represent the number of palm and finger joint points respectively. The three-dimensional heat map uses a Gaussian function, and its mean is fixed at the corresponding real value joint center position. The feature map T is in L p Under the guidance of , the palm-dominated feature P0 is generated, and then P0 is subtracted from the feature map T through skip connections to obtain the finger feature F - . Similarly, in L f The finger-dominated feature F0 and the palm feature P generated under the guidance of - Then, the noise P0 generated by the upper branch and the palm feature P of the lower branch are cross-connected. - The refined palm feature P1 is obtained by adding them together, and the refined finger feature F1 is obtained similarly.

[0029] (2.3) Forward propagation to the feature enhancement module. To improve the accuracy and robustness of the estimation, the refined finger features F1 and palm features P1 obtained in step (2.2) are enhanced accordingly. A feature enhancement module is added after each of the two branches generated in step (2.2) to fuse feature maps at different scales, and the refined finger features F1 and palm features P1 are enhanced into finger features F2 and palm features P2. The feature enhancement module is modified based on the codec structure in the V2VPoseNet network. The module is mainly composed of basic units such as volume downsampling blocks, volume residual blocks, and volume upsampling blocks. The composition of the volume residual block is the same as that in step (2.2). The volume downsampling block carries a maximum pooling layer based on the volume basic block in step (2.2) to achieve downsampling of the feature map. The volume upsampling block is composed of a volume deconvolution layer, a volume batch normalization, and an activation function. The volume batch normalization and activation function are the same as those in step (2.2), and the volume deconvolution is achieved by the convolution operation of the nearest neighbor interpolation. The entire feature enhancement module is mainly divided into the front-end encoder part and the back-end decoder part. The encoder reduces the size of the feature map to 1 / 4 of the original size through two volume downsampling blocks, and the decoder restores the size of the feature map to the original size through two volume upsampling blocks. At the same time, the input of the first volume downsampling block of the encoder is added to the output of the second volume upsampling block of the decoder through two sets of jump connections. The input of the second volume downsampling block of the encoder is added to the output of the first upsampling block of the decoder. This structure can fuse feature maps at different scales. The reason for adopting such a structure is that posture estimation is an associative task. Different joints of the hand do not have the best recognition accuracy on the same feature map. If only feature maps of the same scale are used for the final estimation, it will inevitably cause a loss of accuracy.

[0030] (2.4) The enhanced finger features and palm features are forward propagated to the regression module, the palm regression sub-branch and the finger regression sub-branch are fused and the estimated 3D heat map is output through a volume residual block plus a volume basic block. The 3D coordinates are extracted from the output 3D heat map. The mean square error of the 3D heat map of the true value joint position and the 3D heat map output by the network is used as the voxel regression loss, and the mean square error of the true value joint position and the extracted 3D coordinates is used as the joint regression loss. A Gaussian function is used to generate the 3D heat map, where the mean of the Gaussian function is fixed at the true value joint center position.

[0031] The formula for the 3D heat map is as follows:

[0032]

[0033] Represents the three-dimensional heat map of the true value of the nth key point, i n ,j n ,k n Represents the true value voxel coordinate of the nth key point. σ 2 Represents the variance of the Gaussian function.

[0034] The voxel regression loss formula is as follows:

[0035]

[0036] L t represents the voxel regression loss of the estimation network, Represents the three-dimensional heat map of the true value of the nth key point, H n (i,j,k) represents the 3D heat map of the nth joint estimated by the network. N represents the total number of hand joints.

[0037] The joint regression loss formula is as follows:

[0038]

[0039] L g represents the joint regression loss of the estimated network, N represents the total number of hand joints, Represents the true value three-dimensional coordinates of the nth joint point, Represents the three-dimensional coordinates of the nth joint point estimated by the network.

[0040] By weighted fusion of the feature refinement constraint loss, voxel regression loss and joint regression loss in step (2.2), the total loss function of the estimated network can be obtained as follows:

[0041] L = α(L p +L f )+βL t +γLg (9)

[0042] L represents the estimated total network loss, L p ,L f denote the feature refinement constraint loss of the palm and fingers respectively, L t represents the estimated network voxel regression loss, L g represents the estimated network joint regression loss, and α, β, γ are the weights for balancing feature refinement loss, voxel regression loss, and joint regression loss.

[0043] Step (3) performs probability estimation on the three-dimensional heat map output by the estimation network in step (2), determines the maximum likelihood corresponding position of each joint point, and then transforms it into the real coordinate system to obtain the real 3D coordinates of the hand joint points.

[0044] Beneficial effects of the present invention: The present invention uses a three-dimensional voxelization method to process input and output. On the one hand, it improves the perspective distortion caused by the traditional method of representing depth images, as well as the high nonlinearity of regression learning. On the other hand, it realizes model parameter compression, making the model converge more quickly and easier to train. At the same time, a multi-task information complementation strategy is introduced. This network structure that conforms to the physical characteristics of gestures improves the estimation accuracy while also being more robust. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the overall framework of the present invention.

[0046] Figure 2 It is a schematic diagram of the reference point refining network structure of the present invention.

[0047] Figure 3 It is a schematic diagram of the structure of the feature extraction and refinement module of the present invention.

[0048] Figure 4 It is a schematic diagram of the structure of the feature enhancement module of the present invention.

[0049] Figure 5 Schematic diagram of the regression module structure of the present invention.

[0050] Figure 6 This is a flow chart of the reference point refining network training method of the present invention.

[0051] Figure 7 This is a flow chart of the estimation network training method of the present invention. DETAILED DESCRIPTION

[0052] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0053] The present invention can be used for three-dimensional hand gesture estimation in various forms. The overall framework of the present invention is as follows: Figure 1 As shown, it is subdivided into each network model, and the reference point refined network structure is as follows Figure 2 As shown in the figure, the feature extraction and refinement module structure is as follows Figure 3 As shown, the feature enhancement module structure is as follows Figure 4 As shown, the regression module structure is as follows Figure 5 shown.

[0054] This embodiment is applied to the task of 3D hand gesture estimation in a somatosensory game based on Microsoft Kinect. The specific implementation scheme discussed is only used to illustrate the implementation of the present invention, but does not limit the scope of the present invention.

[0055] The following mainly focuses on the task of 3D hand gesture estimation in a somatosensory game based on Microsoft Kinect to explain the implementation scheme of the present invention in detail. The task is mainly divided into the training of the reference point refinement network (such as Figure 5 ) and the training of the estimation network (as shown in Figure 6 shown):

[0056] (1) Training of reference point refinement network:

[0057] According to the preprocessing steps in the invention content, the original image is rotated at a random angle in the interval [-50,50] to expand the number of data set samples, the data is subjected to depth threshold processing to obtain the calculated reference points of the hand area, and then the data is input into the reference point refinement network. The input image size is 320×240×32, and the last 32 represents the number of channels of the network. When the image is forward propagated, it passes through convolution, maximum pooling, full connection, and Dropout regularization layers to obtain the output three-dimensional offset. The mean square error of the output three-dimensional offset and the three-dimensional offset from the calculated reference point to the root joint position of the middle finger of the hand is used as the loss function, and the RMSProp optimization algorithm is used for training to achieve end-to-end training, and the training is carried out until the model converges. After that, the trained model is used to process the depth image of the data set to obtain the three-dimensional offset between the calculated reference point and the root joint position of the middle finger of the hand, and finally the offset is added to the calculated reference point to obtain the accurate reference point of each depth image and save it.

[0058] (2) Training of estimation network:

[0059] The original depth image is voxelized in three dimensions according to the voxelization step in the content of the invention, and each pixel of the depth image is reprojected into three-dimensional space. The three-dimensional space is discretized according to the predefined voxel size, and then a cubic frame is drawn around the reference point obtained by the reference point refinement network to extract the target object. The network input value of the voxels occupied by the projected depth point is set to 1, otherwise it is set to 0. The voxelized data is translated to any integer value in the range of [-6,6] to expand the number of data samples of the input network. Then the processed three-dimensional voxelized data is sent to the estimation network for forward propagation. The input data dimension is 96×96×96×64, and the last 64 represents the number of channels. The input data is first forward propagated to the feature extraction and refinement module, the front end of which is a volume basic block and three volume residual blocks. The size of their kernel function is 8×8×8×64. The data is extracted to the initial feature map T through the front end of the module, and then T is sent to the two branches respectively. The upper finger regression branch first refines the palm loss L p Under the guidance of , a volume basic block is used to generate the palm feature P0, and P0 is subtracted from the feature map T through the jump connection to obtain F - The lower palm regression branch has a finger thinning loss L f Under the guidance of , a volume basic block is used to generate the finger feature F0, and F0 is subtracted from the feature map T through the jump connection to obtain P - . P0 and P - , F0 and F - The refined palm feature P1 and finger feature F1 are added together. Then, the refined features are forward propagated to their respective feature enhancement modules. After up-sampling and down-sampling of the codec, the features at different scales are fused to obtain the enhanced palm feature P2 and finger feature F2. Finally, they are forward propagated to the regression module. The regression module merges P2 and F2 to obtain the feature map M, and obtains the output heat map through a volume residual block and a volume basic block. The estimated three-dimensional coordinates of the joints are extracted from the heat map, and the mean square error between the true value three-dimensional heat map and the network estimated three-dimensional heat map is calculated as the voxel regression loss. The true value joint three-dimensional coordinates and the network estimated three-dimensional coordinates of the joints are calculated as the joint regression loss. The weighted feature refinement loss, voxel regression loss and joint regression loss are used as the total loss, and the weights are updated by back propagation. The weight update uses the RMSProp optimization algorithm until the model converges.

[0060] (3) Output of gesture posture

[0061] Obtain the 3D coordinates of the hand according to the hand joint acquisition steps, perform probability estimation on the three-dimensional heat map output by the estimation network, obtain the maximum likelihood response position of each hand joint, and then convert it to the real coordinate system to obtain the real 3D coordinates of each hand joint.

Claims

1. A 3D voxel hand gesture estimation method based on multi-task information complementarity, characterized in that: The specific steps are as follows: Step (1) Train a reference point refinement network to obtain accurate reference points (1.1) The original depth image of the hand is rotated by the angle of the center, and then the depth threshold processing of the hand area is performed to obtain the calculation reference point; (1.2) Train the reference point refinement network, use the modified 2DRefNet network structure, and use the three-dimensional offset from the reference point to the base joint position of the middle finger of the hand after depth threshold processing as the label to train the reference point refinement network; each two-dimensional convolution operation at the front end of the 2DRefNet network is connected to a two-dimensional average pooling operation; each full connection operation at the back end of the 2DRefNet network is connected to a Dropout to avoid overfitting of the model due to excessive data generated; finally, the three-dimensional offset is output through a full connection operation; the mean square error of the three-dimensional offset output by the network and the label is calculated as the loss function to supervise the training of the reference point refinement network; then use the trained model to refine all the data of the gesture depth dataset to obtain the accurate reference point corresponding to each depth image; The reference point refined network loss function is as follows: L r represents the loss of the reference point refined network, P com represents the three-dimensional offset of the network output, Indicates the three-dimensional offset from the calculated reference point to the base joint position of the middle finger of the hand; Step (2) trains an estimation network using the accurate reference points of the image obtained in step (1) and the true values ​​of the hand joint positions for supervised training: (2.1) Data preprocessing: First, the original depth image is converted into a three-dimensional voxel form; each pixel of the original depth image is reprojected into a three-dimensional space, and the three-dimensional space is discretized according to the predefined voxel size. Then, a cubic box is drawn around the reference point of the image obtained in step (1) to extract the target object. If the voxel is occupied by any projected depth point, the voxel value of the input V(i,j,k) of the estimation network is 1, otherwise it is 0; then the voxelized depth image is scaled and translated to expand the number of samples; (2.2) The voxelized depth image is forward propagated to the feature extraction and refinement module to complete the extraction and refinement of the preliminary features of the input data, paving the way for subsequent feature enhancement. The feature extraction and refinement module is modified from the CrossInfoNet network structure and adopts a modular design. It contains two basic units, namely the volume residual block and the volume basic block. The volume basic block consists of volume convolution, volume batch normalization, and activation function. The volume residual block is extended from the two-dimensional ResNet structure and also contains the hierarchical operation of the volume basic block. The formula for volume batch normalization is as follows: y i =γx * +d (3) x i represents the input data before batch processing, μ B , Represent the mean and variance of the batch data respectively, ε is a positive number to prevent the divisor from being 0, γ and δ represent the scale factor and translation factor learned by the network respectively; x * is the normalized data, y i is the normalized output data; in order to obtain a smoother output result, the activation function uses the RELU function. The voxelized depth image data is first sent to the front end of the module, and the initial feature extraction is completed after a volume basic block and three volume residual blocks to obtain the feature map T; at the back end of the module, the path is divided into two by introducing feature refinement constraints, the upper branch focuses on finger features, and the lower branch focuses on palm features; The formula for feature refinement constraints is as follows: L p ,L f Represent the feature refinement constraint loss of palm and fingers respectively, 3D heatmaps representing the true value joint positions of the palm and fingers, respectively, Represents the three-dimensional heat map of the palm and finger joint positions output by the network, P and F represent the number of palm and finger joint points respectively; the three-dimensional heat map uses a Gaussian function, and its mean is fixed at the corresponding true value joint center position; the feature map T is in L p Under the guidance of , the palm-dominated feature P0 is generated, and then P0 is subtracted from the feature map T through skip connections to obtain the finger feature F-; Similarly, in L f The finger-dominated feature F0 and the palm feature P- are generated under the guidance of , and then the noise P0 generated by the upper branch and the palm feature P- of the lower branch are added by cross connection to obtain the refined palm feature P1. Similarly, the refined finger feature F1 can be obtained; (2.3) Forward propagation to the feature enhancement module. In order to improve the accuracy and robustness of the estimation, the refined finger feature F1 and palm feature P1 obtained in step (2.2) are enhanced accordingly. A feature enhancement module is added after each of the two branches generated in step (2.2) to fuse feature maps at different scales, and the refined finger feature F1 and palm feature P1 are enhanced into finger feature F2 and palm feature P2. The feature enhancement module is modified based on the codec structure in the V2VPoseNet network. The module is mainly composed of a volume downsampling block, a volume residual block, and a volume upsampling block. The composition of the volume residual block is consistent with that in step (2.2). The volume downsampling block carries a maximum pooling layer on the basis of the volume basic block in step (2.2) to achieve downsampling of the feature map. The volume upsampling block is composed of a volume deconvolution layer, a volume batch normalization, and an activation function. The volume batch normalization and activation function are consistent with those in step (2.2), and the volume deconvolution achieves upsampling of the feature map through a convolution operation of nearest neighbor interpolation. The entire feature enhancement module is mainly divided into the front-end encoder part and the back-end decoder part; the encoder reduces the feature map size to 1 / 4 of the original through two volume downsampling blocks, and the decoder restores the feature map size to the original size through two volume upsampling blocks. At the same time, the input of the first volume downsampling block of the encoder is added to the output of the second volume upsampling block of the decoder through two sets of jump connections, and the input of the second volume downsampling block of the encoder is added to the output of the first upsampling block of the decoder. This structure can fuse feature maps at different scales. (2.4) The enhanced finger features and palm features are forward propagated to the regression module, the palm regression sub-branch and the finger regression sub-branch are fused and the estimated 3D heat map is output through a volume residual block plus a volume basic block; the 3D coordinates are extracted from the output 3D heat map, and the mean square error of the 3D heat map of the true value joint position and the 3D heat map output by the network is used as the voxel regression loss, and the mean square error of the true value joint position and the extracted 3D coordinates is used as the joint regression loss; a Gaussian function is used to generate the 3D heat map, where the mean of the Gaussian function is fixed at the center position of the true value joint; The formula for the 3D heat map is as follows: Represents the three-dimensional heat map of the true value of the nth key point, i n ,j n ,k n Represents the true value voxel coordinate of the nth key point; σ 2 represents the variance of the Gaussian function; The voxel regression loss formula is as follows: L t represents the voxel regression loss of the estimation network, Represents the three-dimensional heat map of the true value of the nth key point, H n (i, j, k) represents the three-dimensional heat map of the nth joint point estimated by the network; N represents the total number of hand joint points; The joint regression loss formula is as follows: L g represents the joint regression loss of the estimated network, N represents the total number of hand joints, Represents the true value three-dimensional coordinates of the nth joint point, Represents the three-dimensional coordinates of the nth joint point estimated by the network; By weighted fusion of the feature refinement constraint loss, voxel regression loss and joint regression loss in step (2.2), the total loss function of the estimated network can be obtained as follows: L=α(L p +L f )+βL t +ζL g (9) L represents the estimated total network loss, L p ,L f denote the feature refinement constraint loss of the palm and fingers respectively, L t represents the estimated network voxel regression loss, L g represents the estimated network joint regression loss, α, β, ζ are the weights for balancing feature refinement loss, voxel regression loss and joint regression loss; Step (3) performs probability estimation on the three-dimensional heat map output by the estimation network in step (2), determines the maximum likelihood corresponding position of each joint point, and then transforms it into the real coordinate system to obtain the real 3D coordinates of the hand joint points.

Images