Method for constructing a 3D hand posture model and method for estimating a 3D hand posture

Abstract

The present invention discloses a method for constructing a three-dimensional hand posture model and a three-dimensional hand posture estimation method, which belongs to the field of three-dimensional posture estimation technology. The method comprises the following steps: cropping an RGB image carrying interactive hand information, setting evenly distributed three-dimensional anchor points on the cropped image, and obtaining three-dimensional coordinate information of each three-dimensional anchor point; inputting the image after setting the three-dimensional anchor points into a feature enhancement model to obtain three-dimensional offsets and weights of each three-dimensional anchor point to all corresponding estimated joint points; fusing the three-dimensional offsets and weights with the three-dimensional coordinate information to obtain the three-dimensional coordinates of the hand joint points, and then determining a loss function; and using the loss function to train a three-dimensional hand posture estimation network to obtain a target three-dimensional hand posture model. The present invention constructs interactivity between local three-dimensional anchor points, giving them global spatial and contextual perception, thereby better capturing the joint information of the joint points to be estimated and reducing the self-occlusion and mutual occlusion problems of the interactive hands.

Description

Technical Field

[0001] The present invention belongs to the technical field of image posture estimation, and more specifically, relates to a method for constructing a three-dimensional hand posture model and a three-dimensional hand posture estimation method. Background Art

[0002] There are many 3D hand pose estimation methods. The current 3D hand pose estimation methods can be mainly divided into the following two types.

[0003] The first method is to estimate three-dimensional hand pose based on depth images: a commonly used method is to extract features from the depth map and then use regression to obtain the coordinate position information of each joint point. Using regression methods, a neural network is used to regress and predict the three-dimensional information of the hand. Using the structure of encoding and decoding layers, a probability density function, called a heat map, is generated for each joint point of the hand. The predicted joint point coordinate position is obtained from the position with the highest probability in the heat map. Alternatively, a graph convolutional neural network is used to extract information from the image. A two-dimensional convolutional neural network is used to obtain three branches on the original depth map: the weight of each three-dimensional anchor point, the planar offset of each three-dimensional anchor point relative to other joint points, and the depth offset. The result of the three-dimensional pose estimation of the human body is obtained by weighted summation of each three-dimensional anchor point.

[0004] The second type is three-dimensional hand pose estimation based on RGB images. Since RGB images lack three-dimensional information, it is very challenging to predict the three-dimensional joint positions of the hand through RGB images. Using network structures such as PoseNet, the hand area is segmented from the image, the pose is estimated, and it is fitted or mapped into three-dimensional space to infer its position. To solve the problem of hand occlusion, a multi-view method is used for hand pose estimation. It is effective in most viewpoints, but it is not very effective when the hand is severely occluded. Although current methods can improve the problems of occlusion and perspective changes in RGB images, single hands also have their limitations. It will be difficult to identify multiple hand targets in the image, or when the hand interacts with other objects.

[0005] However, the task of interactive hand pose estimation from RGB images still faces many difficulties. These difficulties can be roughly divided into two aspects: First, occlusion poses many challenges to interactive hand prediction in hand interaction tasks; second, the characteristics of the corresponding joints of the left and right hands are very similar, making it very difficult to distinguish the characteristics of each hand joint. Currently proposed methods have difficulty effectively solving problems such as poor prediction results in the presence of occlusion, slow operation speed, and inability to effectively distinguish hand joints. Summary of the Invention

[0006] In response to the above-mentioned defects or improvement needs of the prior art, the present invention provides a method for constructing a three-dimensional hand posture model and a method for estimating a three-dimensional hand posture. The purpose is to construct the interactivity between local three-dimensional anchor points so that they have global space and context perception, so as to better capture the joint information of the joint points to be estimated, thereby reducing the self-occlusion and mutual occlusion problems that often occur in interactive hands, thereby solving the technical problem of low accuracy of three-dimensional hand posture estimation due to occlusion in the prior art.

[0007] To achieve the above objectives, according to one aspect of the present invention, a method for constructing a three-dimensional hand posture model is provided, comprising:

[0008] S1: Cropping the RGB image carrying the interactive hand information to obtain a first hand image;

[0009] S2: Setting a plurality of evenly distributed three-dimensional anchor points on the first hand image to obtain a second hand image, and obtaining three-dimensional coordinate information of each of the three-dimensional anchor points in space;

[0010] S3: Inputting the second hand image into a feature enhancement model to perform feature extraction and feature enhancement, and obtaining three-dimensional offsets and weights of each of the three-dimensional anchor points to all corresponding estimated joint points;

[0011] S4: fusing the three-dimensional offsets of each of the three-dimensional anchor points to the corresponding estimated joint points, their weights, and the three-dimensional coordinate information to obtain the three-dimensional coordinates of the hand joint points corresponding to each of the three-dimensional anchor points;

[0012] S5: determining a loss function according to the three-dimensional coordinates of each of the hand joint points;

[0013] S6: Inputting the training samples into the 3D hand posture estimation network to be trained for training, and adjusting the network parameters using the loss function during the training process, and finally obtaining a trained 3D hand posture model.

[0014] In one embodiment, the S2 includes:

[0015] Evenly divide the first hand image into N image blocks, set a three-dimensional anchor point at the center of each image block and obtain its plane coordinates;

[0016] Taking the depth value of the root node coordinate of the hand in each of the image blocks as the center, a depth value is set before and after the depth value of the center at a preset depth in the world coordinate system to obtain three depth coordinates of the three-dimensional anchor point;

[0017] The plane coordinates of the N three-dimensional anchor points and the corresponding three depth coordinates are fused to obtain 3N three-dimensional coordinate information corresponding to the N three-dimensional anchor points.

[0018] In one embodiment, the S4 includes:

[0019] The three-dimensional coordinates of the estimated joint point corresponding to the three-dimensional anchor point a are expressed as: plane coordinates and depth coordinate position Among them, a∈A, and Respectively represent the plane coordinates and depth coordinate positions of the three-dimensional anchor point a; and They represent the plane coordinate offset and depth coordinate offset from the 3D anchor point a to the corresponding estimated joint point j respectively; The normalized weight of the three-dimensional anchor point a to the corresponding estimated joint point j represents its contribution value.

[0020] In one embodiment, W j (a) represents the weight from the 3D anchor point a to the corresponding estimated joint point j.

[0021] In one embodiment, the feature enhancement model includes:

[0022] a multi-scale feature extractor for extracting features from the second hand image using a ResNet-50 network structure, selecting output features from the last three layers of the network structure as a multi-scale feature pyramid, fusing these features to obtain first image features, processing the first image features into features of the same planar size using convolutional layers and group normalization layers, and then flattening and concatenating these features to obtain second image features;

[0023] a feature enhancement module, configured to input the second image feature and the spatial position code of the three-dimensional anchor point into multiple coding layers for enhancement to obtain a third image feature;

[0024] a 3D anchor feature interaction module, configured to extract 3D anchor features from the second hand image using a 3D anchor estimator to obtain a fourth image feature, input the fourth image feature and the third image feature into multiple decoding layers for decoding, and thereby output a fifth image feature;

[0025] An offset-weight prediction module is configured to receive the fifth image feature using two multi-layer perceptrons (MLPs), and one MLP predicts a coordinate offset O. j (a), Another MLP predicts the weight W j (a).

[0026] In one embodiment, each encoding layer includes a self-attention structure and a cross-attention structure, and the weights of the MLP in all encoding layers are shared;

[0027] For the self-attention structure, the input value is: Q = D + P q , K=D+P q , V=D; D represents the fifth image feature, P q The spatial position encoding of the 3D anchor point is expressed as P q =MLP(PE(a q ));

[0028] For the cross attention structure, the input value is: Q = D + P q , K=a q , V=E, K represents the reference point in the deformable attention model is selected as the coordinate position of q three-dimensional anchor points, and E represents the third image feature;

[0029] All coding layers extract the third image features corresponding to each 3D anchor point and establish the interaction between 3D anchor points.

[0030] In one embodiment, the loss function includes a first loss function and a second loss function; and S5 includes:

[0031] Generating the first loss function according to the three-dimensional coordinates and true value information of each of the hand joints as a first supervisory signal to supervise the obtained three-dimensional hand posture estimation result;

[0032] The second loss function is generated according to the three-dimensional coordinates of each of the hand joint points and the weight corresponding to each three-dimensional anchor point, and is used as a second supervisory signal to supervise the assigned three-dimensional anchor point weights.

[0033] In one embodiment, the first loss function is:

[0034] and Represent the true value and predicted value of the plane coordinates of the j-th joint point, respectively. and Represent the true value and predicted value of the depth coordinate of the j-th joint point respectively; α represents the given weight, L τ (·) loss represents smooth L1 loss;

[0035] The second loss function is: τ1 and τ2 are given as 1 and 3 respectively to calculate the smooth depth loss value;

[0036] The loss function is: loss=λ1loss1+λ2loss2, where λ1 and λ2 are given hyperparameters.

[0037] According to another aspect of the present invention, a three-dimensional hand posture estimation method is provided, comprising: inputting an image to be identified into the trained three-dimensional hand posture model to obtain a three-dimensional hand posture estimation result of the target object; wherein the image to be identified carries hand interaction information of the target object.

[0038] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described above are implemented.

[0039] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0040] (1) The present invention proposes a method for constructing a three-dimensional hand posture model. By constructing the interactivity between local three-dimensional anchor points, the model has global space and context perception, so as to better capture the joint information of the joint points to be estimated, thereby reducing the self-occlusion and mutual occlusion problems that often occur in interactive hands; at the same time, since each three-dimensional anchor point is used to estimate the three-dimensional offset of all the joint points to be estimated, the model can more accurately learn which three-dimensional anchor points can make a greater contribution to the joint points to be estimated through weights, thereby assigning larger weights; finally, the set three-dimensional anchor points are in three-dimensional space, which has a more accurate estimation effect on the prediction of depth values ​​in hand posture estimation.

[0041] (2) The present invention solves the problems of poor accuracy and slow model running speed in current hand posture estimation methods, and effectively solves the problems of strong occlusion and strong feature similarity in interactive hand postures. After training on a large-scale unlabeled dataset, it has the beneficial effects of strong robustness and good generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The figure is a flow chart of a method for constructing a three-dimensional hand posture model provided by one embodiment of the present invention.

[0043] Figure 2 It is a schematic diagram of the principle of a method for constructing a three-dimensional hand posture model provided by an embodiment of the present invention.

[0044] Figure 3 This is an information flow diagram of the method provided by one embodiment of the present invention during the training and testing phases.

[0045] Figure 4 This is the setting of the coding layer proposed in the method provided in one embodiment of the present invention.

[0046] Figure 5 This is the setting of the decoding layer proposed in the method provided in one embodiment of the present invention.

[0047] Figure 6 It is a schematic diagram of the process of setting three-dimensional anchor points and predicting joint point coordinates through three-dimensional anchor points proposed in the method provided in one embodiment of the present invention.

[0048] Figure 7 This is a schematic diagram of the 3D posture estimation results of the method provided by one embodiment of the present invention under different interactive hand image inputs and the weight sizes of 3D anchor points at different depth layers.

[0049] Figure 8 This is a schematic diagram of the weights of three-dimensional anchor points at different depth layers for different joint points according to the method provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0051] Example 1

[0052] like Figure 1 As shown, the present invention provides a method for constructing a three-dimensional hand posture model, comprising:

[0053] S1: Cropping the RGB image carrying the interactive hand information to obtain a first hand image;

[0054] S2: setting a plurality of evenly distributed three-dimensional anchor points on the first hand image to obtain a second hand image, and obtaining three-dimensional coordinate information of each three-dimensional anchor point in space;

[0055] S3: Input the second hand image into the feature enhancement model for feature extraction and feature enhancement, and obtain the 3D offsets and weights of each 3D anchor point to all corresponding estimated joint points;

[0056] S4: Fusing the 3D offsets of each 3D anchor point to the corresponding estimated joint point, its weight, and 3D coordinate information to obtain the 3D coordinates of the hand joint point corresponding to each 3D anchor point;

[0057] S5: Determine the loss function based on the three-dimensional coordinates of each hand joint;

[0058] S6: Input the training samples into the 3D hand posture estimation network to be trained for training. During the training process, the network parameters are adjusted using the loss function to finally obtain a trained 3D hand posture model.

[0059] Figure 2It is a schematic diagram of the principle of the method for constructing a three-dimensional hand posture model provided by an embodiment of the present invention. The three-dimensional hand posture estimation task is to predict the three-dimensional spatial position information of the hand joints in a given depth image or RGB image. Unlike the single-hand posture estimation task, the interactive hand posture estimation task needs to simultaneously predict the positions of the joints of both hands in three-dimensional space. In view of the problems existing in the three-dimensional hand posture estimation task based on interactive hands, such as similar hand joint features, serious interactions between hands, and self-occlusion, the present invention proposes a method for constructing a three-dimensional hand posture model, which adopts the method of setting local three-dimensional anchor points to accurately estimate the details of the hand joints through the local three-dimensional anchor points. At the same time, the attention mechanism is used to enable the model to capture the link relationship between the global hand joints in the image, collect hand semantic information, and complete the three-dimensional posture estimation of the interactive hand.

[0060] In this embodiment, unlike the modeling method that uses the hand parameter model for model fitting, this algorithm does not need to use any prior knowledge of the hand parameter model. Only one RGB image is needed to complete the neural network verification of the three-dimensional hand joint points. The information flow diagram in the training phase and the test phase is shown in the figure. Figure 3 For the task of 3D hand pose estimation, this algorithm uses local 3D anchor points, treating each 3D anchor point as a local regressor to regress the 3D offsets of all hand joints. These offsets are then weighted through weight fusion to obtain the 3D coordinates of all hand joints.

[0061] Example 2

[0062] S2 sets uniformly and densely distributed 3D anchor points in the image and obtains their 3D coordinate information in space, including:

[0063] For a uniformly and densely distributed 3D anchor point system, a 256×256 input image is divided into 256 blocks with a step size of 16. A 3D anchor point is set at the center of each block, generating a total of 256 3D anchor point coordinates. Three depth values ​​are set in the world coordinate system, centered at the hand's root node depth value, 10 cm before and after the center depth value. These 3D anchor point coordinates are then combined to produce a total of 256×3 3D anchor points, evenly distributed across the image and at two locations in front and behind the image. The resulting 3D anchor point coordinates represent their plane coordinates in the image coordinate system and their depth coordinates in the world coordinate system. These 3D anchor point coordinates are saved during 3D anchor point generation for later use in joint point coordinate prediction.

[0064] Example 3

[0065] For each 3D anchor point a, its plane coordinate position and depth coordinate position can be expressed as and The plane coordinate offset and depth coordinate offset of the three-dimensional anchor point to the predicted joint point are obtained as above and are expressed as and The normalized weight of the three-dimensional anchor point to the predicted joint point represents its contribution value, which is expressed as From this, the plane coordinate position of the predicted joint point can be obtained and depth coordinate position

[0066]

[0067] Example 4

[0068] For the weight W from the 3D anchor point a to the joint point j j (a), normalize it and use Soft-max for weighting. The formula is: in, W j (a) Weights after weighting.

[0069] Example 5

[0070] S3 includes:

[0071] A1. Input the RGB image into the feature enhancement model to extract and enhance the image features.

[0072] A2. Get the weight and offset of the 3D anchor point.

[0073] A3. The coordinate positions of the three-dimensional anchor points are integrated to obtain the three-dimensional coordinates of the hand joint points.

[0074] like Figure 4 So and Figure 5 As shown in the figure, A1 first performs multi-scale feature extraction on the cropped RGB image (the second hand image), extracts image features through the ResNet-50 network structure, selects the output features of the last three layers of the network structure as the multi-scale feature pyramid, and fuses the multi-layer features to obtain the input features. The Encoder structure and Decoder structure are then used to further extract and enhance the image features. Next, A2 obtains the predicted weights and offsets through two multi-layer perceptron (MLP) structures. Figure 6 As shown in FIG, A3 obtains the three-dimensional coordinates of the hand joint points by fusing the position coordinates of the three-dimensional anchor points.

[0075] Specifically, for the input fusion feature pyramid (first image feature), the convolution layer and the Group-Normalization layer are used to process them into features of the same plane size. Then they are flattened and concatenated to obtain the second image feature. The generated second image feature will be added to the position code to obtain the third image feature. The feature encoding formula is: P xy =PE(x,y), where the PE operation encodes the position information in the feature through sine transform, and x and y represent the coordinate positions in the feature map.

[0076] Input the third image feature into the multi-scale Deformable Attention model, the formula is:

[0077] Q=F+P xy ,K=ref(F),V=F,

[0078] Among them, F represents the input image feature, and the ref operation represents the selection of the reference point.

[0079] Through such a multi-scale Deformable Attention model, the features of the image are enhanced. A total of 6 layers of encoder structure are selected here.

[0080] For the enhanced image features, the three-dimensional anchor point features in the image are extracted by a three-dimensional anchor point estimator to obtain the fourth image features, and at the same time, feature fusion is performed on the three-dimensional anchor points set above.

[0081] The 3D anchor point features in the image are extracted by the 3D anchor point estimator, including defining the 3D anchor point coordinates as: a q =(x q ,y q ,d q ); where a q represents the qth three-dimensional anchor point, x q ,y q ,d q Represent the plane coordinate value and depth value of the 3D anchor point respectively. For each 3D anchor point, their spatial position encoding features are expressed as: P q =MLP(PE(a q )).

[0082] For the features output by the Decoder structure, they are input into two MLP structures respectively, one MLP structure is used to predict the weight W j (a) represents the weight from the a-th 3D anchor point to the j-th joint point; another MLP structure is used to predict the coordinate offset O j(a) represents the offset from the a-th 3D anchor point to the j-th joint point. j (a) Split into and They respectively represent the plane offset and depth offset from the a-th 3D anchor point to the j-th joint point.

[0083] Example 6

[0084] MLP stands for Multi-Layer Perceptron. In the 6-layer decoding layer structure, each decoder layer contains such a structure, and the weights of MLP in all 6-layer decoding layer decoders are shared.

[0085] Each layer of decoder contains a self-attention structure and a cross-attention structure. For the self-attention structure, the value input to the multi-scale Deformable Attention model is: Q = D + P q ,K=D+P q ,V=D, where D represents the encoding of the decoder output feature. For the cross-attention structure, the value input to the multi-scale Deformable Attention model is: Q=D+P q ,K=a q V = E, where K represents the coordinate positions of q 3D anchor points selected as reference points in the DeformableAttention model, and E represents the output of the encoder structure. Ultimately, after passing through the six-layer decoder structure, the model extracts the features corresponding to each 3D anchor point and establishes interactions between 3D anchor points, thereby establishing relationships between 3D anchor point features.

[0086] Example 7

[0087] During the training phase, a mainstream interactive hand RGB dataset and a given cropping box are used to obtain a cropped RGB image. Large-scale public datasets, including the InterHand2.6M dataset, as well as the depth datasets NYU and HANDS2017, are used to verify the generalization of the model. The cropped images containing interactive hand information are input as training samples into the 3D hand pose estimation network to be trained to obtain the 3D hand pose estimation results. Based on the coordinates of the generated 3D hand joint points and the GT information, a first loss function is generated as the first supervisory signal to supervise the generation of the obtained 3D hand pose. At the same time, based on the set 3D anchor point coordinates and the predicted weights for each 3D anchor point, a second loss function is generated as the second supervisory signal to supervise the assigned 3D anchor point weights.

[0088] During the testing phase, the input interactive hand image also needs to be cropped and the cropped image is input into the 3D pose estimation network based on 3D anchor points to obtain the 3D hand pose estimation result.

[0089] Example 8

[0090] The first and second loss functions, as well as the total loss function, are determined according to the three-dimensional coordinates of the hand joints, including: the first loss function is:

[0091] in, and Represent the GT value and predicted value of the plane coordinates of the j-th joint point, respectively. and They represent the GT value and predicted value of the depth coordinate of the jth joint point respectively. α represents the given weight parameter, and the given value is 0.5. τ (·) loss represents the smooth L1 loss, which is expressed as:

[0092]

[0093] Where τ1=1 and τ2=3 are given as 1 and 3 respectively to calculate the smooth depth loss value.

[0094] The second loss function is:

[0095]

[0096] Here, τ1 and τ2 are also given as 1 and 3.

[0097] Optionally, the total loss function is: loss = λ1loss1 + λ2loss2;

[0098] Where λ1 and λ2 are given hyperparameters, which are set to 3 and 1 respectively to balance the two loss functions.

[0099] This embodiment calculates the first loss function by combining the predicted value with the GT value, thereby improving the accuracy of the prediction. The second loss function is calculated based on the fusion of the three-dimensional anchor point position and the predicted weight, and the GT value of each joint point, thereby improving the generalization performance of the model. The network is trained based on the loss function. During the testing phase, by inputting an RGB image of the cropped hand area, the network directly predicts the coordinate position of the hand in three-dimensional space without the need for any model priors, resulting in not only accurate prediction results but also a faster running speed.

[0100] In the training phase, the trained network can be obtained as follows: Figure 7 and Figure 8 During testing, only the RGB image is needed as input to obtain the 3D hand pose estimation result of the target object.

[0101] Example 9

[0102] The present invention provides a three-dimensional hand posture estimation method, comprising: inputting an image to be identified into the trained three-dimensional hand posture model to obtain a three-dimensional hand posture estimation result of the target object; wherein the image to be identified carries the hand interaction information of the target object.

[0103] Example 10

[0104] The present invention provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the method when executed by a processor.

[0105] It will be easily understood by those skilled in the art that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for constructing a three-dimensional hand posture model, characterized in that: include: S1: Cropping the RGB image carrying the interactive hand information to obtain a first hand image; S2: Setting a plurality of evenly distributed three-dimensional anchor points on the first hand image to obtain a second hand image, and obtaining three-dimensional coordinate information of each of the three-dimensional anchor points in space; S3: Inputting the second hand image into a feature enhancement model to perform feature extraction and feature enhancement, and obtaining three-dimensional offsets and weights of each of the three-dimensional anchor points to all corresponding estimated joint points; S4: fusing the three-dimensional offsets of each of the three-dimensional anchor points to the corresponding estimated joint points, their weights, and the three-dimensional coordinate information to obtain the three-dimensional coordinates of the hand joint points corresponding to each of the three-dimensional anchor points; S5: determining a loss function according to the three-dimensional coordinates of each of the hand joint points; S6: inputting the training samples into the 3D hand posture estimation network to be trained for training, and adjusting the network parameters using the loss function during the training process, to finally obtain a trained 3D hand posture model; The S4 includes: the three-dimensional coordinates of the estimated joint point corresponding to the three-dimensional anchor point α are expressed as: plane coordinates and depth coordinate position ; ; Among them, α∈A, and Represent the plane coordinates and depth coordinate positions of the three-dimensional anchor point α respectively; and They represent the plane coordinate offset and depth coordinate offset of the three-dimensional anchor point α to the corresponding estimated joint point j respectively; The normalized weight of the three-dimensional anchor point α corresponding to the estimated joint point j represents its contribution value; , Represents the weight from the 3D anchor point α to the corresponding estimated joint point j.

2. The method for constructing a three-dimensional hand posture model according to claim 1, wherein: The S2 includes: Evenly divide the first hand image into N image blocks, set a three-dimensional anchor point at the center of each image block and obtain its plane coordinates; Taking the depth value of the root node coordinate of the hand in each of the image blocks as the center, a depth value is set before and after the depth value of the center at a preset depth in the world coordinate system to obtain three depth coordinates of the three-dimensional anchor point; The plane coordinates of the N three-dimensional anchor points and the corresponding three depth coordinates are fused to obtain 3N three-dimensional coordinate information corresponding to the N three-dimensional anchor points.

3. The method for constructing a three-dimensional hand posture model according to claim 1, wherein: The feature enhancement model includes: a multi-scale feature extractor, configured to extract features from the second hand image, select output features of the last three layers of the network structure as a multi-scale feature pyramid, fuse them to obtain first image features, process the first image features into features of the same plane size using convolutional layers and group normalization layers, and then flatten and concatenate them to obtain second image features; a feature enhancement module, configured to input the second image feature and the spatial position code of the three-dimensional anchor point into multiple coding layers for enhancement to obtain a third image feature; a 3D anchor feature interaction module, configured to extract 3D anchor features from the second hand image using a 3D anchor estimator to obtain a fourth image feature, input the fourth image feature and the third image feature into multiple decoding layers for decoding, and thereby output a fifth image feature; An offset-weight prediction module is used to receive the fifth image feature using two multi-layer perceptrons (MLPs), one of which predicts a coordinate offset. , another MLP predicts the weights .

4. The method for constructing a three-dimensional hand posture model according to claim 3, wherein: Each encoding layer contains a self-attention structure and a cross-attention structure, and the weights of the MLP in all encoding layers are shared; For the self-attention structure, the input value is: ; D represents the fifth image feature, Encode the spatial position of the 3D anchor point, ; For the crisscross attention structure, the input value is: , Indicates that the reference point in the deformable attention model is selected as the coordinate position of q three-dimensional anchor points, and E represents the third image feature; All coding layers extract the third image features corresponding to each 3D anchor point and establish the interaction between 3D anchor points.

5. The method for constructing a three-dimensional hand posture model according to claim 1, wherein: The loss function includes a first loss function and a second loss function; S5 includes: Generating the first loss function according to the three-dimensional coordinates and true value information of each of the hand joints as a first supervisory signal to supervise the obtained three-dimensional hand posture estimation result; The second loss function is generated according to the three-dimensional coordinates of each of the hand joint points and the weight corresponding to each three-dimensional anchor point, and is used as a second supervisory signal to supervise the assigned three-dimensional anchor point weights.

6. The method for constructing a three-dimensional hand posture model according to claim 5, wherein: The first loss function is: ; and Represent the true value and predicted value of the plane coordinates of the j-th joint point, respectively. and Represent the true value and predicted value of the depth coordinate of the j-th joint point respectively; Represents a given weight, Loss represents smooth L1 loss; The second loss function is: ; and Given as 1 and 3 respectively; The loss function is: ,in, and is a given hyperparameter.

7. A three-dimensional hand posture estimation method, characterized in that: include: Inputting the image to be recognized into the trained three-dimensional hand posture model according to any one of claims 1 to 6 to obtain a three-dimensional hand posture estimation result of the target object; The image to be identified carries the hand interaction information of the target object.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 7 are implemented.

Images