Three-dimensional gesture estimation method fusing hand and object features

Abstract

The invention provides a three-dimensional gesture estimation method fusing hand and object features, and relates to the technical field of gesture recognition. The method specifically comprises the following steps: acquiring a plurality of different hand and object images containing 3D postures of a hand and an object, constructing an image data set, and dividing a training set from the image data set according to a preset proportion; constructing a hand occlusion estimation joint model fusing the hand object features; inputting the training set into a hand occlusion estimation joint model fused with the hand and object features for training to obtain a trained hand occlusion estimation joint model fused with the hand and object features; and acquiring a hand and object image containing 3D postures of a hand and an object, and inputting the hand and object image into the trained hand occlusion estimation joint model fusing the hand and object features for gesture estimation to obtain a three-dimensional gesture estimation result. According to the method, the hand feature information can be better extracted under the severe shielding condition by designing the hand shielding estimation joint model fusing the hand feature.

Description

Technical Field

[0001] The present invention relates to the technical field of gesture recognition, and in particular to a three-dimensional gesture estimation method integrating hand-object features. Background Art

[0002] Posture estimation is a hot topic in the field of computer vision, aiming to accurately capture the position and shape of human posture. Among them, hand posture estimation occupies an important position and is one of the most critical research in human-computer interaction. As the future society develops towards intelligence, the demand for hand posture in virtual reality (VR) and augmented reality (AR) is increasing, and hand posture estimation has become an important research topic in the field of computer vision.

[0003] In recent years, with the rapid development of deep learning, research on 3D hand pose estimation has made significant progress. Deep learning has provided new possibilities for improving the accuracy of hand pose estimation. At present, 3D hand pose estimation methods based on RGB images usually directly predict 3D hand joints or 3D hand mesh vertex coordinates from a single RGB image. Cai et al. proposed a weakly supervised deep regularization method to transform the complex 3D coordinate estimation loss into a more intuitive and easy-to-handle depth map loss, thereby effectively solving the problem of difficulty in obtaining 3D key point markers. However, this method has the problems of complex training process, heavy computational burden, overfitting, and heavy dependence on data quality. Ge et al. integrated hand surface mesh estimation into the network, refined the 2D hand estimated from the color image through a graph convolutional network, and then estimated the mesh structure of the hand surface, and used this hand surface mesh to regress the 3D hand information. However, in practical applications, it is extremely difficult to obtain real 3D hand mesh data, which limits the application scope of this method in practical scenarios.

[0004] Compared with the above methods that only rely on joint coordinates, 3D meshes provide richer and more detailed hand geometry information. By refining vertex features through graph convolution, the morphology and structure of the hand can be more accurately described. In their study, Chen et al. cleverly extracted two-dimensional geometric information from images and exploited the inherent consistency between two-dimensional and three-dimensional. Even in the presence of noise, this geometric information can be effectively used to build an accurate hand reconstruction model. Ishii et al. proposed an innovative strategy to estimate three-dimensional poses in different directions by running multiple graph convolutional networks in parallel. These graph convolutional networks are able to capture the diversity of hand poses and generate more accurate three-dimensional pose estimates by fusing poses in multiple directions.

[0005] Although the above methods have made significant progress in solving the hand pose estimation task, the frequent interaction between hands and objects in real-world scenarios still poses a huge challenge to the pose estimation task. With the release of hand-object interaction benchmark datasets such as HO-3D and Dex-YCB, researchers have studied these datasets and proposed a unified model based on learning methods to jointly estimate the poses of hands and objects. Given that convolutional neural networks have shown excellent performance in hand interaction scenarios, most of the existing hand pose estimation methods in hand-object interaction scenarios also use convolutional neural networks as the basic architecture. Hasson et al. proposed a new loss function that can reflect the physical constraints between hands and objects, thereby optimizing the performance of pose estimation, and designed a generative network to achieve accurate estimation of hand and object poses. In addition, Tekin et al. further integrated the four tasks of hand pose estimation, object pose estimation, object recognition, and activity recognition into a unified framework and combined it with recurrent neural networks. By exploiting the relationship between tasks, they successfully improved the accuracy and reliability of hand and object pose estimation. Hasson et al. cleverly used the photometric consistency between adjacent frames as a constraint to solve the problem of interactive hand-object pose estimation, and introduced optical flow and hand masks to guide the estimation of hand and object poses. The distortion flow was regressed through the rendering process, thereby improving the accuracy and robustness of pose estimation. Zhang et al. proposed a joint learning network to estimate the 3D position of hand joints and simultaneously segment the hand and object, making full use of the depth information of the hand and object and the 3D position of the hand joints, and jointly optimizing to achieve reconstruction of hand-object interaction. Zheng et al. introduced a cross-view interaction network, which is jointly trained in an end-to-end manner. By utilizing cross-view related features and enforcing multi-view Figure 1 The single-view estimator is refined by using consistency to achieve collaborative learning.

[0006] Given the physical constraints, there is a high correlation between the postures of the hands and objects in the interaction. Therefore, the appearance information of the hands and objects can serve as useful contextual information for each other. In order to utilize this information, graph neural networks have been introduced into the hand-object interaction method. Doosti et al. first used a convolutional neural network-based method to estimate the 2D postures of the hands and objects. A graph neural network was used to learn the mapping from 2D postures to 3D postures, thereby obtaining accurate 3D postures of the hands and objects. Kulon et al. first used spiral convolution to directly estimate the 3D hand mesh. Spiral convolution performs graph convolution operations in the spatial domain and can capture the local and global structural information of the hand. Chen et al. further proposed a camera space mesh recovery network, which uses spiral convolution to estimate the vertices of the 3D hand mesh, deeply explores the impact of semantic aggregation on 3D hand reconstruction, and proposes an effective method to recover the 3D hand posture in the camera space. Yu et al. merged strided convolution into the GCN model, explicitly using the changing characteristics of different posture joints to predict their local corresponding 3D posture positions. In order to achieve efficient 3D reconstruction of hands for mobile terminals. Chen et al. proposed a lightweight network MobRecon. The network uses a depth-separable spiral convolution, which not only ensures high-precision 3D reconstruction of the hand, but also achieves a high real-time frame rate. In summary, these research works have effectively improved the accuracy and efficiency of 3D posture estimation in hand-object interaction scenarios by utilizing techniques such as graph neural networks and spiral convolution.

[0007] In recent years, Transformer has achieved remarkable results in the field of natural language processing and has demonstrated its advancedness in visual tasks. Many scholars have begun to explore the application of Transformer to hand pose estimation tasks in hand-object interaction scenarios. Lin et al. used Transformer to simulate the interaction between global vertices and successfully reconstructed the 3D human mesh structure from a single RGB image. Yang et al. used Transformer to capture the specific spatial dependencies between key points and 2D human poses, thereby improving the accuracy of pose estimation. In the work of Liu et al., a Transformer-based contextual reasoning module was proposed, which uses the features of the hand region to enhance the features of the object region when the object interacts with the hand. However, this enhanced object feature is only used for 6D object pose estimation and does not involve 3D hand mesh estimation. Ren et al. proposed a decoupled iterative refinement framework to achieve pixel-aligned hand reconstruction while effectively modeling the spatial relationship between hands. Graph convolutional networks and Transformers are used to perform intra-hand and inter-hand information interaction in the 3D joint feature space, respectively.

[0008] However, in practical applications, hand posture estimation still faces many challenges, including the complexity and diversity of the photographic environment. Problems such as the diversity of hand shapes, occlusion, and overlap have a serious impact on the accuracy of gesture estimation, resulting in the accuracy of gesture estimation still facing challenges in practical applications. Summary of the invention

[0009] In view of the above-mentioned deficiencies in the prior art, and in order to solve the problem of poor robustness caused by gesture complexity and occlusion when applying existing methods for three-dimensional gesture estimation, the present invention designs a hand occlusion estimation joint model that integrates hand-object features, and proposes a three-dimensional gesture estimation method that integrates hand-object features, which can better extract hand feature information under severe occlusion conditions.

[0010] The present invention proposes a three-dimensional hand gesture estimation method that integrates hand-object features, the method comprising the following steps:

[0011] Step 1: Obtain several different hand-object images containing 3D postures of hands and objects, construct an image dataset, and then divide the training set from the image dataset according to a preset ratio;

[0012] Step 2: Construct a joint hand occlusion estimation model that integrates hand-object features;

[0013] Step 3: Input the training set into the hand occlusion estimation joint model that integrates hand-object features for training, and obtain a trained hand occlusion estimation joint model that integrates hand-object features;

[0014] Step 4: Collect hand-object images containing the 3D postures of the hand and the object, and input them into the trained hand occlusion estimation joint model that integrates hand-object features to perform gesture estimation and obtain the 3D gesture estimation result;

[0015] The hand occlusion estimation joint model integrating hand-object features described in step 2 includes: a hand-object feature extraction module, a hand-object feature fusion module, a feature enhancement module and a regressor module;

[0016] The hand-object feature extraction module is used to extract features of different scales from the input hand-object image, obtain a hand feature map and an object feature map, and transmit them to the hand-object feature fusion module;

[0017] The hand-object feature fusion module is used to perform feature fusion on the hand feature map and the object feature map to obtain a hand-object feature map containing hand-object fusion features and transmit the map to the feature enhancement module;

[0018] The feature enhancement module is used to enhance the hand-object feature map containing the hand-object fusion feature by using the self-attention mechanism and the multi-layer perceptron to obtain the final hand-object feature map and transmit it to the regressor module;

[0019] The regressor module is used to generate a three-dimensional hand posture and a hand shape parameter according to the final feature map of the hand-object, and obtain a three-dimensional hand joint coordinate by performing a three-dimensional grid transformation on the three-dimensional hand posture and the hand shape parameter, thereby obtaining a three-dimensional hand gesture estimation result;

[0020] The hand-object feature extraction module includes: a backbone network, a cyclic feature pyramid module integrating feature residuals, an average pooling layer, and a channel attention module connected in sequence; wherein the average pooling layer adopts a residual connection;

[0021] The backbone network is used to refine the features of the input hand-object image to obtain hand-object feature maps at different scales;

[0022] The cyclic feature pyramid module with integrated feature residuals is used to extract features from hand-object feature maps at different scales to obtain hand-object image features; the cyclic feature pyramid module with integrated feature residuals is composed of a plurality of groups of serially connected feature residual networks and feature pyramid units, and for any group of serially connected feature residual networks and feature pyramid units, the output of the feature residual network is used as the input of the feature pyramid unit, wherein the feature residual network is used to split the hand-object feature maps at different scales into multiple subgroups along the channel dimension, and perform feature extraction and residual connection on each subgroup to obtain a residual network feature image; the feature pyramid unit is used to perform secondary feature extraction on hand-object feature maps at different scales by reversely transmitting the output of the feature residual network in the same group as the feature pyramid unit to the input of the feature residual network to obtain hand-object image features;

[0023] The average pooling layer is used to perform average pooling on the hand-object image features, and fuse the hand-object image features after average pooling with the hand-object image features before average pooling using a residual connection to obtain a fused hand-object feature map;

[0024] The channel attention module is used to process the fused hand-object feature map using a channel attention mechanism to obtain a hand feature map and an object feature map;

[0025] The hand-object feature fusion module includes: an attention module and a projection module connected in sequence; wherein the projection module adopts a residual connection;

[0026] The attention module is used to generate embedded representations of queries, keys and values ​​according to the hand feature map and the object feature map, and process the embedded representations of queries, keys and values ​​using a cross attention mechanism, and fuse the obtained output vector with the object feature map to obtain enhanced query features;

[0027] The projection module is used to extract features and adjust the spatial resolution of the enhanced query features to obtain projection features, and fuse the projection features with the enhanced query features through residual connection to obtain a hand-object feature map containing hand-object fusion features;

[0028] The projection module includes: a layer normalization layer, a lower projection layer, an activation function and an upper projection layer connected in sequence;

[0029] The feature enhancement module includes, in sequence: three parallel 2D convolutional layers, a self-attention module and a perceptron module; wherein the perceptron module adopts a residual connection;

[0030] The three parallel 2D convolutional layers are used to linearly project the input hand-object feature map containing the hand-object fusion feature to obtain three 2D convolution vectors as the embedded representations of the query, key and value in the feature enhancement module;

[0031] The self-attention module is used to perform self-attention calculation on the embedded representations of the query, key and value in the feature enhancement module based on the multi-head attention mechanism, obtain the output vector of the multi-head attention mechanism and perform feature fusion with hand-object fusion to obtain the output feature of the self-attention module;

[0032] The perceptron module is used to perform feature learning and feature extraction on the output features of the self-attention module, and perform feature addition on the features obtained by feature extraction and the features obtained by feature learning after scaling, so as to obtain the output features of the perceptron module;

[0033] The perceptron module includes: layer normalization, a multi-layer perceptron, a lower perceptron, an activation function, and an upper perceptron; wherein the layer normalization and the multi-layer perceptron are connected in series; wherein the lower perceptron, the activation function, and the upper perceptron are connected in series; and the layer normalization and the multi-layer perceptron connected in series are connected in parallel with the lower perceptron, the activation function, and the upper perceptron connected in series, and the output of the upper perceptron is scale-transformed and feature-added with the output of the multi-layer perceptron to obtain the output feature of the perceptron module; and the output feature of the perceptron module is fused with the output feature of the self-attention module by using a residual connection to obtain a final feature map of the hand-object;

[0034] The step 3 further comprises:

[0035] Step 3.1: Set the initial learning rate, decay factor, batch size, and maximum iteration rounds, and start iterative training;

[0036] Step 3.2: Input the hand-object images in the training set into the joint hand occlusion estimation model that integrates hand-object features, and use the backbone network to refine the features of the hand-object images to obtain hand-object feature maps at different scales;

[0037] Step 3.3: Use the cyclic feature pyramid module with integrated feature residuals to extract features of hand-object feature maps at different scales to obtain hand-object image features P2;

[0038] Step 3.4: Perform average pooling on the hand-object image feature P2, and use residual connection to fuse the hand-object image feature after average pooling with the hand-object image feature P2 to obtain a fused hand-object feature map;

[0039] Step 3.5: Use the channel attention mechanism to process the fused hand-object feature map to obtain the hand feature map F hand and object feature map F object ;

[0040] Step 3.6: Get the hand feature map F hand The hand features and object features in F object The object features in the hand feature map are used as the query, and the intersection area of ​​the hand feature map and the object feature map are used as the key and value to generate an embedding representation of the query, key and value;

[0041] Step 3.7: Use the cross-attention mechanism to process the embedding representations of the query, key, and value to obtain the output vector, and then compare the output vector with the object feature map F object Fusion is performed to obtain enhanced query features Q τ ;

[0042] Step 3.8: Use the projection module to enhance the query feature Q τ Projection is performed to obtain the projection feature H, and the projection feature H is combined with the enhanced query feature Q τ Fusion is performed to obtain the hand-object fusion feature

[0043] Step 3.9: Fusion of opponent features Perform linear projection and use the three 2D convolution vectors obtained as the embedding representations of the query, key and value in the feature enhancement module. Perform self-attention calculation on the embedding representations of the query, key and value in the feature enhancement module based on the multi-head attention mechanism to obtain the output vector of the multi-head attention mechanism and combine it with the hand-object fusion feature. Merge to obtain the output features of the self-attention module;

[0044] Step 3.10: The output features of the self-attention module are divided into two paths, one of which uses the series-connected layer normalization and multi-layer perceptron for feature learning to obtain the output of the multi-layer perceptron; the other path uses the series-connected lower perceptron layer, activation function and upper perceptron layer for feature extraction to obtain the output of the upper perceptron layer; the output of the upper perceptron layer is scaled and then added to the output of the multi-layer perceptron to obtain the output features of the perceptron module;

[0045] Step 3.11: Fuse the output features of the perceptron module with the output features of the self-attention module to obtain the final features of the hand and object and generate the final feature map of the hand and object;

[0046] Step 3.12: Based on the final feature map of the hand and object, a single hourglass network is used to generate a two-dimensional heat map of each joint of the hand. Several residual blocks connected in series are used to perform feature fusion on the two-dimensional heat map of each joint of the hand, and the output of the residual block is vectorized to obtain a multi-dimensional feature vector.

[0047] Step 3.13: Generate three-dimensional hand posture and hand shape parameters according to the multidimensional feature vector, and perform three-dimensional mesh transformation according to the three-dimensional hand posture and hand shape parameters by constructing a regression matrix of the hand joints to generate three-dimensional hand joint coordinates, and then obtain a three-dimensional hand gesture estimation result according to the three-dimensional hand joint coordinates;

[0048] Step 3.14: Determine whether the current iteration round has reached the maximum iteration round. If so, end the training to obtain the trained hand occlusion estimation joint model that integrates hand-object features; otherwise, return to step 3.2;

[0049] The method of step 3.3 is specifically as follows: for any feature residual network in the cyclic feature pyramid module of integrated feature residual, the hand-object feature map U is input into the feature residual network, where U∈R H×W×C , H, W and C are the height, width and number of channels of the image respectively; the hand-object feature map U is subjected to two convolutions and two feature convolutions to generate two sampling results X α and X β , X α and X β Multiply and add to X β Add together to get the output X of a feature residual network i ;

[0050] X i =X α ×X β +X β

[0051]

[0052] Where P i (·) is the upsampling function; δ(·) is the activation function; APool(U) represents the average pooling of the object feature map U; is the attention convolution channel f attn (·) output, and The output of the attention channel And the structure of the attention convolution layer μ attn as a parameter.

[0053] Then, the feature pyramid unit of the same group as the feature residual network is used to reversely transmit the output of the feature residual network to the input of the feature residual network, perform secondary feature extraction on the hand feature map, and obtain the hand image feature;

[0054] The hand feature map F described in step 3.5 hand and object feature map F object It is expressed as:

[0055] F object =CAM object ×[Up(APool(P2))+P2]

[0056] F hand =CAM hand ×[Up(APool(P2))+P2]

[0057] Where Up is the upsampling function; APool(·) is the average pooling layer; CAM object and CAM hand Represent the channel attention modules for objects and hands respectively;

[0058] The hand-object fusion features described in step 3.8 The calculation method is:

[0059] E=Conv(Q τ ,W1)+b1

[0060] H=ConvTranspose(ReLU(E),W2)+b2

[0061] Where E is the output of the lower projection layer in the projection module; Conv(·) represents convolution; ConvTranspose(·) represents bias convolution; W1 and W2 represent the convolution kernel of the lower projection layer and the convolution kernel of the upper projection layer respectively; b1 and b2 represent the bias of the lower projection layer and the bias of the upper projection layer respectively; ReLU(·) is the linear unit activation function;

[0062]

[0063] Where H(Q τ ) represents the enhanced query feature Q τ Perform projection;

[0064] The output of the upper perception layer described in step 3.10 is expressed as:

[0065]

[0066] in represents the output of the upper perception layer; ReLU(·) represents the ReLU activation function; LN(·) represents layer normalization; X θ represents the output features of the self-attention module; W down is the parameter of the lower perception layer, and W up is the parameter of the upper perception layer, and where n and Represent the initial dimension and the intermediate dimension respectively, and satisfy

[0067] The output feature of the perceptron module is expressed as:

[0068]

[0069] Where X ρ represents the output features of the perceptron module; S is the scale.

[0070] The beneficial effects of adopting the above technical solution are:

[0071] (1) The method of the present invention proposes a hand-object feature extraction module based on a residual network. This module introduces a residual network that integrates spatial and channel attention on the basis of a recurrent feature pyramid network. By fine-tuning the hand and object feature information of the input image, the hand feature map and the object feature map can be extracted from the image respectively, providing a more accurate basis for subsequent feature fusion.

[0072] (2) The method of the present invention proposes a hand-object feature fusion module, which uses the object feature map to enhance the hand feature map, so that it pays more attention to the hand information contained in the object. Even when the object severely occludes the hand, the module can still effectively extract the hand information, further optimizing the overall network model and improving the prediction accuracy of the model.

[0073] (3) The method of the present invention proposes a feature enhancement module for enhancing feature extraction of a multi-layer perceptron. The module significantly enhances the nonlinear modeling capability of the network by introducing a series of nonlinear activation functions and inter-layer connection mechanisms. The module not only improves the efficiency of feature extraction and information transmission, but also enhances the adaptability of the model to data.

[0074] In summary, the proposed method of the present invention proposes a joint model for hand occlusion estimation that integrates hand-object features, which can extract object features from severely occluded feature maps and integrate them with hand features, thereby effectively processing the occluded part of the hand and more accurately estimating the three-dimensional hand posture. The residual network is used to process the image, which is combined with the cyclic feature pyramid structure to obtain the object and hand feature maps respectively; the object and hand feature maps are integrated through sampling convolution and single-head attention mechanism to make up for the feature information of the part of the hand feature map that is occluded by the object; the new sampling method is applied to the feedforward network of the feature fusion module to improve the efficiency of information transmission and realize the encoding of hand information. The method of the present invention effectively solves the competition problem between gesture and object posture estimation tasks, especially significantly improves the accuracy of gesture estimation in the case of occlusion. The experimental results show that compared with other existing methods, the method of the present invention can obtain lower joint point errors, can better estimate hand posture in scenes where hand features and object features are occluded, and achieves higher accuracy and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 is a flow chart of a three-dimensional hand gesture estimation method integrating hand-object features in this embodiment;

[0076] Figure 2 is a structural diagram of a joint model for hand occlusion estimation that integrates hand-object features in this embodiment;

[0077] Figure 3 Schematic diagram of the hand-object feature extraction process in this embodiment;

[0078] Figure 4 Schematic diagram of the hand-object feature fusion process in this embodiment;

[0079] Figure 5 is a schematic diagram of the feature enhancement process in this embodiment;

[0080] Figure 6 is a schematic diagram of qualitative comparison of the hand occlusion estimation joint model integrating hand-object features in this embodiment on the HO-3D dataset;

[0081] Figure 7 Schematic diagram of the comparison of PCK curve and PCV curve under different errors of the joint model of hand occlusion estimation integrating hand-object features in this embodiment; wherein Figure (a) is a schematic diagram of the comparison of PCK curve under different errors; Figure (b) is a schematic diagram of the comparison of PCV curve under different errors;

[0082] Figure 8The figures are result diagrams of using a monocular camera to take photos of hands with obstructions at different viewing angles and perform 3D gesture estimation in this embodiment; wherein Figure (a) is a result diagram of 3D gesture estimation for a photo of a hand with obstructions at a frontal viewing angle; Figure (b) is a result diagram of 3D gesture estimation for a photo of a hand with obstructions at a side viewing angle; and Figure (c) is a result diagram of 3D gesture estimation for a photo of a hand with obstructions at a bottom viewing angle. DETAILED DESCRIPTION

[0083] In order to facilitate the understanding of the present application, the specific embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and embodiments. The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thoroughly understood.

[0084] In this embodiment, firstly, the residual network in the feature attention mechanism is used to process the image, and it is combined with the cyclic feature pyramid structure to obtain the hand feature map and the object feature map respectively, so as to improve the perception and expression ability of the image; secondly, the sampling convolution and the single-head attention mechanism are used to fuse the object feature map and the hand feature map, so that the hand feature map contains as many interaction area features as possible, so as to make up for the lack of features in the interaction area; thirdly, the new sampling method is applied to the feedforward network of the hand-object feature fusion module to improve the efficiency of information transmission and realize hand information encoding; finally, the processed hand features are input into the regressor decoding network to obtain the final three-dimensional gesture estimation result.

[0085] This embodiment provides a three-dimensional gesture estimation method that integrates hand and object features. Figure 1 As shown, the method comprises the following steps:

[0086] Step 1: Obtain several different hand-object images containing 3D postures of hands and objects, build an image dataset, and then divide the training set from the image dataset according to a preset ratio.

[0087] In this embodiment, several hand-object images of different 3D postures of hands and objects are obtained from existing data sets, such as the HO3D data set and the Dex-YCB data set, and an image data set is constructed, and then the image data set is divided into a training set and a test set in a ratio of 7:3.

[0088] Step 2: Build a joint hand occlusion estimation model that integrates hand-object features.

[0089] The hand occlusion estimation joint model integrating hand-object features includes: a hand-object feature extraction module, a hand-object feature fusion module, a feature enhancement module and a regressor module.

[0090] In this embodiment, if Figure 2 As shown in the figure, first, the hand-object image is input into the hand-object feature extraction module, which is responsible for extracting feature information at different scales from the input RGB image, thereby obtaining the hand feature map and the object feature map respectively. Secondly, the hand-object feature fusion module is used to effectively combine the object feature map with the hand feature map to extract a more accurate hand-object feature map. Thirdly, the hand-object feature map is encoded using a feature enhancement module with a novel feedforward network. Finally, the real-time 3D hand model loader (Metric-Affine HandModel, MANO) parameters are generated through the regressor module, and then passed to the real-time 3D hand model loader MANO to obtain the final 3D hand mesh.

[0091] The hand-object feature extraction module is used to extract features of different scales from the input hand-object image, obtain a hand feature map and an object feature map, and transmit them to the hand-object feature fusion module.

[0092] The hand-object feature extraction module includes: a backbone network, a cyclic feature pyramid module integrating feature residuals, an average pooling layer, and a channel attention module connected in sequence; wherein the average pooling layer adopts a residual connection.

[0093] The backbone network is used to refine the features of the input hand-object image to obtain hand-object feature maps at different scales.

[0094] The cyclic feature pyramid module with integrated feature residuals is used to extract features from hand-object feature maps at different scales to obtain hand-object image features; the cyclic feature pyramid module with integrated feature residuals is composed of a plurality of groups of serially connected feature residual networks and feature pyramid units, and for any group of serially connected feature residual networks and feature pyramid units, the output of the feature residual network is used as the input of the feature pyramid unit, wherein the feature residual network is used to split the hand-object feature maps at different scales into multiple subgroups along the channel dimension, and perform feature extraction and residual connection on each subgroup to obtain a residual network feature image; the feature pyramid unit is used to perform secondary feature extraction on hand-object feature maps at different scales by reversely transmitting the output of the feature residual network in the same group as the feature pyramid unit to the input of the feature residual network to obtain hand-object image features.

[0095] The average pooling layer is used to perform average pooling on the hand-object image features, and use residual connection to fuse the hand-object image features after average pooling with the hand-object image features before average pooling to obtain a fused hand-object feature map.

[0096] The channel attention module is used to process the fused hand-object feature map using the channel attention mechanism to obtain a hand feature map and an object feature map.

[0097] In this embodiment, if Figure 3 As shown in the figure, the input image is refined through the backbone network, and its parameters are fixed and do not participate in the subsequent feature extraction part. Subsequently, a circular feature pyramid module (Circular Feature PyramidNet, CiFPN) with integrated feature residuals is used to extract features of hand-object feature maps at different scales. Among them, the feature residual network first splits the input feature map into multiple subgroups along the channel dimension, and then integrates the dependencies of each subgroup feature in the spatial domain and channel dimension, and realizes component feature fusion through channel conversion operation. This method can effectively combine the contextual information of spatial and channel information, accurately activate the important features of the feature map, and thus realize efficient extraction of the input feature map. The circular feature pyramid reversely transmits the output of each layer of the feature residual network to the input of the feature residual network, realizing secondary extraction of effective image information and effectively eliminating redundant information.

[0098] The hand-object feature fusion module is used to perform feature fusion on the hand feature map and the object feature map, obtain a hand-object feature map containing hand-object fusion features, and transmit the map to the feature enhancement module.

[0099] In this embodiment, if Figure 4 As shown in Figure 1, the hand-object feature fusion module combines the functions of the attention module and the projection module, aiming to enhance the hand feature map using the extracted object feature map. In severely occluded scenes, the object in the hand can often serve as a key feature cue for the hand posture.

[0100] The hand-object feature fusion module includes: an attention module and a projection module connected in sequence; wherein the projection module adopts a residual connection.

[0101] The attention module is used to generate embedded representations of queries, keys and values ​​based on the hand feature map and the object feature map, and use a cross-attention mechanism to process the embedded representations of queries, keys and values, and fuse the obtained output vector with the object feature map to obtain enhanced query features.

[0102] The projection module is used to extract features and adjust the spatial resolution of the enhanced query features to obtain projection features, and fuse the projection features with the enhanced query features through residual connections to obtain a hand-object feature map containing hand-object fusion features.

[0103] The projection module includes: a layer normalization layer, a lower projection layer, an activation function and an upper projection layer connected in sequence.

[0104] In this embodiment, the hand-object feature fusion module receives the hand feature map and the object feature map as input. Inside the module, the object feature is used as the query query, and the intersection area of ​​the hand and the object is used as the key and value. Through this design, the query position in the object feature can be fused with the information of the interaction area, thereby enhancing the hand feature map. In the case where the object occludes the hand, the hand occlusion estimation joint model that fuses the hand-object features can still accurately capture the information related to the hand posture, improving the accuracy and robustness of feature extraction. For the projection module, the upper projection layer is implemented using transposed convolution, and the lower projection layer is implemented using ordinary convolution. The hand-object feature fusion module can effectively combine hand features and object features to achieve accurate extraction and enhancement of hand information under occlusion.

[0105] The feature enhancement module is used to utilize the self-attention mechanism and the multi-layer perceptron to perform feature enhancement on the hand-object feature map containing the hand-object fusion feature, obtain the final hand-object feature map and transmit it to the regressor module.

[0106] The feature enhancement module includes, in sequence: three parallel 2D convolutional layers, a self-attention module and a perceptron module; wherein the perceptron module adopts a residual connection.

[0107] The three parallel 2D convolutional layers are used to perform linear projection on the input hand-object feature map containing the hand-object fusion feature to obtain three 2D convolution vectors and use them as embedded representations of queries, keys and values ​​in the feature enhancement module.

[0108] The self-attention module is used to perform self-attention calculation on the embedded representations of queries, keys and values ​​in the feature enhancement module based on the multi-head attention mechanism, obtain the output vector of the multi-head attention mechanism and perform feature fusion with hand-object fusion to obtain the output features of the self-attention module.

[0109] The perceptron module is used to perform feature learning and feature extraction on the output features of the self-attention module respectively, and add the features obtained by feature extraction to the features obtained by feature learning after scaling, so as to obtain the output features of the perceptron module.

[0110] The perceptron module includes: layer normalization, a multi-layer perceptron, a lower perceptron, an activation function and an upper perceptron; wherein the layer normalization and the multi-layer perceptron are connected in series; the lower perceptron, the activation function and the upper perceptron are connected in series; and the layer normalization and the multi-layer perceptron connected in series are connected in parallel with the lower perceptron, the activation function and the upper perceptron connected in series, and the output of the upper perceptron is scale-transformed and feature-added with the output of the multi-layer perceptron to obtain the output feature of the perceptron module; the output feature of the perceptron module is fused with the output feature of the self-attention module by using a residual connection to obtain a final feature map of the hand-object.

[0111] In this embodiment, if Figure 5 As shown in the figure, a new sampling method is proposed in the feature enhancement module and applied to the multilayer perceptron module, aiming to enhance the perception ability of the features. The feature enhancement module improves the transfer learning ability of the model by fine-tuning a small number of additional parameters, which can effectively avoid interference between different features. The feature enhancement module is mainly composed of two types of sub-layers, namely the single-head self-attention layer and the multilayer perceptron (Multilayer Perceptron, MLP). The perceptron module consists of two parts: one part is the original MLP network, that is, the output features of the self-attention module are sent to the series normalization layer and the MLP layer respectively; the other part introduces a lightweight network structure, including two fully connected layers: the lower perception layer and the upper perception layer, and the GELU activation function is used for activation between the two fully connected layers. Specifically, the lightweight network structure includes parameters The lower perception layer and parameters are The upper perception layer, where n and Represent the initial dimension and the intermediate dimension respectively, and satisfy Between the upper and lower perception layers, a ReLU activation function is added to increase its nonlinear characteristics. Finally, the scale factor is integrated with the original MLP network through a residual connection. The feature enhancement module effectively combines the advantages of the self-attention mechanism and the new perceptron module to achieve feature enhancement and optimization.

[0112] The regressor module is used to generate three-dimensional hand posture and hand shape parameters according to the final feature map of the hand-object, and obtain three-dimensional hand joint coordinates by performing three-dimensional grid transformation on the three-dimensional hand posture and hand shape parameters, thereby obtaining a three-dimensional hand gesture estimation result.

[0113] In this embodiment, the regressor module is designed to generate posture and shape parameters, and finally generate a three-dimensional hand mesh by passing these parameters to the corresponding layers. Specifically, a single-block hourglass network is first used, the enhanced final features are used as input, and a two-dimensional heat map of each joint is output. Subsequently, in order to further improve the expressiveness of the features, four residual blocks are connected in series to effectively fuse the features of each layer. Finally, the output of the residual block is vectorized to obtain a 2048-dimensional feature vector. The vector is then passed to the fully connected layer for predicting MANO posture parameters and shape parameters, that is, three-dimensional hand posture and shape parameters of the hand. After obtaining the MANO posture parameters and shape parameters, the three-dimensional mesh of the static posture is transformed using the joint regression matrix, and the forward kinematics principle is used to obtain accurate three-dimensional hand joint coordinates. Finally, based on these joint coordinates, an accurate three-dimensional hand mesh is generated.

[0114] Step 3: Input the training set into the hand occlusion estimation joint model that integrates hand-object features for training, and obtain a trained hand occlusion estimation joint model that integrates hand-object features.

[0115] Step 3.1: Set the initial learning rate, decay factor, batch size, and maximum iteration rounds, and start iterative training.

[0116] Step 3.2: Input the hand-object images in the training set into the joint hand occlusion estimation model that integrates hand-object features, and use the backbone network to refine the features of the hand-object images to obtain hand-object feature maps at different scales.

[0117] Step 3.3: Use the cyclic feature pyramid module with integrated feature residuals to extract features of hand-object feature maps at different scales to obtain the hand-object image feature P2.

[0118] For any feature residual network in the cyclic feature pyramid module that integrates feature residuals, the hand-object feature map U is input into the feature residual network, where U∈R H×W×C , H, W and C are the height, width and number of channels of the image respectively; the hand-object feature map U is subjected to two convolutions and two feature convolutions to generate two sampling results X α and X β , X α and X β Multiply and add to X β Add together to get the output X of a feature residual network i ;

[0119] X i =X α ×X β +X β (1)

[0120]

[0121] Where P i (·) is the upsampling function used to expand the output of the attention channel; δ(·) is the activation function; APool(U) represents the average pooling of the opponent feature map U; is the attention convolution channel f attn (·) output, and the output is the output of the attention channel And the structure of the attention convolution layer μ attn as a parameter.

[0122] Then, the feature pyramid unit of the same group as the feature residual network is used to reversely transmit the output of the feature residual network to the input of the feature residual network, perform secondary feature extraction on the hand feature map, and obtain the hand image feature;

[0123] Step 3.4: Perform average pooling on the hand-object image feature P2, and use residual connection to fuse the hand-object image feature after average pooling with the hand-object image feature P2 to obtain a fused hand-object feature map.

[0124] In this embodiment, the hand-object image feature P2 is averaged and processed, which can further aggregate feature information and reduce the amount of calculation. The pooled result is merged with the original hand-object image feature P2, so that the feature map retains the original spatial structure and incorporates the global information after pooling, thereby enhancing the expressiveness of the feature.

[0125] Step 3.5: Use the channel attention mechanism to process the fused hand-object feature map to obtain the hand feature map F hand and object feature map F object .

[0126] F object =CAM object ×[Up(APool(P2))+P2] (4)

[0127] F hand =CAM hand ×[Up(APool(P2))+P2] (5)

[0128] Where Up is the upsampling function used to restore the pooled feature map to the same size as the original output; APool(·) is the average pooling layer; CAM object and CAM hand Represent the channel attention modules for objects and hands, respectively.

[0129] Step 3.6: Get the hand feature map F hand The hand features and object features in F object The object features in the hand feature map are used as the query, and the intersection area of ​​the hand feature map and the object feature map are used as the key and value, and the embedding representation of the query, key and value is generated;

[0130] Q=w q F object ,K=w k F hand ,V=w v F hand (6)

[0131] Where Q is the embedding representation of the query; K is the embedding representation of the key; V is the embedding representation of the value; w q , w k , w v They are all independent one-dimensional convolutions.

[0132] In this embodiment, a method consisting of three parameters w and w is applied to the hand features and the object features. q , w k , w v Independent 1D convolutions are performed to generate embeddings of the query, key, and value. The embeddings of the query, key, and value are fed into the attention module, where a cross-attention mechanism is performed between the query and each spatial position key. In the attention module, the output Att(Q, K, V) is calculated by solving the weighted average.

[0133] Step 3.7: Use the cross-attention mechanism to process the embedding representations of the query, key, and value to obtain the output vector and combine the projected feature H with the enhanced query feature Q τ Fusion is performed to obtain the hand-object fusion feature

[0134] The output vector is represented as:

[0135]

[0136] Where Att(Q,K,V) represents the output vector; Softmax(·) represents the Softmax function; d K Represents the feature dimension of the key.

[0137] In this embodiment, the embedded representations of the query, key, and value are fed into an attention module, where a cross-attention mechanism is performed between the query and each spatial position key, i.e., in the attention module, the output vector is calculated by solving the weighted average.

[0138] The enhanced query feature Q τ It is expressed as:

[0139] Q τ =F object +Att(Q,K,V) (8)

[0140] Step 3.8: Use the projection module to enhance the query feature Q τ Projection is performed to obtain the projection feature H, and the projection feature H is combined with the enhanced query feature Q τ Fusion is performed to obtain the hand-object fusion feature

[0141] E=Conv(Q τ ,W1)+b1 (9)

[0142] H=ConvTranspose(ReLU(E),W2)+b2 (10)

[0143] Where E is the output of the lower projection layer in the projection module; Conv(·) represents convolution; ConvTranspose(·) represents bias convolution; W1 and W2 represent the convolution kernel of the lower projection layer and the convolution kernel of the upper projection layer respectively; b1 and b2 represent the bias of the lower projection layer and the bias of the upper projection layer respectively; ReLU(·) is the linear unit activation function;

[0144]

[0145] Where H(Q τ ) represents the enhanced query feature Q τ Projection.

[0146] Step 3.9: Fusion of opponent features Perform linear projection and use the three 2D convolution vectors obtained as the embedding representations of the query, key and value in the feature enhancement module. Perform self-attention calculation on the embedding representations of the query, key and value in the feature enhancement module based on the multi-head attention mechanism to obtain the output vector of the multi-head attention mechanism and combine it with the hand-object fusion feature. Merge to obtain the output features of the self-attention module;

[0147]

[0148] Where X θ represents the mark generated by the multi-head attention mechanism at the θth layer and serves as the output feature of the self-attention module; Attention(·) represents the multi-head attention mechanism; d represents the output channel;

[0149] In this embodiment, for each input vector, three linear transformations, namely weighting and biasing, are performed to generate embedded representations of queries, keys, and values. In the multi-head attention mechanism, this step is actually performed θ times, where θ is the number of heads, and each head has an independent weight matrix, thereby dividing the input vector into θ different subspaces. For each subspace, the self-attention mechanism is applied to calculate the attention weight, and the weighted sum V is taken to obtain the output of each head. The outputs of all heads are concatenated, and then subjected to a final linear transformation and layer normalization to obtain the output of the multi-head attention.

[0150] X η =MLP(LN(X' θ ))+X θ (13)

[0151] Where X η represents the output of the multi-head attention mechanism at the nth encoding block; LN(·) represents layer normalization; X' θ Represents the encoding block of the multi-head attention mechanism at layer n;

[0152] In this embodiment, the multi-head self-attention mechanism is an extension of the Transformer, which enhances the expressiveness of the model by calculating multiple self-attention mechanisms in parallel. The feature enhancement module is regarded as a Transformer containing multiple encoders, and the feature map is encoded using a feature enhancement module with a novel feedforward network. Each encoder contains several encoding blocks, and each encoding block is mainly composed of two types of sub-layers, a single-head self-attention layer and a multi-layer perceptron MLP, namely a self-attention module and a perceptron module; and the encoding block in each encoder is an encoding block of an encoding layer in the Transformer. In the self-attention layer, the input is linearly projected and further reformulated into three vectors, and self-attention calculations are performed. The multi-head self-attention mechanism is an extension of the Transformer, which enhances the expressiveness of the model by calculating multiple self-attention mechanisms in parallel.

[0153] Step 3.10: The output features of the self-attention module are divided into two paths. One path uses the series-connected layer normalization and multi-layer perceptron for feature learning to obtain the output of the multi-layer perceptron; the other path uses the series-connected lower perceptron layer, activation function and upper perceptron layer for feature extraction to obtain the output of the upper perceptron layer; the output of the upper perceptron layer is scaled and then added to the output of the multi-layer perceptron to obtain the output features of the perceptron module.

[0154] The output of the upper perception layer is expressed as:

[0155]

[0156] in represents the output of the upper perception layer; ReLU(·) represents the ReLU activation function; W down is the parameter of the lower perception layer, and W up is the parameter of the upper perception layer, and where n and Represent the initial dimension and the intermediate dimension respectively, and satisfy

[0157] The output feature of the perceptron module is expressed as:

[0158]

[0159] Where X ρ represents the output features of the perceptron module; S is the scale.

[0160] Step 3.11: Fuse the output features of the perceptron module with the output features of the self-attention module to obtain the final hand-object features and generate the final hand-object feature map.

[0161] Fh =X θ +X ρ (16)

[0162] where F h Indicates the final characteristics of the hand object.

[0163] Step 3.12: Based on the final feature map of the hand-object, a single hourglass network is used to generate a two-dimensional heat map of each joint of the hand. Several residual blocks connected in series are used to perform feature fusion on the two-dimensional heat map of each joint of the hand, and the output of the residual block is vectorized to obtain a multi-dimensional feature vector.

[0164] Step 3.13: Generate three-dimensional hand posture and hand shape parameters based on the multidimensional feature vector, and perform three-dimensional mesh transformation based on the three-dimensional hand posture and hand shape parameters by constructing a regression matrix of the hand joints to generate three-dimensional hand joint coordinates, and then obtain the three-dimensional hand gesture estimation result based on the three-dimensional hand joint coordinates.

[0165] Step 3.14: Determine whether the current iteration round has reached the maximum iteration round. If so, end the training and obtain the trained hand occlusion estimation joint model that integrates hand-object features; otherwise, return to step 3.2.

[0166] Step 4: Collect hand-object images containing the 3D postures of the hand and the object, and input them into the trained hand occlusion estimation joint model that integrates hand-object features to perform gesture estimation and obtain the 3D gesture estimation result.

[0167] In this embodiment, the HO3D dataset and the Dex-YCB dataset are respectively used to evaluate the method of the present invention; wherein the HO3D dataset is used to evaluate the relevant methods, and the key indicators returned by the online evaluation server CodaLab include the average joint error, the grid error (unit: mm), and the F-Score score. These indicators can objectively evaluate the performance of each method. Four evaluation indicators are used to evaluate the Dex-YCB dataset, namely COCO, BOP, HPE and Grasp. The method of the present invention focuses on the human pose estimation (HumanPose Estimation, HPE) method, and evaluates the performance of the model by calculating the mean position error (Mean Per Joint Position Error, MPJPE) of each key point and the mean position error (Mean Per Joint Position Error, PAMPJPE) after Prof.

[0168] The average position error is used to measure the difference between the training data and the real data. Specifically, for each hand-object image sample I in the training set, the Euclidean distance between the predicted joint position and the real joint position is summed and then divided by the number of joints to obtain the corresponding error value. The smaller the value, the closer the predicted result is to the real result, and the higher the accuracy of the posture estimation. The formula for the average position error is:

[0169]

[0170] PAMPJPE=PA(MPJPE) (18)

[0171] Where MPJPE is the mean position error; N is the total number of samples; ||·||2 is the second norm; P v G is the training grid for the network; v is the real data grid; PAMPJPE is the average position error after Proctor alignment; PA is Proctor alignment.

[0172] F-score mainly includes F@5 and F@15, which represent the calculated values ​​at the error thresholds of 5 and 15. Its calculation mainly involves precision P and recall L, which are calculated at different joint positions and error thresholds to more comprehensively evaluate the performance of the model. Its calculation method is:

[0173]

[0174] Where P θ is the number of true positive examples; P δ is the number of false positive examples; N θ is the number of false negative examples.

[0175] The online evaluation server also draws the relevant key point correct estimation ratio (PCK) curve and the vertex correct estimation ratio (PCV) curve according to the evaluation indicators of the model. The PCK curve and PCV curve refer to the prediction accuracy change curves of key points and vertices when the distance threshold is within a certain range. In the HO3D dataset, the distance threshold range is 0 to 5 cm. The curves of PCK and PCV values ​​changing with the distance threshold can be drawn according to this range. The calculation method is:

[0176]

[0177] in and Represent the three-dimensional coordinates of the pth key point and vertex respectively; and denote the real coordinates of key points and vertices respectively; l is the threshold between the artificially set and the real distance, in mm; T denotes the data set used to evaluate the indicator; υ(·) is the indicator function; and They represent the percentage of the pth key point and vertex above the threshold l respectively.

[0178] In this embodiment, an existing data set is used to experimentally verify a 3D gesture estimation method that integrates hand-object features. The experiment uses an NVIDIA A10 Tensor Core graphics card, an operating system of Ubuntu 18.04, a development framework of PyTorch, and an optimizer using the Adam method. Model training is performed on the Dex-YCB data set and the HO-3D data set, respectively. In order to better adapt to the characteristics of different data sets, different hyperparameter settings are used for each data set in order to achieve the best performance.

[0179] (1) HO3D Dataset and Experiments

[0180] HO3D is a large-scale dataset of 3D poses of hands and objects, covering 68 sequences with 66,034 training images and 11,524 test images. These sequences are captured in multi-camera and single-camera settings, with 10 different subjects operating 10 different objects in the YCB dataset, all images are taken from a third-person perspective, and the object pose annotations for the training and test sets are provided directly by the dataset.

[0181] The training was conducted on the HO-3D dataset, with the initial learning rate set to 2.5e-5. The learning rate was adjusted to decrease every ten training rounds, with a decay factor of 0.7 and a batch size of 16. After the training, the training results were submitted to the official server for evaluation to obtain objective performance indicators. During the evaluation process, the method of the present invention focused on comparing the average joint error PAMPJPE, the average grid error PAMPVPE after Prof. Schmidt alignment, in millimeters, and indicators such as the F score. The experimental results show that the method of the present invention is superior to the recently proposed MobRecon method in most indicators. As shown in Table 1, the experimental comparison results of different methods are given in detail, in which the posture representation is used to describe the hand-object spatial posture. In addition, in order to more comprehensively evaluate the model performance, the experimental results before Prof. Schmidt alignment are given in Table 2, which further verifies the effectiveness and robustness of the method of the present invention in dealing with hand posture estimation and object interaction tasks.

[0182] Table 1 Comparative experiments on the HO3D dataset

[0183]

[0184] Note: Bold fonts indicate the best results in each column.

[0185] Table 2 Comparative experiments on the HO3D dataset before data alignment

[0186]

[0187] Note: Bold fonts indicate the best results in each row.

[0188] (2) Dex-YCB Dataset and Experiments

[0189] The Dex-YCB dataset provides rich image and video resources, which are often used to study tasks such as hand posture estimation and object grasping. The dataset contains 582K images from more than 1,000 video sequences. These images are acquired through 8 RGB-D cameras, and RGB images and depth maps from 8 perspectives are recorded at a frame rate of 30fps, with a resolution of up to 640*480, to ensure the precision and accuracy of the data.

[0190] The model was trained on the Dex-YCB dataset, with an initial learning rate of 2.5e-5, and the learning rate was adjusted downward after each training round, with a decay factor of 0.7, a total number of training rounds of 25, and a batch size of 16. As shown in Table 3, compared with the method of Liu et al., which has the best result index among the related methods, the performance of the method of the present invention on the two evaluation indicators of PAMPJPE and MPJPE was improved by 0.84mm and 1.04mm, respectively, which verifies that the method of the present invention has a higher accuracy in processing hand posture estimation and object grasping tasks.

[0191] Table 3 Comparative experiments on the Dex-YCB dataset

[0192]

[0193] Note: Bold fonts indicate the best results in each row.

[0194] In this implementation, the hand occlusion estimation joint model integrating hand-object features is qualitatively compared with other existing methods on the HO-3D dataset. Figure 6 As shown, starting from the leftmost side, the first column is the input image, the second column is the front 3D gesture estimation results and the side 3D gesture estimation results obtained by METRO for the HO-3D dataset, and the third column is the front 3D gesture estimation results and the side 3D gesture estimation results obtained by the method proposed by METROSpurr et al. for the HO-3D dataset. The last column is the front 3D gesture estimation results and the side 3D gesture estimation results obtained by the method of the present invention for the HO-3D dataset. Figure 6 It can be seen that the 3D gesture obtained by the method of the present invention is more accurate. Figure 7The PCK and PCV curves of the method of the present invention under different errors on the data set are shown. Figure 7 It can be seen that compared with the PCK curves of alignment key point error and scale translation alignment key point error, the PCK curve of Proof alignment key point error reaches the percentage range of the optimal correct key point when the threshold distance is 3cm. The PCV curve of the model is better after alignment than before alignment, and reaches the percentage range of the optimal correct vertex when the threshold distance is 3cm. These indicators quantify the performance of the model in the pose estimation task.

[0195] To further verify the robustness of the model in real occlusion environments, a monocular camera was used to take photos of hands under different viewing angles and occlusions, and a series of posture estimation experiments were conducted. Figure 8 As shown, the experimental results show that even in the case of severe occlusion, the method of the present invention can accurately give reasonable hand posture prediction results through object-to-hand feature fusion and feature enhancement technology.

[0196] (3) Ablation experiment

[0197] The data sets used in the ablation experiments are all HO-3D data sets, and the experimental results are shown in Table 4. The modules proposed by the method of the present invention all play a key role in improving the overall performance of the method, and the mutual fusion of the three modules can significantly improve the evaluation indicators. Among them, the Joint and Mesh indicators were reduced by 8.8% and 5.1% respectively, and F@5 and F@15 were improved by 6.2% and 0.6% respectively. In the ablation experiment, HEM represents the hand-object feature extraction module, HFM represents the hand-object feature fusion module, and FIM represents the feature enhancement module. The following ablation experiments are carried out on the residual network and the recurrent feature pyramid network in the hand-object feature extraction module, the sampling convolution and single-head attention mechanism network of the hand-object feature fusion module, and the sampling method of the feedforward network in the feature enhancement module to verify the effectiveness of the method of the present invention.

[0198] Table 4 Comparative test results of different modules

[0199]

[0200] Note: Bold fonts indicate the best results in each column.

[0201] The role of residual network and cyclic feature pyramid network: By adding different structures to the hand-object feature extraction module, the influence of the residual network and cyclic feature pyramid network in the hand-object feature extraction module on the final result is verified. The network model Base contains feature pyramid (FPN) and Resnet50 structures. CFPN is to replace only the FPN network in Base, and FRN is to replace only the Resnet50 network in Base. CFPN+FRN is to replace the FPN and Resnet50 networks in Base. As shown in Table 5, the use of CFPN and FRN can improve the evaluation indicators on the dataset respectively, and the combination of CFPN and FRN can significantly improve all evaluation indicators on the dataset, that is, the Joint and Mesh indicators are reduced by 4.8% and 3.9%, respectively, and the F@5 and F@15 indicators are improved by 4.1% and 0.9%. In summary, the experimental results verify the effectiveness of the model in learning hand and object features under the joint action of CFPN and FRN networks.

[0202] Table 5 Effectiveness of residual network and cyclic feature pyramid network

[0203]

[0204] Note: Bold fonts indicate the best results.

[0205] The role of sampling convolution and single-head attention mechanism network: In order to verify the role of sampling convolution and single-head attention mechanism network in the hand-object feature fusion module on the hand-object feature fusion, the experiments carried out include: the initial network model (Initial) using only the multi-head attention mechanism (HAM), the Initial model with sampling convolution (SConv), the Initial model using only the single-head attention mechanism (SAM), and the method combining SConv+SAM, that is, the method of the present invention. The results are shown in Table 6. The method of the present invention reduces the Joint and Mesh indicators by 4.9% and 7.7%, respectively, and improves the F@5 and F@15 indicators by 6.7% and 0.6%, thereby verifying that under the joint effect of sampling convolution and single-head attention mechanism network, the hand-object feature fusion module achieves better fusion results.

[0206] Table 6 Effectiveness of sampling convolution and single-head attention mechanism network

[0207]

[0208] The role of sampling methods in feedforward networks: The effectiveness of sampling methods in feedforward networks is verified by designing different connection methods. Table 7 shows the experimental results when the sampling size is half the original size of the input image. Among them, the feedforward network in the basic network model adopts the structure of a multilayer perceptron (MLP). Based on MLP, the method of the present invention adopts two different connection methods, series and parallel, to combine the sampling network with MLP. It can be seen from Table 7 that the parallel sampling method has achieved the best effect on the data set, and has reduced the Joint and Mesh indicators by 5.8% and 5.8%, respectively, and increased the F@5 and F@15 indicators by 7.7% and 0.5%. The experiment shows that under the role of sampling methods in feedforward networks, the model can obtain more accurate results.

[0209] Table 7 Effectiveness of sampling methods in feedforward networks

[0210]

[0211] Note: Bold fonts indicate the best results.

[0212] Ablation experiments were conducted on the HO3D dataset. The experimental results show that the model proposed by the method of the present invention has an average joint error of 9.3, an average grid error of 9.3, and F-cores values ​​of 54.2 and 95.9. In order to verify the robustness of the model, a supplementary comparative experiment was conducted on the Dex-YCB dataset. The experimental results show that the method of the present invention has an average joint position error of 14.23 and an average joint position error after transformation of 5.74, and the above indicators are better than the existing mainstream methods.

[0213] In summary, the analysis of the results of the ablation experiment fully demonstrates the effectiveness of the method of the present invention and the importance of each module.

[0214] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A three-dimensional hand gesture estimation method integrating hand-object features, characterized in that: The method comprises the following steps: Step 1: Obtain several different hand-object images containing 3D postures of hands and objects, construct an image dataset, and then divide the training set from the image dataset according to a preset ratio; Step 2: Construct a joint hand occlusion estimation model that integrates hand-object features; Step 3: Input the training set into the hand occlusion estimation joint model that integrates hand-object features for training, and obtain a trained hand occlusion estimation joint model that integrates hand-object features; Step 4: Collect hand-object images containing the 3D postures of the hand and the object, and input them into the trained hand occlusion estimation joint model that integrates hand-object features to perform gesture estimation and obtain the 3D gesture estimation result.

2. According to claim 1, a three-dimensional hand gesture estimation method integrating hand-object features is characterized in that: The hand occlusion estimation joint model integrating hand-object features described in step 2 includes: a hand-object feature extraction module, a hand-object feature fusion module, a feature enhancement module and a regressor module; The hand-object feature extraction module is used to extract features of different scales from the input hand-object image, obtain a hand feature map and an object feature map, and transmit them to the hand-object feature fusion module; The hand-object feature fusion module is used to perform feature fusion on the hand feature map and the object feature map to obtain a hand-object feature map containing hand-object fusion features and transmit the map to the feature enhancement module; The feature enhancement module is used to enhance the hand-object feature map containing the hand-object fusion feature by using the self-attention mechanism and the multi-layer perceptron to obtain the final hand-object feature map and transmit it to the regressor module; The regressor module is used to generate three-dimensional hand posture and hand shape parameters according to the final feature map of the hand-object, and obtain three-dimensional hand joint coordinates by performing three-dimensional grid transformation on the three-dimensional hand posture and hand shape parameters, thereby obtaining a three-dimensional hand gesture estimation result.

3. According to claim 2, a three-dimensional hand gesture estimation method integrating hand-object features is characterized in that: The hand-object feature extraction module includes: a backbone network, a cyclic feature pyramid module integrating feature residuals, an average pooling layer, and a channel attention module connected in sequence; The average pooling layer uses residual connection; The backbone network is used to refine the features of the input hand-object image to obtain hand-object feature maps at different scales; The cyclic feature pyramid module with integrated feature residuals is used to extract features from hand-object feature maps at different scales to obtain hand-object image features; the cyclic feature pyramid module with integrated feature residuals is composed of a plurality of sets of serially connected feature residual networks and feature pyramid units, and for any set of serially connected feature residual networks and feature pyramid units, the output of the feature residual network is used as the input of the feature pyramid unit, wherein the feature residual network is used to split the hand-object feature maps at different scales into multiple subgroups along the channel dimension, and perform feature extraction and residual connection on each subgroup to obtain a residual network feature image; The feature pyramid unit is used to perform secondary feature extraction on hand-object feature maps at different scales by reversely transmitting the output of the feature residual network in the same group as the feature pyramid unit to the input of the feature residual network, so as to obtain hand-object image features; The average pooling layer is used to average pool the hand-object image features, and fuse the hand-object image features after average pooling with the hand-object image features before average pooling by using residual connection to obtain a fused hand-object feature map; The channel attention module is used to process the fused hand-object feature map using the channel attention mechanism to obtain a hand feature map and an object feature map.

4. According to claim 3, a three-dimensional hand gesture estimation method integrating hand-object features is characterized in that: The hand-object feature fusion module includes: an attention module and a projection module connected in sequence; wherein the projection module adopts a residual connection; The attention module is used to generate embedded representations of queries, keys and values ​​according to the hand feature map and the object feature map, and process the embedded representations of queries, keys and values ​​using a cross attention mechanism, and fuse the obtained output vector with the object feature map to obtain enhanced query features; The projection module is used to extract features and adjust the spatial resolution of the enhanced query features to obtain projection features, and fuse the projection features with the enhanced query features through residual connection to obtain a hand-object feature map containing hand-object fusion features; The projection module includes: a layer normalization layer, a lower projection layer, an activation function and an upper projection layer connected in sequence.

5. The three-dimensional hand gesture estimation method integrating hand-object features according to claim 4, characterized in that: The feature enhancement module includes, in sequence: three parallel 2D convolutional layers, a self-attention module and a perceptron module; wherein the perceptron module adopts a residual connection; The three parallel 2D convolutional layers are used to linearly project the input hand-object feature map containing the hand-object fusion feature to obtain three 2D convolution vectors as the embedded representations of the query, key and value in the feature enhancement module; The self-attention module is used to perform self-attention calculation on the embedded representations of the query, key and value in the feature enhancement module based on the multi-head attention mechanism, obtain the output vector of the multi-head attention mechanism and perform feature fusion with hand-object fusion to obtain the output feature of the self-attention module; The perceptron module is used to perform feature learning and feature extraction on the output features of the self-attention module, and perform feature addition on the features obtained by feature extraction and the features obtained by feature learning after scaling, so as to obtain the output features of the perceptron module; The perceptron module includes: layer normalization, a multi-layer perceptron, a lower perceptron, an activation function and an upper perceptron; wherein the layer normalization and the multi-layer perceptron are connected in series; the lower perceptron, the activation function and the upper perceptron are connected in series; and the layer normalization and the multi-layer perceptron connected in series are connected in parallel with the lower perceptron, the activation function and the upper perceptron connected in series, and the output of the upper perceptron is scale-transformed and feature-added with the output of the multi-layer perceptron to obtain the output feature of the perceptron module; the output feature of the perceptron module is fused with the output feature of the self-attention module by using a residual connection to obtain a final feature map of the hand-object.

6. The method for estimating a three-dimensional hand gesture by integrating hand-object features according to claim 5, characterized in that: The step 3 further comprises: Step 3.1: Set the initial learning rate, decay factor, batch size, and maximum iteration rounds, and start iterative training; Step 3.2: Input the hand-object images in the training set into the joint hand occlusion estimation model that integrates hand-object features, and use the backbone network to refine the features of the hand-object images to obtain hand-object feature maps at different scales; Step 3.3: Use the cyclic feature pyramid module with integrated feature residuals to extract features of hand-object feature maps at different scales to obtain hand-object image features P2; Step 3.4: Perform average pooling on the hand-object image feature P2, and use residual connection to fuse the hand-object image feature after average pooling with the hand-object image feature P2 to obtain a fused hand-object feature map; Step 3.5: Use the channel attention mechanism to process the fused hand-object feature map to obtain the hand feature map F hand and object feature map F object ; Step 3.6: Get the hand feature map F hand The hand features and object features in F object The object features in the hand feature map are used as the query, and the intersection area of ​​the hand feature map and the object feature map are used as the key and value to generate an embedding representation of the query, key and value; Step 3.7: Use the cross-attention mechanism to process the embedding representations of the query, key, and value to obtain the output vector, and then compare the output vector with the object feature map F object Fusion is performed to obtain enhanced query features Q τ ; Step 3.8: Use the projection module to enhance the query feature Q τ Projection is performed to obtain the projection feature H, and the projection feature H is combined with the enhanced query feature Q τ Fusion is performed to obtain the hand-object fusion feature Step 3.9: Fusion of opponent features Perform linear projection and use the three 2D convolution vectors obtained as the embedding representations of the query, key and value in the feature enhancement module. Perform self-attention calculation on the embedding representations of the query, key and value in the feature enhancement module based on the multi-head attention mechanism to obtain the output vector of the multi-head attention mechanism and combine it with the hand-object fusion feature. Merge to obtain the output features of the self-attention module; Step 3.10: The output features of the self-attention module are divided into two paths, one of which uses the series-connected layer normalization and multi-layer perceptron for feature learning to obtain the output of the multi-layer perceptron; the other path uses the series-connected lower perceptron layer, activation function and upper perceptron layer for feature extraction to obtain the output of the upper perceptron layer; the output of the upper perceptron layer is scaled and then added to the output of the multi-layer perceptron to obtain the output features of the perceptron module; Step 3.11: Fuse the output features of the perceptron module with the output features of the self-attention module to obtain the final features of the hand and object and generate the final feature map of the hand and object; Step 3.12: Based on the final feature map of the hand and object, a single hourglass network is used to generate a two-dimensional heat map of each joint of the hand. Several residual blocks connected in series are used to perform feature fusion on the two-dimensional heat map of each joint of the hand, and the output of the residual block is vectorized to obtain a multi-dimensional feature vector. Step 3.13: Generate three-dimensional hand posture and hand shape parameters according to the multidimensional feature vector, and perform three-dimensional mesh transformation according to the three-dimensional hand posture and hand shape parameters by constructing a regression matrix of the hand joints to generate three-dimensional hand joint coordinates, and then obtain a three-dimensional hand gesture estimation result according to the three-dimensional hand joint coordinates; Step 3.14: Determine whether the current iteration round has reached the maximum iteration round. If so, end the training and obtain the trained hand occlusion estimation joint model that integrates hand-object features; otherwise, return to step 3.

2.

7. The method for estimating a three-dimensional hand gesture by integrating hand-object features according to claim 6, characterized in that: The method of step 3.3 is specifically as follows: for any feature residual network in the cyclic feature pyramid module of integrated feature residual, the hand-object feature map U is input into the feature residual network, where U∈R H×W×C , H, W and C are the height, width and number of channels of the image respectively; the hand-object feature map U is subjected to two convolutions and two feature convolutions to generate two sampling results X α and X β , X α and X β Multiply and add to X β Add together to get the output X of a feature residual network i ; X i =X α ×X β +X β Where P i (·) is the upsampling function; δ(·) is the activation function; APool(U) represents the average pooling of the object feature map U; is the attention convolution channel f attn (·) output, and The output of the attention channel And the structure of the attention convolution layer μ attn As a parameter; Then, the feature pyramid unit of the same group as the feature residual network is used to reversely transmit the output of the feature residual network to the input of the feature residual network, perform secondary feature extraction on the hand object feature map, and obtain the hand object image features.

8. The method for estimating a three-dimensional hand gesture by integrating hand-object features according to claim 7, characterized in that: The hand feature map F described in step 3.5 hand and object feature map F object It is expressed as: F object =CAM object ×[Up(APool(P2))+P2] F hand =CAM hand ×[Up(APool(P2))+P2] Where Up is the upsampling function; APool(·) is the average pooling layer; CAM object and CAM hand Represent the channel attention modules for objects and hands, respectively.

9. The method for estimating a three-dimensional hand gesture by integrating hand-object features according to claim 8, characterized in that: The hand-object fusion features described in step 3.8 The calculation method is: E=Conv(Q τ ,W1)+b1 H=ConvTranspose(ReLU(E),W2)+b2 Where E is the output of the lower projection layer in the projection module; Conv(·) represents convolution; ConvTranspose(·) represents bias convolution; W1 and W2 represent the convolution kernel of the lower projection layer and the convolution kernel of the upper projection layer respectively; b1 and b2 represent the bias of the lower projection layer and the bias of the upper projection layer respectively; ReLU(·) is the linear unit activation function; Where H(Q τ ) represents the enhanced query feature Q τ Projection.

10. The method for estimating three-dimensional hand gestures by integrating hand-object features according to claim 9, characterized in that: The output of the upper perception layer described in step 3.10 is expressed as: in represents the output of the upper perception layer; ReLU(·) represents the ReLU activation function; LN(·) represents layer normalization; X θ Represents the output features of the self-attention module; W down is the parameter of the lower perception layer, and W up is the parameter of the upper perception layer, and where n and Represent the initial dimension and the intermediate dimension respectively, and satisfy The output feature of the perceptron module is expressed as: Where X ρ represents the output features of the perceptron module; S is the scale.

Images