A three-dimensional hand pose estimation method fusing hand and object features

Abstract

The application provides a three-dimensional hand gesture estimation method fusing hand and object features, relates to the technical field of hand gesture estimation, and constructs a three-dimensional hand gesture estimation network fusing hand and object features, which comprises a feature extraction module, a double-flow hand feature pyramid subnetwork, a hand feature dynamic adjustment module, a double-flow attention collaborative feature enhancement module and a decoder. A loss function of the three-dimensional hand gesture estimation network fusing hand and object features is established, the three-dimensional hand gesture estimation network fusing hand and object features is trained, and a hand gesture estimation result is obtained based on the trained three-dimensional hand gesture estimation network fusing hand and object features. The application captures local details and global semantic dependencies through bidirectional cross-scale information aggregation, alleviates the channel information loss problem of a traditional feature pyramid, combines hand-object geometric constraints and semantic complementary characteristics, enhances the cross-modal feature alignment capability, and realizes the deep fusion of hand-object geometric constraints and semantic complementarity.

Description

Technical Field

[0001] This invention belongs to the field of hand pose estimation technology, and particularly relates to a three-dimensional hand pose estimation method that integrates hand features. Background Technology

[0002] Gesture pose estimation, as an important research direction in computer vision, has wide application value in scenarios such as virtual reality, augmented reality, and robot learning. However, due to the high dexterity of the hand, complex self-occlusion, and occlusion problems during hand-object interaction, this task still faces many challenges. In recent years, gesture estimation methods based on monocular RGB images have made significant progress, especially gesture pose estimation in hand-object interaction scenarios, which has become a current research hotspot.

[0003] Park et al. proposed a network called HandOcc-Net, which employs a feature pyramid and spatial attention mechanism, combined with a feature injection module to handle occlusion issues. HandOccNet enhances the feature representation ability of occluded regions through the feature injection mechanism. This method designs two modules: FIT (Feature Injecting Transformer) and SET (Self-Enhancing Transformer). It uses an attention mechanism to learn the mapping relationship between occluded and unoccluded regions and optimizes the features of occluded regions, significantly improving the accuracy of 3D hand mesh estimation, especially on complex interactive datasets. Lin et al. proposed HFL-Net, which combines single-stream and dual-stream backbone networks to achieve harmonious hand-object feature learning and enhances the interaction between hand and object features through a self-attention mechanism, thereby improving the accuracy of hand pose recognition.

[0004] Although both HandOccNet and HFL-Net combine FPN and Transformer methods, neither can reproduce hand details effectively in cases of severe occlusion. When HandOccNet uses a single-stream backbone network to extract features from both the hand and object simultaneously, a "competition" occurs between these features. While HFL-Net uses a two-stream backbone network, learning hand and object features together in layers one, two, and five, and independently in layers three and four, its hand and object feature extraction network is still the traditional FPN. Its simple convolutional channel reduction leads to the loss of channel information. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this paper combines the methods of FPN and Transformer, and adopts a two-stream network structure for hand and object feature extraction to construct a three-dimensional hand pose estimation network that integrates hand and object features. This improves the effectiveness of hand extraction information and greatly enhances the accuracy of hand pose estimation.

[0006] This invention provides a three-dimensional hand pose estimation method that integrates hand features, comprising the following steps:

[0007] A 3D hand pose estimation network integrating hand and object features is constructed, including a feature extraction module, a two-stream hand feature pyramid sub-network, a hand feature dynamic adjustment module, a two-stream attention collaborative feature enhancement module, and a decoder;

[0008] Establish a loss function for a 3D hand pose estimation network that integrates hand and object features, train the 3D hand pose estimation network that integrates hand and object features, and obtain the trained 3D hand pose estimation network that integrates hand and object features.

[0009] The hand gesture image is acquired, and the hand gesture estimation result is obtained based on the trained 3D hand gesture estimation network that integrates hand and object features.

[0010] Furthermore, the feature extraction module includes several stages for extracting feature maps at different levels, and selecting feature maps at multiple levels as input feature maps for the dual-stream hand feature pyramid sub-network.

[0011] Furthermore, the dual-stream hand feature pyramid sub-network includes two structurally identical hand feature pyramid modules, which extract hand feature maps and object feature maps respectively.

[0012] Furthermore, the specific methods for extracting hand feature maps and object feature maps are as follows:

[0013] The feature maps of different levels in the input dual-stream hand feature pyramid subnetwork are weighted and summed. The importance of the feature maps at different levels is adaptively adjusted through learnable weights to obtain the first feature map. ;

[0014] An enhanced weighted fusion unit mapping block is constructed, and a hand feature pyramid module is constructed by inserting the enhanced weighted fusion unit mapping block into the lateral connection path of the bidirectional feature pyramid network BiFPN.

[0015] The first feature is processed using an enhanced weighted fusion unit, the Mapping block. Spatial enhancement and channel adjustment are performed to obtain the second feature. Based on the first feature Second feature Obtain the intermediate feature map;

[0016] Two hand feature pyramid modules fuse intermediate feature maps output by enhanced weighted fusion units (Mapping blocks) at different levels through bidirectional paths from top to bottom and bottom to top, resulting in hand feature maps and object feature maps, respectively.

[0017] Furthermore, the enhanced weighted fusion unit (Mapping block) includes: a first 1×1 convolution, a 3×3 convolution, a second 1×1 convolution, and a SiLU activation function; the first 1×1 convolution, the 3×3 convolution, and the second 1×1 convolution are used sequentially to perform spatial enhancement and channel adjustment on the features input to the enhanced weighted fusion unit (Mapping block); the output features of the first 1×1 convolution are concatenated with the output features of the second 1×1 convolution; and the intermediate feature map output by the enhanced weighted fusion unit (Mapping block) is obtained through the SiLU activation function.

[0018] The first feature is processed using an enhanced weighted fusion unit, the Mapping block. Spatial enhancement and channel modulation are performed, specifically through the following methods:

[0019] The first feature was processed sequentially using 1×1 convolution, 3×3 convolution, and 1×1 convolution. Perform spatial enhancement and channel adjustment to enhance the primary feature. Spatial representation capability, resulting in the second feature. ;

[0020] The second feature and the first feature after 1×1 convolution processing The intermediate feature maps are concatenated and obtained by using the SiLU activation function to obtain the output of the enhanced weighted fusion unit Mapping block.

[0021] Furthermore, the hand feature dynamic adjustment module includes a local feature extraction layer and a cross-sample global attention mechanism based on external memory, used to adjust the hand feature map and generate a global hand feature map. The specific method is as follows:

[0022] For the hand feature map input to the local feature extraction layer, local details are extracted through convolution operations, and the channel dimensions are adjusted to obtain local hand features. ;

[0023] Establish a cross-sample global attention mechanism based on external memory, using a learnable key-value memory matrix. Dynamically aggregating common features across samples, the specific method is as follows:

[0024] Set globally shared learnable parameters, including the key-memory matrix. Sum value memory matrix ,in, The number of memory units, Input the number of channels. The number of output channels is denoted by ; the key memory matrix is ​​used to encode general key features of hand pose, with each memory unit corresponding to a potential hand structure prior; the value memory matrix is ​​used to store the semantic response corresponding to the key features for dynamically adjusting feature importance.

[0025] Local hand features Expand as Where N = H × W, and B is the batch size, i.e., the number of feature maps processed in a batch, and the key-memory matrix is ​​calculated. Attention weights ;

[0026] Adjust attention weights using layer normalization The dimensional order is used to generate normalized attention weights. ;

[0027] The normalized attention weight A and the value memory matrix are used. Weighted summation is used to generate a global hand feature map. .

[0028] Furthermore, the dual-stream attention collaborative feature enhancement module is used to enhance the features of the global hand feature map, hand feature map and object feature map, extract fine-grained features of the gesture posture image, and obtain the feature-enhanced hand feature map.

[0029] Furthermore, the specific method for extracting fine-grained features from the gesture image to obtain the enhanced hand feature map is as follows:

[0030] Feature encoding and serialization are performed on the hand feature map and the object feature map. The number of channels in the hand feature map is adjusted by 1×1 convolution to generate the query matrix Q. The number of channels in the object feature map is adjusted by 1×1 convolution to generate the key matrix K and the value matrix V.

[0031] Hand feature diagram and object feature map Multi-head attention mechanism computation is performed, dividing the query matrix Q, key matrix K, and value matrix V into h groups, calculating the single-head attention for each group, concatenating the h single-head attentions, and then integrating them with global hand features after linear projection. Adding elements together yields the chiral-object cooperative features. ;

[0032] Collaborative features of adversaries A two-branch refinement and feature enhancement process is performed. The first branch is residual preservation, which is used to retain the initial fused features and maintain information integrity, resulting in the output of the first branch. ;

[0033] The second branch includes layer normalization and a multilayer perceptron (MLP) to stabilize the training process, alleviate the gradient explosion problem, and refine features through nonlinear transformations to obtain the output of the second branch. ;

[0034] The output of the first branch The output of the second branch The features are added together to generate a hand feature map with enhanced features.

[0035] Furthermore, the decoder performs hand pose estimation based on the enhanced hand feature map to obtain the hand pose estimation result.

[0036] Furthermore, a total loss function for a 3D hand pose estimation network that integrates hand and object features is established, including hand pose loss and object pose loss, as shown in the following formula:

[0037]

[0038] in, For loss of hand posture, Object pose loss, The collaborative weighting coefficient controls the intensity of the influence of object information on hand optimization;

[0039] Hand pose loss includes 2D joint detection loss. 3D vertex and joint loss and MANO parameter loss As shown in the formula below:

[0040]

[0041] in, , , To balance the coefficients of each loss function weight;

[0042] 2D joint loss The following formula is used to supervise the coordinate prediction of 2D joints in the hand:

[0043]

[0044] Where B is the batch size. This refers to the number of joints in the hand. The predicted 2D coordinates of the i-th joint in the b-th sample. Let be the true 2D coordinates of the i-th joint in the b-th sample;

[0045] 3D Vertex and Joint Loss As shown in the formula below:

[0046]

[0047] Where V is the number of 3D vertices of the hand as defined in the MANO parameter loss. Let be the predicted 3D coordinates of the j-th vertex in the b-th sample. Let J be the true 3D coordinates of the j-th vertex in the b-th sample. The predicted 3D coordinates of the j-th joint in the b-th sample. Let J be the true 3D coordinates of the j-th joint in the b-th sample;

[0048] The MANO parameter loss is shown in the following formula:

[0049]

[0050] in, Let b be the predicted shape parameters for the b-th sample. The true shape parameters are for the b-th sample. Let b be the pose parameters predicted for the b-th sample. The pre-true pose parameters in the b-th sample;

[0051] Object pose coordination loss includes 2D projection loss. and confidence loss ;

[0052] 2D projection loss As shown in the formula below:

[0053]

[0054] in, This represents the number of corner points of the object's 3D bounding box. Let k be the predicted 2D projection coordinates of the k-th corner point. The true 2D projected coordinates of the k-th corner point;

[0055] Confidence loss As shown in the formula below:

[0056]

[0057] in, A confidence level exists for the predicted object. These are true binary labels, where 0 indicates invisible and 1 indicates visible.

[0058] The beneficial effects of adopting the above technical solution are as follows: The 3D hand pose estimation method fused with hand and object features provided by this invention significantly improves the accuracy of hand pose estimation in complex interactive scenarios through multi-level feature interaction and cross-modal attention mechanisms. Addressing the shortcomings of existing methods in multi-scale feature fusion, occlusion robustness, and hand-object feature interference, a dual-stream hand feature pyramid sub-network is designed. Through dynamic scale perception and cross-level information aggregation, the feature degradation problem of traditional pyramid structures is effectively alleviated. A dynamic adjustment mechanism based on external memory is introduced into the hand feature dynamic adjustment module to adaptively enhance hand semantic expression in occluded areas and suppress background noise interference. Furthermore, a dual-stream attention collaborative feature enhancement module is constructed to achieve deep fusion of hand-object geometric constraints and semantic complementarity through a bidirectional feature guidance strategy. Attached Figure Description

[0059] Figure 1 The three-dimensional hand pose estimation network structure diagram that integrates hand-object features provided in Embodiment 1 of the present invention is shown below;

[0060] Figure 2 The dual-flow hand feature pyramid sub-network structure diagram provided in Embodiment 1 of the present invention;

[0061] Figure 3 Structural diagram of the hand feature dynamic adjustment module provided in Embodiment 1 of the present invention;

[0062] Figure 4 Structure diagram of the dual-stream attention collaborative feature enhancement module provided in Embodiment 1 of the present invention;

[0063] Figure 5 The visualization results of hand pose estimation on the Dex-YCB dataset using different methods provided in Embodiment 1 of this invention;

[0064] Figure 6 The visualization results of hand pose estimation on the HO3D dataset using different methods provided in Embodiment 1 of the present invention;

[0065] Figure 7 A comparison chart of MPJPE and PAMPJPE metrics for the Dex-YCB dataset provided in Embodiment 1 of this invention;

[0066] Figure 8 A comparison chart of metrics between Joint and Mesh in the HO3D dataset provided in Embodiment 1 of this invention;

[0067] Figure 9 A comparison chart of the metrics of HO3D dataset F@5 and F@15 provided in Embodiment 1 of this invention. Detailed Implementation

[0068] The specific implementation methods of this application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0069] Example 1

[0070] A three-dimensional hand pose estimation method incorporating hand features includes the following steps:

[0071] Construct a 3D hand pose estimation network that integrates hand-object features, such as... Figure 1 As shown, it includes a feature extraction module, a two-stream hand feature pyramid sub-network, a hand feature dynamic adjustment module, a two-stream attention collaborative feature enhancement module, and a decoder;

[0072] Establish the total loss function of the 3D hand pose estimation network that integrates hand and object features, train the 3D hand pose estimation network that integrates hand and object features, and obtain the trained 3D hand pose estimation network that integrates hand and object features.

[0073] The hand gesture image is acquired, and the hand gesture estimation result is obtained based on the trained 3D hand gesture estimation network that integrates hand and object features.

[0074] The feature extraction module includes several stages for extracting feature maps at different levels, and selects feature maps at multiple levels as input feature maps for the dual-stream hand feature pyramid sub-network.

[0075] In this embodiment, the feature extraction module includes five stages, namely Stage 1 to Stage 5. The feature map output by Stage 1 has a resolution of 1 / 4 of the input image and 64 channels. The feature map output by Stage 1 retains the texture, edge, and color information of the underlying layer of the input image.

[0076] The feature map output by Stage2 has a resolution of 1 / 4 of the input image and 256 channels. The feature map output by Stage2 is a low-level semantic feature map, which preserves the detailed information of the input image.

[0077] The feature map output by Stage3 has a resolution of 1 / 8 of the input image and 512 channels. The feature map output by Stage3 is a mid-level semantic feature map, which preserves the complex local structural information of the input image.

[0078] The feature map output by Stage4 has a resolution of 1 / 16 of the input image and 1024 channels. The feature map output by Stage4 is a high-level semantic feature map with rich semantic information.

[0079] The feature map output by Stage5 has a resolution of 1 / 32 of the input image and 2048 channels. The feature map output by Stage5 is a global semantic feature map, which has the strongest semantic information.

[0080] Select the feature maps output from Stage 2 to Stage 5 or the feature maps at different levels output from Stage 3 to Stage 5 as the input feature maps for the dual-stream hand feature pyramid sub-network;

[0081] Dual-stream hand feature pyramid subnetwork DS-FPN, such as Figure 2 As shown, it includes two structurally identical hand feature pyramid modules, which extract hand feature maps respectively. and object feature map The specific method is as follows:

[0082] The feature maps of different levels in the input dual-stream hand feature pyramid subnetwork are weighted and summed. The importance of the feature maps at different levels is adaptively adjusted through learnable weights to obtain the first feature map. As shown in the formula below:

[0083]

[0084] in, , For learnable weights, For learning rate, This is the upsampling result for high-level features. This refers to the lateral connectivity features of the current layer; in this embodiment, the learnable weights... , Initialize to 1, set the learning rate. ;

[0085] To overcome the problem of insufficient spatial information modeling in the feature fusion of the traditional Bi-FPN feature pyramid network, this embodiment constructs an enhanced weighted fusion unit, Mapping block, which includes: a first 1×1 convolution, a 3×3 convolution, a second 1×1 convolution, and a SiLU activation function;

[0086] The first feature was processed sequentially using 1×1 convolution, 3×3 convolution, and 1×1 convolution. Perform spatial enhancement and channel adjustment to enhance the primary feature. Spatial representation capability, resulting in the second feature. As shown in the following formula

[0087]

[0088] in, For 1×1 convolution, It is a 3×3 convolution;

[0089] 1×1 convolution is used to compress the first feature. According to the low-rank theory of matrices, high-dimensional feature matrices often contain information redundancy. Dimensionality reduction can approximately preserve the main feature patterns. Therefore, the first feature... The number of channels is compressed to 1 / 4. By performing efficient computation in the low-dimensional space, the parameter explosion caused by directly processing high-dimensional features is avoided, significantly reducing computational redundancy. 3×3 convolutions are used to extract local spatial features such as hand joint bending and fingertip direction in the reduced-dimensional space. The local receptive field of the 3×3 convolution kernel can capture the spatial correlation between adjacent pixels. 1×1 convolutions restore the dimensionality by adjusting the number of feature channels to the target dimension, maintaining multi-scale compatibility. Sequentially using 1×1, 3×3, and 1×1 convolutions, a non-linear mapping is established between low-dimensional and high-dimensional spaces through linear projection, enhancing feature expressive power. The number of channels is restored to the target dimension, ensuring multi-scale feature compatibility. The dimensionality-up operation reconstructs a high-dimensional representation by linearly combining low-dimensional features, providing feature maps adapted to different resolution tasks for subsequent layers.

[0090] The second feature and the first feature after 1×1 convolution processing The intermediate feature maps are concatenated and then processed through the SiLU activation function to obtain the output of the enhanced weighted fusion unit, the Mapping block. The SiLU activation function is shown in the following formula:

[0091]

[0092] in, Let x be the sigmoid function, and x be the second characteristic. and the first feature after 1×1 convolution processing Features after splicing;

[0093] Compared to the traditional ReLU activation function, introducing the Sigmoid function makes the gradient non-zero in the negative interval, alleviating the neuron death problem of the ReLU activation function and improving the stability of model training. Sigmoid adaptive weighting uses the Sigmoid function to weight the feature map pixel by pixel, enhancing key regions, such as the response intensity of the fingertip, and suppressing low-confidence regions.

[0094] An enhanced weighted fusion unit (Mapping block) is inserted into the lateral connection path of the Bidirectional Feature Pyramid Network (BiFPN). By fusing intermediate feature maps output from different levels of the Mapping block via top-down and bottom-up bidirectional paths, the hand feature map is obtained. and object feature map Traditional Feature Pyramid Networks (FPNs) fuse multi-scale features via a unidirectional top-down path, which can enhance low-level semantic information. However, their unidirectional transmission mechanism leads to severe loss of hand details (such as fingertips and joint textures) during multiple upsampling processes, and they lack robustness against complex occlusion scenarios. In this embodiment, referencing the traditional Bidirectional Feature Pyramid Network (BiFPN), a dual-stream hand feature pyramid network (DS-FPN) is constructed by fusing multi-scale features via both top-down and bottom-up bidirectional paths. Employing a bidirectional multi-scale feature fusion mechanism, DS-FPN effectively alleviates the problem of missing local information through cross-level feature interaction, resulting in better accuracy and efficiency in extracting hand feature information. It also demonstrates significant effectiveness in extracting object features to aid hand reconstruction.

[0095] Compared to the single fusion path of BiFPN, the dual-stream hand feature pyramid sub-network DS-FPN achieves two cross-layer interactions of features. First, the bottom-up flow focuses on extracting and enhancing local detail features; second, the top-down flow effectively injects high-level semantics and global pose information; finally, the second bottom-up flow deeply fuses the detail and global information obtained in the first two steps, forming an optimized path of "details → global → details + global". This embodiment adopts a dual-path structure of "bottom-up → top-down → bottom-up again" and a channel-space dual-stream interaction mechanism to achieve iterative optimization and dynamic enhancement of hand features.

[0096] The Dynamic Hand Feature Adjustment (DAM) module is used to adjust the hand feature map and generate a global hand feature map, such as... Figure 3 As shown, it includes a local feature extraction layer and a cross-sample global attention mechanism based on external memory;

[0097] For the hand feature map of the input local feature extraction layer Local details are extracted through convolution operations, and local hand features are obtained by adjusting the channel dimensions. As shown in the formula below:

[0098]

[0099] in, The weights are 3×3 convolution kernel weights, and * represents the convolution operation. The bias term is ReLU, and ReLU is the ReLU activation function. The number of channels for the output features;

[0100] To model the global dependencies of hand features and reduce computational overhead, this implementation establishes a cross-sample global attention mechanism based on external memory, using a learnable key-value memory matrix. Dynamically aggregating common features across samples, the specific method is as follows:

[0101] Set globally shared learnable parameters, including the key-memory matrix. Sum value memory matrix ,in, The number of memory units, Input the number of channels. The number of output channels is denoted by ; the key memory matrix is ​​used to encode general key features of hand pose, with each memory unit corresponding to a potential hand structure prior; the value memory matrix is ​​used to store the semantic response corresponding to the key features for dynamically adjusting feature importance.

[0102] Local hand features Expand as Where N = H × W, and B is the batch size, i.e., the number of feature maps processed in a batch, and the key-memory matrix is ​​calculated. Attention weights As shown in the formula below:

[0103]

[0104] in, Local features Transpose of;

[0105] Adjust attention weights using layer normalization The dimensional order is used to generate normalized attention weights. As shown in the formula below:

[0106]

[0107] Where LayerNorm is the layer normalization;

[0108] The normalized attention weight A and the value memory matrix are used. Weighted summation is used to generate a global hand feature map. As shown in the formula below:

[0109]

[0110] This process implicitly encodes common hand pose characteristics across samples, such as joint bending patterns, through an external memory matrix, while achieving efficient computation with linear complexity.

[0111] Two-stream attention collaborative feature enhancement module (HFE), such as Figure 4 As shown, this is used to analyze the global hand feature map and the hand feature map. and object feature map Feature enhancement is performed to extract fine-grained features from the gesture image, resulting in a feature-enhanced hand feature map.

[0112] In complex interaction scenarios, such as hand grasping or occlusion interference, traditional hand feature extraction methods often struggle to simultaneously capture local details and global semantic relationships. To address this challenge, this embodiment constructs a hand feature enhancement module (HFE). Through a collaborative design of cross-modal attention guidance and multi-granularity feature fusion, efficient feature optimization is achieved. The HFE module dynamically adjusts the global hand features output by the module. Based on this foundation, object features are introduced as a cross-modal contextual guide source. A multi-head attention mechanism is used to establish a hand-object spatial association, enabling feature responses to adaptively focus on key interaction regions (such as tool grip points and occlusion boundaries). To further balance the robustness and discriminativeness of feature representations, the module adopts a dual-path fusion strategy: on the one hand, it preserves the original feature details to maintain the integrity of low-level information; on the other hand, it refines high-order semantic features through layer normalization and multilayer perceptrons to suppress background noise interference.

[0113] Hand feature diagram and object feature map Feature encoding and serialization are performed, and the number of channels is adjusted through 1×1 convolution to generate the query matrix Q, the key matrix K, and the value matrix V, as shown in the following formula:

[0114]

[0115]

[0116]

[0117] Hand feature diagram and object feature map To perform multi-head attention mechanism computation, the query matrix Q, key matrix K, and value matrix V are divided into h groups, and the single-head attention for each group is calculated. The i-th single-head attention... As shown in the formula below:

[0118]

[0119] in, To obtain the i-th query submatrix by dividing the query matrix Q into h groups, To obtain the i-th key submatrix by dividing the key matrix K into h groups, Let C be the i-th value submatrix obtained by dividing the value matrix V into h groups, and let C be the number of channels in the feature map Q. Channel dimensions for each group;

[0120] The h single-head attentions are concatenated, linearly projected, and then combined with the global hand features. Adding elements together yields the chiral-object cooperative features. As shown in the formula below:

[0121]

[0122]

[0123] Collaborative features of adversaries A two-branch refinement and feature enhancement process is performed. The first branch is residual preservation, which is used to retain the initial fused features and maintain information integrity, resulting in the output of the first branch. As shown in the formula below:

[0124]

[0125] The second branch includes layer normalization and a multilayer perceptron (MLP) to stabilize the training process, alleviate the gradient explosion problem, and refine features through nonlinear transformations to obtain the output of the second branch. As shown in the formula below:

[0126]

[0127]

[0128] in, The chiral-object cooperative features are those obtained after nonlinear transformation. This is a multilayer perceptron, which consists of two linear layers and a GELU activation function, as shown in the following formula:

[0129]

[0130] in, Expand weights for features. For feature compression weights, and All are bias terms;

[0131] Feature extension will transform the chiral-object co-features after nonlinear transformation The dimension is expanded from C to 4C, which enhances the model capacity and captures complex nonlinear relationships, such as the dynamic relationship between hand joints; feature compression restores the dimension to C, preventing parameter explosion and filtering redundant information.

[0132] The output of the first branch The output of the second branch Add them together to generate a hand feature map with enhanced features. As shown in the formula below:

[0133]

[0134] The Hand Feature Enhancement (HFE) module employs a two-stage adaptive fusion approach: The first stage injects object semantics (such as tool shape and grip direction) into hand features through cross-modal attention, enhancing the response intensity of interactive areas (such as fingertips and palm); the second stage balances feature preservation and refinement based on a dual-branch structure, with the first branch preventing information loss and the second branch suppressing noise and enhancing discriminativity. Ultimately, this results in a richer and more accurate optimized hand feature map that incorporates object information.

[0135] The decoder is based on the enhanced hand feature map. Hand pose estimation is performed to obtain the hand pose estimation results.

[0136] A loss function for a 3D hand pose estimation network fusing hand and object features is established. This network is then trained to obtain a well-trained 3D hand pose estimation network. The core objective of this embodiment is hand pose estimation, and object pose estimation is introduced as a cooperative constraint to enhance the spatial localization capability of hand joints through cross-modal feature interaction. Therefore, the loss function also needs to design an object cooperative loss. The total loss function of the 3D hand pose estimation network fusing hand and object features includes hand pose loss and object pose loss, as shown in the following formula:

[0137]

[0138] in, For loss of hand posture, Object pose loss, The collaborative weighting coefficient controls the intensity of the influence of object information on hand optimization;

[0139] The overall loss function of the 3D hand pose estimation network that integrates hand and object features includes hand pose loss and object pose loss. The hand pose loss directly optimizes the hand pose accuracy by calculating joint angle deviations and spatial position errors, while the object pose loss models the interaction rationality based on the geometric consistency of the hand-object contact area. The two are weighted and fused to form a joint optimization objective. While strengthening the learning of hand features, the network also uses object pose information to constrain the physical feasibility of hand movements, thereby effectively improving the robustness and spatial consistency of pose estimation in occluded scenes.

[0140] Hand posture loss Including 2D keypoint detection loss 3D vertex and joint loss and MANO parameter loss As shown in the formula below:

[0141]

[0142] in, , , To balance the coefficients of each loss function weight;

[0143] 2D joint loss To supervise the coordinate prediction of 2D hand joints, the mean squared error (MSE) is used. For each joint in each batch, the L2 loss between the predicted coordinates and the true coordinates is calculated and averaged, as shown in the following formula:

[0144]

[0145] in, This refers to the number of joints in the hand. The predicted 2D coordinates of the i-th joint in the b-th sample. The coordinates are the true 2D coordinates of the i-th joint in the b-th sample; in this embodiment, ;

[0146] 3D Vertex and Joint Loss It is a weighted sum of vertex L2 loss and joint L2 loss after normalization according to their respective quantities. The vertex L2 loss calculates the difference between the predicted and actual values ​​of all 3D vertices, and the joint L2 loss calculates the difference between the predicted and actual values ​​of all 3D joints, as shown in the following formula:

[0147]

[0148] Where V is the number of 3D vertices of the hand as defined in the MANO parameter loss. Let be the predicted 3D coordinates of the j-th vertex in the b-th sample. Let J be the true 3D coordinates of the j-th vertex in the b-th sample. The predicted 3D coordinates of the j-th joint in the b-th sample. Let J be the true 3D coordinates of the j-th joint in the b-th sample;

[0149] The MANO parameter loss is calculated by using the L2 loss for both the shape parameter β and the attitude parameter θ, as shown in the following formula:

[0150]

[0151] in, Let b be the predicted shape parameters for the b-th sample. The true shape parameters are for the b-th sample. Let b be the pose parameters predicted for the b-th sample. The pre-true pose parameters in the b-th sample;

[0152] Object pose coordination loss includes 2D projection loss. and confidence loss As shown in the formula below:

[0153]

[0154] in, , To balance the coefficients of each loss function weight;

[0155] 2D projection loss The 2D projection positions of key points of the constrained object are determined using the mean absolute error (MAE), as shown in the following formula:

[0156]

[0157] in, This represents the number of corner points of the object's 3D bounding box. Let k be the predicted 2D projection coordinates of the k-th corner point. The coordinates of the k-th corner point are the true 2D projection coordinates; in this embodiment, ;

[0158] Confidence loss The confidence score used to monitor the existence of an object is shown in the following formula:

[0159]

[0160] in, A confidence level exists for the predicted object. These are true binary labels, where 0 indicates invisible and 1 indicates visible.

[0161] In this embodiment, two publicly available datasets, Dex-YCB and HO3D, are used to validate hand pose estimation. The Dex-YCB dataset focuses on fine-grained hand pose estimation and dynamic hand-object interaction research. It contains dynamic grasping sequences from 10 classes of YCB objects, covering complete interactive actions such as grasping, translation, rotation, and release. The training set contains approximately 500,000 frames, and the test set contains 5,000 frames, covering diverse combinations of operator and object instances. All data in the Dex-YCB dataset were acquired using a high-precision optical motion capture system (such as Vicon) and multiple synchronized RGB-D cameras (such as Intel RealSense) to ensure millimeter-level accuracy of hand and object poses. It also provides multi-view images and depth information to enhance the robustness of 3D reconstruction. The annotation information is generated based on the MANO parametric hand model, containing 45-dimensional pose parameters and 10-dimensional shape parameters. The 6D pose of the object is accurately annotated using rigid body registration technology, supporting joint analysis of hand-object cooperative poses. The official strategy adopts a hierarchical partitioning approach of S0-S2: S0 is partitioned by object instance to verify the model's generalization to unseen objects, S1 is partitioned by operator to evaluate cross-user adaptability, and S2 is a hybrid partitioning for comprehensive performance evaluation. Researchers are required to submit results through a designated platform to ensure evaluation consistency.

[0162] The Dex-YCB dataset provides two metrics: MPJPE and PA-MPJPE. MPJPE directly calculates the misaligned global joint position error to reflect absolute spatial accuracy, while PA-MPJPE eliminates global pose offset through Procrustes alignment and focuses on evaluating the geometric consistency of local hand poses. The combination of the two metrics can comprehensively measure the practical performance of the model in real-world scenarios.

[0163] The HO3D dataset focuses on complex hand-object interaction 3D perception benchmark datasets, aiming to promote research on joint estimation of hand pose and object pose in dynamic scenes. It includes a sequence of RGB-D images (640×480 resolution) acquired simultaneously from multiple views, providing 21-joint 3D hand pose annotations (based on the MANO model) and 6D poses (3D translation + rotation) of interactive objects, covering diverse grasping actions of 12 common objects (such as scissors, cups, and wrenches). The dataset is divided into a 66,000-frame training set and an 11,000-frame test set, simulating challenges in real-world scenes such as heavy occlusion (hand-object occlusion rate exceeding 70%), rapid hand movements (fingertip speed > 20cm / s), and sudden changes in lighting.

[0164] For hand pose estimation, hand pose error and interaction rationality are used as evaluation indicators. The hand pose error is measured by MPJPE (mean joint position error, in millimeters) and PA-MPJPE (error after rigid alignment) to measure the 3D joint positioning accuracy. The interaction rationality is evaluated by penetration volume (overlapping area of ​​hand-object model) and F@5mm (contact ratio of hand joints to object surface within 5mm).

[0165] The test set annotations are strictly confidential, and prediction results must be submitted through the official online evaluation server to ensure a fair and reliable evaluation process. This mechanism makes it one of the most authoritative technology verification platforms in the field of hand-object interaction.

[0166] This embodiment uses an NVIDIA A10 Tensor Core graphics card and a Windows operating system. The development framework is PyTorch. During training, the Adam optimizer and a weight decay of 5e-4 are used. The batch size is set to 64, and the total number of training iterations is 60. The initial learning rate is 1e-4, decaying every 10 epochs. All implementations are based on PyTorch.

[0167] The results of hand pose estimation using the 3D hand pose estimation network with hand and object features constructed in this embodiment and other models on the Dex-YCB dataset are shown in Table 1. Among them, the 3D hand pose estimation network with hand and object features constructed in this embodiment outperforms several advanced network models in both the MPJPE and PAMPJPE key metrics, showing significant advantages and achieving better results.

[0168] Table 1 Comparison of hand gesture estimation results using different methods on the Dex-YCB dataset

[0169]

[0170] Specifically, the 3D hand pose estimation network integrating hand and object features constructed in this embodiment achieves a 0.8mm improvement over SemGCN (Shuang et al., 2024) and a 0.2mm improvement over HOFEC (Gu et al., 2024) on the MPJPE metric, demonstrating higher accuracy in error reduction. Simultaneously, on the PAMPJPE metric, the network achieves a 0.2mm improvement over SemGCN (Shuang et al., 2024) and a 0.1mm improvement over SimpleHand (Zhou et al., 2024) and HOFEC (Gu et al., 2024), achieving the lowest error and further validating the model's superior ability in geometric alignment and detail fitting of hand poses. Overall, the 3D hand pose estimation network integrating hand and object features constructed in this embodiment has achieved state-of-the-art results on the Dex-YCB dataset, surpassing models such as SimpleHand (Zhou et al., 2024), demonstrating its excellent competitiveness in this task. This result not only demonstrates the model's strong generalization ability and robustness in complex scenarios (such as occlusion, lighting changes, and complex gesture deformations), but also reflects the model's innovations in network structure optimization, feature extraction strategies, and geometrically constrained training methods.

[0171] In summary, the 3D hand pose estimation network integrating hand and object features constructed in this embodiment has achieved leading levels in several key metrics, providing new ideas and practical evidence for the field of gesture pose estimation. While improving overall accuracy, the model also maintains low error and captures details, fully demonstrating its potential and wide applicability in practical applications.

[0172] To verify the robustness of the 3D hand pose estimation network integrating hand and object features constructed in this embodiment in complex hand-object interaction scenarios, the performance of the 3D hand pose estimation network integrating hand and object features constructed in this embodiment and other mainstream algorithms on the HO3D test set is shown in Table 2. The 3D hand pose estimation network integrating hand and object features constructed in this embodiment shows excellent performance in multiple indicators and has obvious advantages compared with other advanced methods.

[0173] Table 2 Comparison of hand gesture estimation results using different methods on the HO3D dataset

[0174]

[0175] Specifically, the 3D hand pose estimation network integrating hand and object features constructed in this embodiment achieves scores of 9.2 and 9.1 for joint error and mesh error, respectively, demonstrating extremely low error and matching or surpassing the outstanding model of Chen et al. (2022b). Simultaneously, on the F@15 metric, the 3D hand pose estimation network integrating hand and object features constructed in this embodiment slightly surpasses Lin et al.'s (2023) 95.9 with a score of 96.1, exhibiting stronger stability under a high fault tolerance threshold. Furthermore, the 3D hand pose estimation network integrating hand and object features constructed in this embodiment achieves an F@5 metric of 54.1, representing a further improvement over Lin et al.'s (2023) model and significantly outperforming earlier methods (such as Hampali and BinsFormer). Overall, the 3D hand pose estimation network integrating hand and object features constructed in this embodiment achieves both high accuracy and high stability while reducing error, demonstrating strong generalization ability and overall advantages in gesture pose estimation tasks.

[0176] To more intuitively demonstrate the superior performance of the hand feature enhancement module (HFE) in the gesture pose estimation task, this embodiment provides a visual analysis of the prediction results of a 3D hand pose estimation network that integrates hand features. Figure 5 and Figure 6 The images show a visual comparison of the Dex-YCB and HO3D datasets, respectively. By comparing the predicted gestures with the actual gestures, the accuracy and detail performance of the hand feature enhancement module (HFE) in joint localization, mesh reconstruction, and overall pose fitting can be clearly observed.

[0177] To verify the effectiveness of the 3D hand pose estimation network that integrates hand features constructed in this embodiment, a systematic verification was conducted on the Dex-YCB and HO3D standard datasets. A progressive module integration strategy was adopted, using the traditional Feature Pyramid Network (FPN) as the baseline model, and introducing three core modules in sequence: the dual-stream hand feature pyramid sub-network DS-FPN, the hand feature dynamic adjustment module DAM, and the dual-stream attention collaborative feature enhancement module HFE, as shown in Tables 3 and 4. The experimental results show that the DS-FPN module reduces the mean joint position error (MPJPE) from the baseline of 12.56 mm to 12.47 mm (relative improvement rate of 10.6%) through the cross-scale attention mechanism. The DAM module further optimizes the MPJPE to 12.44 mm in occluded scenes, and the HFE module finally achieves a breakthrough in accuracy from 5.45 mm to 5.42 mm, especially with a significant reduction in error in the occluded area at the base of the thumb. Experiments have shown that the modular design of DS-FPN, DAM and HFE has complementary advantages in multi-scale feature modeling, dynamic interactive adaptation and cross-modal semantic alignment, which has substantially improved the model in terms of accuracy and robustness.

[0178] Table 3 Comparison of gesture pose estimation results for each module superimposed on the Dex-YCB dataset.

[0179]

[0180] Table 4 Comparison of gesture estimation results for each module on the HO3D dataset.

[0181]

[0182] To more intuitively compare the performance improvement in gesture pose estimation after sequentially stacking each module, this embodiment plots curves for the corresponding metrics on the Dex-YCB and HO3D datasets. A comparison of the MPJPE and PAMPJPE metrics on the Dex-YCB dataset is shown below. Figure 7 As shown, the metrics comparison between Joint and Mesh in the HO3D dataset is as follows: Figure 8 As shown, the metrics comparison between F@5 and F@15 in the HO3D dataset is as follows: Figure 9 As shown. By Figures 7-9 It can be seen that the layered modules have significantly improved the metrics on both the Dex-YCB and HO3D datasets. Therefore, each module has effectively improved accuracy, robustness and efficiency in the gesture estimation task.

[0183] Example 2:

[0184] This embodiment proposes an electronic device, including: one or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors perform the three-dimensional hand pose estimation method that fuses hand features.

[0185] The electronic device may be a mobile phone, computer, or tablet computer, etc., and includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the three-dimensional hand pose estimation method that fuses hand features as described in the embodiments. It is understood that the electronic device may also include an input / output (I / O) interface and communication components.

[0186] The processor is used to execute all or part of the steps in the three-dimensional hand pose estimation method that fuses hand features as described in the above embodiments. The memory is used to store various types of data, which may include, for example, instructions for any application or method in the electronic device, as well as application-related data.

[0187] The processor can be implemented as an Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic components, and is used to execute the three-dimensional hand pose estimation method that integrates hand and object features as described in the above embodiments.

[0188] Example 3:

[0189] This embodiment proposes a computer-readable storage medium that stores executable instructions. When these instructions are executed, if they are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0190] The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the three-dimensional hand pose estimation method that integrates hand features as described in the various embodiments of this application.

[0191] The aforementioned storage media include: flash memory, hard disk, multimedia card, card-type memory (e.g., SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR) memory), random access memory (RAM), static random-access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, disk, optical disk, server, APP (Application) application store, and other media capable of storing program verification codes. These media store computer programs, which, when executed by a processor, can implement the various steps of the aforementioned three-dimensional hand pose estimation method that integrates hand and object features.

[0192] Example 4:

[0193] This embodiment proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the aforementioned three-dimensional hand pose estimation method that integrates hand object features.

[0194] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer program product.

[0195] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0196] The scope of protection of this application is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of this disclosure and its equivalents, then the intent of this disclosure also includes these modifications and variations.

Claims

1. A method of three-dimensional hand pose estimation fusing hand and object features, the method comprising: Includes the following steps: A 3D hand pose estimation network integrating hand and object features is constructed, including a feature extraction module, a two-stream hand feature pyramid sub-network, a hand feature dynamic adjustment module, a two-stream attention collaborative feature enhancement module, and a decoder; Establish a loss function for a 3D hand pose estimation network that integrates hand and object features, train the 3D hand pose estimation network that integrates hand and object features, and obtain the trained 3D hand pose estimation network that integrates hand and object features. The hand gesture image is acquired, and the hand gesture estimation result is obtained based on the trained 3D hand gesture estimation network that integrates hand and object features. The dual-stream hand feature pyramid subnetwork includes two structurally identical hand feature pyramid modules, which extract hand feature maps and object feature maps respectively; The hand feature dynamic adjustment module includes a local feature extraction layer and a cross-sample global attention mechanism based on external memory, used to adjust the hand feature map and generate a global hand feature map. The specific method is as follows: For the hand feature map input to the local feature extraction layer, local details are extracted through convolution operations, and the channel dimensions are adjusted to obtain local hand features. ; Establish a cross-sample global attention mechanism based on external memory, using a learnable key-value memory matrix. Dynamically aggregating common features across samples, the specific method is as follows: Set globally shared learnable parameters, including the key-memory matrix. Sum value memory matrix ,in, The number of memory units, Input the number of channels. The number of output channels is denoted by ; the key memory matrix is ​​used to encode general key features of hand pose, with each memory unit corresponding to a potential hand structure prior; the value memory matrix is ​​used to store the semantic response corresponding to the key features for dynamically adjusting feature importance. Local hand features Expand as Where N = H × W, and B is the batch size, i.e., the number of feature maps processed in a batch, and the key-memory matrix is ​​calculated. attention weights ; Adjust attention weights using layer normalization The dimensional order is used to generate normalized attention weights. ; The normalized attention weight A and the value memory matrix are used. Weighted summation is used to generate a global hand feature map. ; The dual-stream attention collaborative feature enhancement module is used to enhance the features of the global hand feature map, hand feature map and object feature map, extract fine-grained features of the gesture posture image, and obtain the feature-enhanced hand feature map.

2. The three-dimensional hand pose estimation method based on the fusion of hand features according to claim 1, characterized in that, The feature extraction module includes several stages for extracting feature maps at different levels, and selects feature maps at multiple levels as input feature maps for the dual-flow hand feature pyramid sub-network.

3. The three-dimensional hand pose estimation method based on the fusion of hand features according to claim 1, characterized in that, The specific methods for extracting hand feature maps and object feature maps are as follows: The feature maps of different levels in the input dual-stream hand feature pyramid subnetwork are weighted and summed. The importance of the feature maps at different levels is adaptively adjusted through learnable weights to obtain the first feature map. ; An enhanced weighted fusion unit mapping block is constructed, and a hand feature pyramid module is constructed by inserting the enhanced weighted fusion unit mapping block into the lateral connection path of the bidirectional feature pyramid network BiFPN. The first feature is processed using an enhanced weighted fusion unit, the Mapping block. Spatial enhancement and channel modulation are performed to obtain the second feature. Based on the first feature Second feature Obtain the intermediate feature map; Two hand feature pyramid modules fuse intermediate feature maps output by enhanced weighted fusion units (Mapping blocks) at different levels through bidirectional paths from top to bottom and bottom to top, resulting in hand feature maps and object feature maps, respectively.

4. The three-dimensional hand pose estimation method based on the fusion of hand features according to claim 3, characterized in that, The enhanced weighted fusion unit (Mapping block) includes: a first 1×1 convolution, a 3×3 convolution, a second 1×1 convolution, and a SiLU activation function; the first 1×1 convolution, the 3×3 convolution, and the second 1×1 convolution are used sequentially to perform spatial enhancement and channel adjustment on the features input to the enhanced weighted fusion unit (Mapping block); the output features of the first 1×1 convolution are concatenated with the output features of the second 1×1 convolution; and the intermediate feature map output by the enhanced weighted fusion unit (Mapping block) is obtained by passing the SiLU activation function. The first feature is processed using an enhanced weighted fusion unit, the Mapping block. Spatial enhancement and channel adjustment are performed using the following methods: The first feature is processed sequentially using a first 1×1 convolution, a 3×3 convolution, and a second 1×1 convolution. Perform spatial enhancement and channel adjustment to enhance the primary feature. Spatial representation capability, resulting in the second feature. ; The second feature and the first feature after 1×1 convolution processing The intermediate feature maps are concatenated and obtained by using the SiLU activation function to obtain the output of the enhanced weighted fusion unit Mapping block.

5. The three-dimensional hand pose estimation method based on the fusion of hand features according to claim 1, characterized in that, The specific method for extracting fine-grained features from gesture pose images to obtain the enhanced hand feature map is as follows: Feature encoding and serialization are performed on the hand feature map and the object feature map. The number of channels in the hand feature map is adjusted by 1×1 convolution to generate the query matrix Q. The number of channels in the object feature map is adjusted by 1×1 convolution to generate the key matrix K and the value matrix V. A multi-head attention mechanism is calculated for the hand feature map and the object feature map. The query matrix Q, key matrix K, and value matrix V are divided into h groups, and the single-head attention of each group is calculated. The h single-head attentions are concatenated, linearly projected, and then combined with the global hand feature map. Adding elements together yields the chiral-object cooperative features. ; Collaborative features of adversaries A two-branch refinement and feature enhancement process is performed. The first branch is residual preservation, which is used to retain the initial fused features and maintain information integrity, resulting in the output of the first branch. ; The second branch includes layer normalization and a multilayer perceptron (MLP) to stabilize the training process, alleviate the gradient explosion problem, and refine features through nonlinear transformations to obtain the output of the second branch. ; The output of the first branch The output of the second branch The features are added together to generate a hand feature map with enhanced features.

6. The three-dimensional hand pose estimation method based on the fusion of hand features according to claim 1, characterized in that, The decoder performs hand pose estimation based on the enhanced hand feature map to obtain the hand pose estimation result.

7. The three-dimensional hand pose estimation method based on the fusion of hand features according to claim 1, characterized in that, The overall loss function of the 3D hand pose estimation network that integrates hand and object features is established, including hand pose loss and object pose loss, as shown in the following formula: in, For loss of hand posture, Object pose loss, The collaborative weighting coefficient controls the intensity of the influence of object information on hand optimization; Hand pose loss includes 2D joint detection loss. 3D vertex and joint loss and MANO parameter loss As shown in the formula below: in, , , To balance the coefficients of each loss function weight; 2D joint loss The following formula is used to supervise the coordinate prediction of 2D joints in the hand: Where B is the batch size. This refers to the number of joints in the hand. The predicted 2D coordinates of the i-th joint in the b-th sample. Let be the true 2D coordinates of the i-th joint in the b-th sample; 3D Vertex and Joint Loss As shown in the formula below: Where V is the number of 3D vertices of the hand as defined in the MANO parameter loss. Let be the predicted 3D coordinates of the j-th vertex in the b-th sample. Let J be the true 3D coordinates of the j-th vertex in the b-th sample. The predicted 3D coordinates of the j-th joint in the b-th sample. Let J be the true 3D coordinates of the j-th joint in the b-th sample; The MANO parameter loss is shown in the following formula: in, Let b be the predicted shape parameters for the b-th sample. The true shape parameters are for the b-th sample. Let b be the pose parameters predicted for the b-th sample. These are the pre-true pose parameters in the b-th sample; Object pose coordination loss includes 2D projection loss. and confidence loss ; 2D projection loss As shown in the formula below: in, This represents the number of corner points of the object's 3D bounding box. Let k be the predicted 2D projection coordinates of the k-th corner point. Let be the true 2D projected coordinates of the k-th corner point; Confidence loss As shown in the formula below: in, A confidence level exists for the predicted object. These are true binary labels, where 0 indicates invisible and 1 indicates visible.

Images