A hand pose estimation method based on cross-scale cross-space feature interaction

Abstract

This invention belongs to the field of computer vision technology, specifically relating to a hand pose estimation method based on cross-scale and cross-spatial feature interaction. The method includes: acquiring a hand image and inputting it into a spatial prior module for processing to obtain spatial prior features; using a Swin-Transformer network and an injection module to process the hand image and spatial prior features to obtain a spatial feature injection feature pyramid containing feature maps of four different sizes; inputting the spatial feature injection feature pyramid into a feature pyramid enhancement module to obtain an enhanced feature pyramid; densely setting multiple 3D anchor points on the hand image and inputting the 3D anchor points and the enhanced feature pyramid into an anchor point interaction module for processing to obtain multiple enhanced anchor point features; calculating the precise hand joint positions based on the enhanced anchor point features and anchor point positions. This invention can accurately capture the overall hand pose and subtle movements, maintaining high-precision 3D joint positioning even in cases of depth blur.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a hand pose estimation method based on cross-scale and cross-spatial feature interaction. Background Technology

[0002] Over the past few decades, with the rapid development of computer vision, hand pose estimation technology has gradually become an important research direction in many fields such as human-computer interaction, virtual reality, augmented reality, and robot control. Hand pose estimation refers to the automatic detection and understanding of the three-dimensional position information of the hand and the relative posture of the finger joints from images or videos. This technology not only provides detailed hand motion recognition but also provides technical support for achieving more natural and intuitive user interaction.

[0003] Currently, research on hand pose estimation mainly focuses on two aspects: first, the detection of hand key points in two-dimensional images, and second, the reconstruction of three-dimensional hand pose based on depth information. Traditional hand pose estimation methods mostly rely on classic image processing techniques, such as hand template matching and feature extraction based on color and texture. However, these methods have poor robustness in complex backgrounds and under different lighting conditions, and cannot meet the high-precision requirements of real-world applications.

[0004] With the rise of deep learning technology, hand pose estimation methods based on convolutional neural networks (CNNs) and image feature learning have made significant progress. By training deep neural networks, especially convolutional neural networks, more refined and robust feature representations can be learned from large-scale labeled data, greatly improving the accuracy and stability of hand keypoint detection. However, traditional convolutional neural network structures still have certain limitations when processing multi-scale and multi-level information, especially in handling complex hand poses and cross-scale feature interactions, where they still struggle to fully capture the details and global information of hand poses.

[0005] To overcome the limitations of traditional CNNs in hand pose estimation, deep learning architectures based on Transformers have become a research hotspot in recent years. Transformers effectively capture global dependencies through self-attention and handle long-range dependencies through hierarchical feature modeling, thus achieving significant results in natural language processing, computer vision, and other fields. Particularly in image processing tasks, the Swin-Transformer, as a novel transformer architecture, can perform hierarchical feature extraction from images through a local window self-attention mechanism while maintaining low computational complexity. The Swin-Transformer exhibits unique advantages in processing fine-grained features and multi-scale information in images, providing richer feature information for hand pose estimation tasks.

[0006] However, despite the progress made in Transformer-based hand pose estimation methods, significant challenges remain due to the complex geometry of the hand and issues such as depth blurring in a single RGB image. How to extract more accurate and semantically meaningful hand features at different levels and scales using deep learning models, and further achieve effective interaction of these features, so that the feature representation possesses both fine-grained local details and global contextual information while also containing depth information, has become an important research direction for improving the accuracy and robustness of hand pose estimation.

[0007] This invention aims to propose a novel hand pose estimation method by combining Swin-Transformer to extract multi-level features and perform cross-scale and cross-spatial feature interaction. This method can establish an effective connection between global and local spatial information, thereby better capturing the details and overall structure of hand pose, overcoming the limitations of existing methods in multi-scale feature extraction and information interaction, and ultimately achieving higher accuracy and more robust hand pose estimation. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention proposes a hand pose estimation method based on cross-scale and cross-spatial feature interaction, which includes:

[0009] S1: Acquire a hand image and input it into the spatial prior module for processing to obtain spatial prior features;

[0010] S2: The hand image and spatial prior features are processed using the Swin-Transformer network and injection module to obtain a spatial feature injection feature pyramid containing four different size feature maps;

[0011] S3: Inject spatial features into the feature pyramid and input them into the feature pyramid enhancement module for processing to obtain the enhanced feature pyramid;

[0012] S4: Densely set multiple 3D anchor points on the hand image, and input the 3D anchor points and the enhanced feature pyramid into the anchor point interaction module for processing to obtain multiple enhanced anchor point features;

[0013] S5: Calculate the offset and weight of each anchor point relative to the joint based on the enhanced anchor point features; calculate the precise hand joint position based on the anchor point offset, anchor point weight, and anchor point position.

[0014] Preferably, the spatial prior module's processing of the hand image includes: performing different convolution operations on the hand image to obtain three feature maps with different resolutions; flattening and stitching the three feature maps with different resolutions to obtain spatial prior features.

[0015] Preferably, the injection module processes the hand image and spatial prior features as follows:

[0016]

[0017] Among them, F i F represents the output of the i-th injection module. sw i This represents the input features of the i-th stage of the Swin-Transformer network. represents spatial prior features, norm(·) represents layer normalization, and CrossAttention(·) represents the cross-attention mechanism.

[0018] Preferably, the process by which the feature pyramid enhancement module processes the spatial features injected into the feature pyramid includes:

[0019] Self-attention calculation is performed on the features of the current layer to obtain self-attention features;

[0020] The features from other layers are sequentially convolved and upsampled, and then cross-attention is calculated with the features of the current layer to obtain cross-attention features.

[0021] The self-attention features and cross-attention features are concatenated to obtain the concatenated features; the concatenated features are processed through a linear layer to obtain the enhanced features of the current layer; the enhanced features of all layers are combined to form the enhanced feature pyramid.

[0022] Preferably, the anchor point interaction module consists of 6 decoders, each of which includes a self-attention module and a cross-attention module.

[0023] Preferably, the process of calculating the offset of the anchor point relative to the joint includes:

[0024] O j (a) = MLP offset (f a )

[0025] Among them, O j (a) represents the offset of anchor point a relative to joint j, f a The enhanced anchor feature output by anchor point a after passing through the anchor point interaction module, MLP offset This represents a multilayer perceptron used to calculate the offset.

[0026] Preferably, the process of calculating the weight of the anchor point relative to the joint includes:

[0027] W j (a) = MLP weight (f a )

[0028] Among them, W j (a) represents the weight of anchor point a relative to joint j, MLP weight f represents the multilayer perceptron used to compute the weights. a This indicates the enhanced anchor point feature output by anchor point a after passing through the anchor point interaction module.

[0029] Preferably, the formula for calculating the position of hand joints is:

[0030]

[0031] in, This represents the estimated planar coordinates of target joint j. This represents the estimated depth coordinates of target joint j. C represents the normalized weight from anchor point a relative to target joint j. i (a) represents the planar coordinates of anchor point a. C represents the planar offset of anchor point a relative to target joint j. d (a) represents the depth coordinates of anchor point a. This represents the depth offset of anchor point a relative to target joint j, and A represents the set of anchor points.

[0032] The beneficial effects of this invention are as follows: Using the Swin-Transformer as the backbone, this invention combines the global feature extraction capability of the Swin-Transformer with the local spatial information generated by convolution, enabling the extracted feature pyramid to retain both global and local spatial information. Furthermore, it achieves feature interaction at different scales through the feature pyramid, returning a feature pyramid of the same size but with richer contextual information. Finally, the anchor point interaction module allows each anchor point to perceive both local and global information, while simultaneously enabling information sharing between anchor points, effectively improving the accuracy and robustness of hand pose estimation. This method can accurately capture the overall hand pose and subtle movements, maintaining high-precision 3D joint localization even in cases of depth ambiguity, providing a more reliable hand pose estimation solution for practical applications. Attached Figure Description

[0033] Figure 1 This is a flowchart of the hand pose estimation method based on cross-scale and cross-spatial feature interaction in this invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] This invention proposes a hand pose estimation method based on cross-scale and cross-spatial feature interaction, such as... Figure 1 As shown, the method includes the following:

[0036] S1: Acquire a hand image and input it into the spatial prior module for processing to obtain spatial prior features.

[0037] Figure 1 In this model, the feature pyramid extraction module includes a spatial prior module, a Swin-Transformer network, and an injection module. The spatial prior module processes the hand image by performing different convolution operations on the hand image to obtain three feature maps of different resolutions; then, it flattens and concatenates these three feature maps to obtain spatial prior features, which are used to assist the Swin-Transformer in extracting the feature pyramid.

[0038] S2: The Swin-Transformer network and injection module are used to process the hand image and spatial prior features to obtain a spatial feature injection feature pyramid containing four different sizes of feature maps.

[0039] This invention employs spatial and local information extracted through convolution to assist the learning of the Swin-Transformer, enabling the model to simultaneously possess the ability to perceive local details and model global dependencies. Specifically:

[0040] The hand image is input into the Swin-Transformer network. The raw features extracted by the Swin-Transformer serve as the query, and the spatial prior features serve as the key and value. These features are aligned in terms of space and channel number in the injection module, and spatial information is injected through a cross-attention mechanism, as shown below:

[0041]

[0042] Among them, F i F represents the output of the i-th injection module. sw i This represents the input features of the i-th stage of the Swin-Transformer network, with the initial input being a hand image; represents spatial prior features, norm(·) represents layer normalization, and CrossAttention(·) represents the cross-attention mechanism.

[0043] The output of the i-th injection module is added to the input features of the i-th stage of the corresponding Swin-Transformer network as the input of the next stage of the Swin-Transformer network. Finally, the features after each block of the Swin-Transformer are extracted to obtain a spatial feature injection feature pyramid containing feature maps of four different sizes.

[0044] S3: The spatial features are injected into the feature pyramid and processed by the feature pyramid enhancement module FPE to obtain the enhanced feature pyramid.

[0045] In this module, each layer of the feature pyramid has both intra-layer and cross-layer interactions. Intra-layer interactions occur through self-attention, resulting in self-attention features. Different layers first undergo dimensionality reduction using 1x1 convolutions to unify the dimensions, then interact through cross-attention to obtain cross-attention features. Taking a feature map with C channels in a feature pyramid as an example: first, self-attention is calculated within this layer; then, the feature maps from other layers are adjusted to have C channels using 1x1 convolutions; upsampling is used to ensure consistent spatial resolution; cross-attention is then calculated; and the self-attention and cross-attention results are concatenated to obtain a feature map with 5C channels. Finally, a linear layer adjusts the channel count back to C, returning a feature map with the same input size but richer features. The enhanced features from all layers constitute the enhanced feature pyramid.

[0046] In this module, feature maps interact across different scales, integrating information from various dimensions. The key function of this process is to fuse and enrich features, sharing information across features at different scales, thereby enabling the model to more comprehensively understand the various levels and details of hand pose. Ultimately, the interacting feature maps return a feature pyramid of the same size as the input, where features at each scale are richer and more accurate, better supporting the prediction of hand joints.

[0047] S4: Multiple 3D anchor points are densely set on the hand image. These 3D anchor points are used as anchor queries, and the enhanced feature pyramid is input into the anchor point interaction module (Decoder) for processing. This links the anchor points together, allowing each anchor point to obtain global information.

[0048] Precise localization of hand keypoints is achieved by densely placing multiple 3D anchor points in the image. Specifically, a reference point is selected in the original image, and 3D anchor points are uniformly placed in the x, y, and z directions with a fixed step size. Each anchor point represents a potential keypoint location of the hand. These anchor points are then used as anchor queries, inputting them along with the enhanced feature pyramid into the anchor point interaction module, enabling interconnection and information sharing among the anchor points. Through this information exchange, the model acquires global spatial features, allowing each anchor point to perceive changes in overall hand pose and contextual information. The goal of this process is to provide accurate 3D spatial location information for hand joints, ensuring that the anchor points comprehensively represent the key parts of the hand.

[0049] Each decoder in the anchor interaction module contains a self-attention module and a cross-attention module. In the self-attention module, the query, key, and value settings are as follows:

[0050] Q = D + P q

[0051] K = D + P q

[0052] V=D

[0053] Where D represents the decoder embedding, the decoder embedding of the first layer decoder is obtained by embedding the output of the augmented feature pyramid, and the decoder embedding of subsequent layers is obtained by embedding the output of the previous layer decoder; P q Spatial encoding for the q-th anchor point: specifically P q =MLP(PE(a q )), a q Let a be the coordinates of anchor point q. q =(x q ,y q ,z q MLP stands for Multilayer Perceptron.

[0054] In the cross-attention module, the query, key, and value settings are as follows:

[0055]

[0056] K = a q

[0057] V = E

[0058] Where E represents the feature map output by the feature pyramid enhancement module. This represents the output of the self-attention layer.

[0059] The anchor point interaction module's processing of 3D anchor points and enhanced feature pyramids includes:

[0060] First, the 3D anchor points are position-encoded and then embedded in the decoder, and the result is summed to form Q and K in the self-attention module. This means Q and K are the same in the sub-attention module. Simultaneously, V is set as the decoder embedding D, and then the result is obtained through the self-attention module. Then, in the cross-attention module, Q, K, and V are set according to the formula above, and cross-attention calculation is performed to obtain the output of the first decoder layer. The subsequent second to last layers are all concatenated based on the first layer, with the overall structure remaining consistent. The only difference is that the decoder embedding of the subsequent layers is obtained by embedding the output of the previous layer's decoder.

[0061] S5: Calculate the offset and weight of each anchor point relative to the joint based on the enhanced anchor point features; calculate the precise hand joint position based on the anchor point offset, anchor point weight, and anchor point position.

[0062] Each 3D anchor point is used as a local estimator to calculate the offset and weight of each anchor point relative to the joint. Specifically, after passing through the anchor point interaction module, each anchor point has certain local and global information. Then, an MLP is added to the offset estimation branch and the weight estimation branch respectively to estimate the offset and weight of each anchor point relative to the joint.

[0063] The process of calculating the offset of the anchor point relative to the joint includes:

[0064] O j (a) = MLP offset (f a )

[0065] Among them, f a The enhanced anchor point feature output after anchor point a passes through the anchor point interaction module, O j (a) represents the offset of anchor point a relative to joint j, including planar offset in the x and y directions. Depth offset in the z direction MLP offset This represents the multilayer perceptron used to calculate the offset. The offset is further decomposed into depth offset and planar offset to optimize for depth and spatial position, respectively.

[0066] The process of calculating the weight of the anchor point relative to the joint includes:

[0067] W j (a) = MLP weight (f a )

[0068] Among them, W j (a) represents the weight of anchor point a relative to joint j, MLP weight This represents a multilayer perceptron used to calculate weights.

[0069] The precise hand joint positions are calculated based on the anchor point's offset relative to the joint, the anchor point weight, and the anchor point's location. Specifically:

[0070] The offset estimates for each anchor point are combined using a weighted fusion method to obtain the final predicted position for each joint. Through weighted fusion, the local estimates for different anchor points are balanced, ensuring that the final joint position reflects both local details and global information. The final predicted target joint can be represented as:

[0071]

[0072] in, This represents the estimated planar coordinates of target joint j. This represents the estimated depth coordinates of target joint j. C represents the normalized weight from anchor point a relative to target joint j. i (a) represents the planar coordinates of anchor point a. C represents the planar offset of anchor point a relative to target joint j. d (a) represents the depth coordinates of anchor point a. This represents the depth offset of anchor point a relative to target joint j, and A represents the set of anchor points.

[0073] By using the above method for hand pose estimation, global and local spatial information of the hand can be extracted from the hand image, and effective feature interaction and global-local information fusion can be performed to accurately restore the overall outline and details of the hand, thus achieving accurate hand pose estimation in deep blurry scenes.

[0074] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A hand pose estimation method based on cross-scale and cross-spatial feature interaction, characterized in that, include: S1: Acquire a hand image and input it into the spatial prior module for processing to obtain spatial prior features; The spatial prior module's processing of the hand image includes: performing different convolution operations on the hand image to obtain three feature maps of different resolutions; flattening and stitching the three feature maps of different resolutions to obtain spatial prior features; S2: The Swin-Transformer network and injection module are used to process the hand image and spatial prior features to obtain a spatial feature injection pyramid containing feature maps of four different sizes; the processing of the hand image and spatial prior features by the injection module is represented as follows: ； in, This represents the output of the i-th injection module. This represents the input features of the i-th stage of the Swin-Transformer network. Represents spatial prior features. Representation layer normalization, This represents the cross-attention mechanism; S3: Inject spatial features into the feature pyramid and input them into the feature pyramid enhancement module for processing to obtain the enhanced feature pyramid; S4: Densely set multiple 3D anchor points on the hand image, and input the 3D anchor points and the enhanced feature pyramid into the anchor point interaction module for processing to obtain multiple anchor point features; S5: Calculate the offset and weight of each anchor point relative to the joint based on the anchor point features; calculate the precise hand joint position based on the anchor point offset, anchor point weight, and anchor point position.

2. The hand pose estimation method based on cross-scale and cross-spatial feature interaction according to claim 1, characterized in that, The feature pyramid enhancement module processes spatial features by injecting them into the feature pyramid, including the following steps: Self-attention calculation is performed on the features of the current layer to obtain self-attention features; The features from other layers are sequentially convolved and upsampled, and then cross-attention is calculated with the features of the current layer to obtain cross-attention features. The self-attention features and cross-attention features are concatenated to obtain the concatenated features; the concatenated features are processed through a linear layer to obtain the enhanced features of the current layer; the enhanced features of all layers are combined to form the enhanced feature pyramid.

3. The hand pose estimation method based on cross-scale and cross-spatial feature interaction according to claim 1, characterized in that, The anchor point interaction module consists of 6 decoders, each of which includes a self-attention module and a cross-attention module.

4. The hand pose estimation method based on cross-scale and cross-spatial feature interaction according to claim 1, characterized in that, The process of calculating the offset of the anchor point relative to the joint includes: ； in, Indicates anchor point The offset relative to joint j, Represents anchor point Enhanced anchor point features output after passing through the anchor point interaction module This represents a multilayer perceptron used to calculate the offset.

5. The hand pose estimation method based on cross-scale and cross-spatial feature interaction according to claim 1, characterized in that, The process of calculating the weight of the anchor point relative to the joint includes: ； in, Indicates anchor point Relative to the weight of joint j, This represents a multilayer perceptron used to compute the weights. This indicates the enhanced anchor point feature output by anchor point a after passing through the anchor point interaction module.

6. The hand pose estimation method based on cross-scale and cross-spatial feature interaction according to claim 1, characterized in that, The formula for calculating the position of hand joints is: ； in, This represents the estimated planar coordinates of target joint j. This represents the estimated depth coordinates of target joint j. Indicates from the anchor point The normalized weights relative to the target joint j Indicates anchor point plane coordinates, Indicates anchor point The planar offset relative to the target joint j Indicates anchor point Depth coordinates Indicates anchor point The depth offset relative to the target joint j. This represents the set of anchor points.

Images