A lightweight 3D hand pose estimation method based on RGB images

Abstract

The present invention relates to a lightweight three-dimensional hand posture estimation method based on RGB images. First, multiple sandglass residual blocks are stacked in series to perform local feature modeling. Secondly, after some sandglass residual blocks, the features are "unfolded" and global feature modeling is performed using a Transformer block structure. The features are then "folded" and feature fusion is performed. Then, a separable self-attention mechanism is introduced into the Transformer block to reduce the computational complexity of self-attention. Finally, a two-dimensional hand posture heat map is obtained by convolution of the feature map, and the two are spliced together and then convolved to obtain a joint vector map. The two are spliced again and then convolved to obtain a three-dimensional hand posture heat map, and finally three-dimensional hand posture estimation is achieved. The present invention directly estimates three-dimensional hand posture from RGB images, and can maintain excellent computational efficiency while having high estimation accuracy. It can have broad application prospects in the fields of human-computer interaction, virtual reality, gesture recognition, etc.

Description

Technical Field

[0001] The present invention relates to a lightweight three-dimensional hand posture estimation method based on RGB images. The method uses multi-layer lightweight sandglass residual connection blocks to extract local image features, and then combines the visual Transformer block structure to extract global image features. Finally, the two-dimensional hand posture heat map is promoted to a three-dimensional posture heat map through convolution, realizing three-dimensional hand posture estimation based on RGB images. The method belongs to the fields of computer vision and deep learning technology. Background Art

[0002] In recent years, thanks to the development of deep learning convolutional neural networks, the task of hand pose estimation has seen rapid progress, with numerous promising methods proposed over the past decade. Based on how the hand pose information is represented, the task of hand pose estimation can be divided into two types: two-dimensional and three-dimensional. Since RGB images require lower equipment and computational costs than RGB-D images with depth information, hand pose estimation based on RGB images is more valuable for research. However, three-dimensional hand pose estimation can more accurately represent the positions of hand joints, making it challenging to accurately estimate the three-dimensional information of hand joints in RGB images lacking depth information.

[0003] Before the rise of deep learning research, traditional machine learning algorithms were commonly used for hand pose estimation. For example, Rehg, Wang, and others developed a visual hand tracking program and used physical peripherals to perform hand tracking and interaction tasks. These traditional methods typically used three-dimensional dynamic models to fit the model. Later, due to breakthroughs in human pose estimation methods, neural networks were widely used for pose estimation. Human pose estimation and hand pose estimation are two closely related research fields. Inspired by human pose estimation, many hand pose estimation methods have been proposed. Spurr et al. proposed a method that uses a generative model to apply to both RGB and depth images. This method uses a semi-supervised learning approach and cross-modal training to obtain a hand spatial model. Cai et al. proposed a weakly supervised method that uses depth images for training and RGB images for testing as joint predictions. This method performs depth image regression on the predicted 3D pose, complementing the weakly supervised model in 3D pose regression.

[0004] In recent years, many hand pose estimation methods based on RGB images have been based on detecting 2D skeletal keypoints of the hand, then using this 2D information to upscale to 3D. Franziska et al. proposed a hand pose estimation model that combines neural networks with kinematic constraints to address the problem of hand tracking and pose estimation from monocular RGB images. They believe that this combination is effective for data generalization and makes the estimation of skeletal keypoints more robust under different camera viewpoints during motion. They also proposed an image conversion method based on geometric consistency, using a neural network to "convert" synthetic data into real data, thereby facilitating the generation of more robust 3D models. Liuhao et al. proposed a method for estimating hand pose and generating hand shapes from RGB images. The method consists of two training steps. In the first stage, the authors use synthetic data for training, using an hourglass network model and a graph convolutional network, and finally linear regression to generate the hand pose. In the second stage, the first-stage model is fine-tuned for training on real data, which improves the model's generalization and performance in real-world scenarios. Yuxiao et al. proposed a monocular hand motion capture method that leverages multimodal data. This method recovers a 2D keypoint representation of the hand from a single image and then uses the 2D keypoints to recover the 3D keypoints and shape of the hand. For 2D keypoint detection, a convolutional network is used to generate a heatmap of the hand keypoints, and heatmap regression is used to find the 2D keypoint representation. For 3D keypoint detection, the 3D hand pose is regressed using the 2D keypoint heatmap and image features, drawing inspiration from the human pose estimation network of VNect.

[0005] Ideally, methods for capturing hand poses for applications such as real-time human-computer interaction should provide direct feedback to users at low cost. Utilizing low-cost RGB cameras can reduce image acquisition costs and power consumption, and achieve high-precision, low-latency 3D hand pose estimation with good accuracy and low computational complexity for practical applications. However, 3D hand pose estimation is extremely challenging, especially for monocular RGB images, where the lack of depth information makes accurate 3D hand pose estimation difficult. Furthermore, monocular RGB images inherently have depth ambiguity, which can cause partial occlusion of the hand pose, making pose estimation difficult. Finally, network structures that achieve high 3D pose estimation accuracy often come with higher computational complexity, significantly impacting practical performance when hardware computing power is insufficient. Therefore, establishing a lightweight network model based on RGB images to achieve accurate 3D hand pose keypoint estimation while maintaining low computational complexity is a current research focus. Summary of the Invention

[0006] In response to the above problems, the present invention designs and implements a lightweight three-dimensional hand posture estimation method based on RGB images, which reduces more computational costs while ensuring estimation accuracy. The present invention first designs a plurality of serially connected sandglass residual convolution blocks for local feature modeling. Secondly, after some residual blocks, the features are "unfolded", and then global feature modeling is performed using the Transformer block structure. The features are then "folded" and feature fusion is performed. Then, a separable self-attention mechanism is introduced in the Transformer block to reduce the computational complexity of self-attention. Finally, the feature map is convolved to obtain a two-dimensional hand posture heat map, and the two are spliced and convolved to obtain a joint vector map, which are spliced again and then convolved to obtain a three-dimensional hand posture heat map, ultimately achieving three-dimensional hand posture estimation.

[0007] The specific steps of a lightweight 3D hand pose estimation method based on RGB images are as follows:

[0008] Step 1: Use multiple sandglass residual blocks to stack in series to ensure the inductive bias characteristics of convolution for local feature modeling.

[0009] Step 2: After the 4th, 5th, and 6th sandglass residual blocks, the feature map is passed through a convolution layer with a 1×1 kernel to adjust the number of feature channels, "expand" the features, and then global feature modeling is performed using the Transformer block structure. The number of channels is then adjusted back to the original size through a convolution layer with a 1×1 kernel size, the features are "folded", and feature fusion is achieved through a 1×1 convolution layer.

[0010] Step 3: Introduce a separable self-attention mechanism into the Transformer block used for global feature modeling to reduce the computational complexity of self-attention.

[0011] Step 4: Convolve the feature map to obtain a 2D hand pose heatmap. Convolve the two together to obtain a joint vector map. Convolve them again to obtain a 3D hand pose heatmap, ultimately achieving 3D hand pose estimation.

[0012] Compared with the existing 3D hand posture estimation method based on RGB images, the present invention has the following characteristics:

[0013] 1. We designed a lightweight feature extraction network to extract features from RGB images and regress 2D and 3D hand pose heatmaps, overcoming the computational speed issues of existing methods. We stacked multiple lightweight sandglass convolutional residual blocks for local feature modeling. This approach retains the reduced computational complexity offered by point-by-point and depth-wise separable convolutions while applying depth-wise separable convolutions to higher-dimensional features, thereby encoding richer spatial information.

[0014] 2. A visual Transformer with separable self-attention is introduced between some sandglass residual blocks in the backbone network. This Transformer has the ability to capture the global receptive field of the input feature map and establish global dependencies in the spatial dimension, thereby learning global visual representation information. The separable self-attention reduces the computational complexity of self-attention and can reduce latency in the actual reasoning of hand pose estimation. Therefore, this invention ensures the accuracy of 3D hand pose estimation while also improving the speed of model calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 This is a flowchart of the implementation of the lightweight three-dimensional hand posture estimation method based on RGB images of the present invention;

[0017] Figure 2 This is a structural diagram of the sandglass of the present invention. DETAILED DESCRIPTION

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0019] The specific steps of a lightweight 3D hand pose estimation method based on RGB images are as follows:

[0020] The present invention provides a lightweight three-dimensional hand posture estimation method based on RGB images. Figure 1 Implementation flowchart.

[0021] like Figure 1 The implementation of the method of the present invention mainly includes four steps: Step 1: stack multiple sandglass residual blocks in series for local feature modeling; Step 2: "expand" the features, then perform global feature modeling with the Transformer block structure, then "fold" the features and perform feature fusion; Step 3: introduce a separable self-attention mechanism in the Transformer block to reduce the computational complexity of self-attention; Step 4: convolve the feature map to obtain a two-dimensional hand posture heat map, concatenate the two and obtain a joint vector map through convolution, and then convolve the two again to obtain a three-dimensional hand posture heat map, finally realizing three-dimensional hand posture estimation.

[0022] The following describes each step in detail.

[0023] Step 1: Use multiple sandglass residual blocks to stack in series to ensure the inductive bias characteristics of convolution for local feature modeling.

[0024] The method of the present invention introduces the sandglass residual block structure, such as Figure 2 The figure shows the Sandglass architecture. The first and last convolutional layers in the main branch are depthwise separable convolutional layers that preserve the number of channels. To simulate the classic bottleneck block structure, two consecutive pointwise convolutions are stacked between the two depthwise separable convolutional layers to first reduce and then increase the number of channels. Because 1×1 pointwise convolutions can weaken spatial information, two 3×3 convolutions are placed outside the two 1×1 convolutions, and the two depthwise separable convolutions are appropriately applied to higher-dimensional features. This allows for encoding richer spatial information and generates more expressive spatial information.

[0025] Assumptions represents the input features, Represents the output features. The module can be expressed as follows:

[0026]

[0027] where φ i,p ,φ i,d Denotes the i-th point-wise convolution and depth-wise convolution, respectively, ensuring that depth-wise separable convolution is processed in high-dimensional space and obtains richer feature representation. The basic information of the sandglass residual block is shown in Table 1, where t and s represent the channel reduction rate and stride, respectively.

[0028] Table 1 Basic information of sandglass residual block

[0029]

[0030] Step 2: After the 4th, 5th, and 6th sandglass residual blocks, the features are "expanded" and global feature modeling is performed using the Transformer block structure. The features are then "folded" and feature fusion is performed.

[0031] Use the previous sandglass block output as the d-dimensional input feature First, an n×n standard convolution layer is applied, and then a 1×1 point convolution layer is used to generate the feature X L In order to make the model have a global representation of spatial inductive bias, the feature X L Expand into N non-overlapping patches, using Here, P = wh, and is the number of patches, h≤n and w≤n are the height and width of the patch respectively. According to formula (3), for each patch, that is, p∈{1,···,P}, the relationship between patches is encoded by Transformer to obtain

[0032] X G (p)= Transformer (X U (p)),1≤p≤P (3)

[0033] Because this operation does not lose the order of the patch, nor the spatial order of the pixels within each patch, it can be folded get Next, point-wise convolution is used to project XF into a low-dimensional space and combined with feature X through a concatenation operation, and then another n×n convolution layer is used to fuse these features. U (p) Use convolution to encode local information of n×n region, X G (p) Encode the global information of the patch at position p, X G Each pixel in can encode information from all pixels in X, so the total effective receptive field is H×W.

[0034] Step 3: Introduce a separable self-attention mechanism into the Transformer block used for global feature modeling to reduce the computational complexity of self-attention.

[0035] The structure of separable self-attention is similar to the multi-head self-attention used in Transformer, which takes the input X input The input branch I is processed using the key K and the value V. I The linear layer converts X inputEach d-dimensional token in is mapped to a scalar. Weight W I Used to represent potential tokens and X input The distance between them is calculated to produce a k-dimensional vector, and then the softmax function is used to produce the context score Instead of calculating the attention score for each token for all k tokens, we can separate the attention score and only calculate the attention score for the potential token. This can reduce the time complexity of calculating the attention score from O(k 2 ) is reduced to O(k).

[0036] According to formula (4), the context vector C v By context score C s Get, among them By having weight The key branch K will enter X input Obtained after linear projection into d-dimensional space.

[0037]

[0038] C v The context information encoded in X input To do this, we use a weight of W V The value of branch V will be input to X input Linear projection to d-dimensional space Then the ReLU activation function is used to generate the output Finally, the output is fed to the Another linear layer produces the final output

[0039] Step 4: Convolve the feature map to obtain a 2D hand pose heatmap. Convolve the two together to obtain a joint vector map. Convolve them again to obtain a 3D hand pose heatmap, ultimately achieving 3D hand pose estimation.

[0040] First, the output feature map F of the backbone network is passed through a two-layer convolutional network to obtain a joint prediction confidence map containing 21 hand joint points, namely the two-dimensional heat map A. The feature map F is then concatenated with the two-dimensional heat map A and passed through two layers of convolution to obtain the joint vector map D. Each pixel in the joint vector map D encodes the direction of each skeletal joint, representing a 3D vector from the parent joint to the child joint. Next, the two-dimensional heat map A, the feature map F, and the joint vector map D are concatenated again and passed through another two-layer convolution to obtain the final joint position map L. The values corresponding to the points with the highest confidence are selected from the graphs represented by the X, Y, and Z coordinate axes as the values of the coordinate axes, thereby obtaining the final 3D hand posture coordinates.

[0041] The loss function L is as follows:

[0042] L=L A +L D +L L (5)

[0043] Among them L A Ensure that the regressed two-dimensional heat map A is close to the true labeled heat map A', as expressed in formula (6).

[0044] As shown in formulas (7) and (8), L D , L L The differences between the ground-truth joint vector map D and the predicted joint position map L are measured respectively. The ground-truth joint position map L' and the joint vector map D' are constructed by flattening the coordinates of the ground-truth joint positions and bone orientations to the size of the heatmap.

[0045]

[0046] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It is apparent to those skilled in the art that various changes, modifications, substitutions, and variations to these embodiments may be made without departing from the principles and spirit of the present invention, and these changes and modifications still fall within the scope of protection of the present invention.

Claims

1. A lightweight 3D hand pose estimation method based on RGB images, characterized by: The following steps are involved: Step 1: Stack multiple sandglass residual blocks in series to ensure the inductive bias characteristics of convolution for local feature modeling; The step 1 specifically includes: A sandglass residual block structure is introduced, in which the first and last convolutional layers in the main branch are depth-separable convolutional layers that keep the number of channels unchanged. The classic bottleneck block structure is simulated, and two consecutive point-by-point convolutions are stacked between the two depth-separable convolutions to first reduce and then increase the number of channels. Because 1×1 point-by-point convolutions will lead to the weakening of spatial information, the two 3×3 convolutions are placed outside the two 1×1 convolutions, and the two depth-separable convolutions are appropriately applied to higher-dimensional features, so that richer spatial information can be encoded to generate more expressive spatial information. represents the input features, Represents the output feature, which is expressed by the following formula: Among them, φ i,p ,φ i,d Represent the i-th point-by-point convolution and depth-wise convolution respectively, ensuring that the depth-wise convolution is processed in a high-dimensional space and obtains richer feature expressions; Step 2: Perform an "expand" operation on the features, then use the Transformer block structure to perform global feature modeling, then "fold" the features and perform feature fusion; The second step specifically includes: Use the sandglass block output as the input feature of dimension d Apply an n×n standard convolution layer, and then use a 1×1 point convolution layer to generate feature X L , so that the model has a global representation of spatial inductive bias, and the feature X L Expand into N non-overlapping patches, using It means that P = wh, and is the number of patches, h≤n and w≤n are the height and width of the patch respectively. According to formula (3), for each patch, that is, p∈{1,···,P}, the relationship between patches is encoded by Transformer to obtain X G (p)= Transformer (X U (p)),1≤p≤P (3) Foldable get Use point-wise convolution to transform X F Projected into a low-dimensional space and combined with feature X through a cascade operation, another n×n convolutional layer is used to fuse these connected features. U (p) Use convolution to encode local information of n×n region, X G (p) Encode the global information of the patch at position p, X G Each pixel in can encode information from all pixels in X, and the total effective receptive field is H×W; Step 3: Introduce a separable self-attention mechanism into the Transformer block used for global feature modeling to reduce the computational complexity of self-attention; The step three specifically includes: The structure of separable self-attention is similar to the multi-head self-attention used in Transformer, which takes the input X input Use input I, key K and value V to process respectively. Input branch I uses the value with weight W. I The linear layer converts X input Each d-dimensional token in is mapped to a scalar with weight W I Used to represent potential tokens and X input The distance between them is calculated to produce a k-dimensional vector, and then the softmax function is used to produce the context score Instead of calculating the attention score of each token for all k tokens, the separable self-attention only calculates the attention score for the potential token, which can reduce the time complexity of calculating the attention score from O(k 2 ) is reduced to O(k); According to formula (4), the context vector C v It is obtained by the context score Cs, where By having weight The key branch K will enter X input After linear projection into d-dimensional space, we get C v The context information encoded in X input All tokens in the same group are shared, with a usage weight of W V The value of branch V will be input to X input Linear projection to d-dimensional space Then the ReLU activation function is used to generate the output The resulting output is fed to a Another linear layer produces the final output Step 4: Convolve the feature map to obtain a 2D hand pose heatmap. Convolve the two together to obtain a joint vector map. Convolve them again to obtain a 3D hand pose heatmap, ultimately achieving 3D hand pose estimation.

2. The lightweight 3D hand pose estimation method based on RGB images according to claim 1, characterized in that: The step 4 specifically includes: First, the output feature map F of the backbone network is passed through a two-layer convolutional network to obtain a joint prediction confidence map containing 21 hand joint points, namely a two-dimensional heat map A. The feature map F is then concatenated with the two-dimensional heat map A and then passed through two layers of convolution to obtain a joint vector map D. Each pixel in the joint vector map D encodes the direction of each skeletal joint, representing a 3D vector from the parent joint to the child joint. The two-dimensional heat map A, the feature map F and the joint vector map D are concatenated again and passed through another two-layer convolution to obtain the final joint position map L. The values corresponding to the points with the highest confidence are selected from the graphs represented by the X, Y, and Z coordinate axes as the values of the coordinate axes to obtain the final 3D hand posture coordinates: The loss function L is as follows: L=L A +L D +L L (5) Among them L A Ensure that the regressed two-dimensional heat map A is close to the true labeled heat map A', as shown in formula (6). As shown in formulas (7) and (8), L D , L L The differences between the real labeled joint vector map D and the joint position map L and the predicted joint vector map D and joint position map L are measured respectively. The real labeled joint position map L' and joint vector map D' are constructed by flattening the coordinates of the real joint position and bone orientation to the size of the heat map.

Images