A feature enhancement based three-dimensional hand pose estimation method

Abstract

The application discloses a three-dimensional hand gesture estimation method based on feature enhancement, and relates to the field of image gesture estimation.The method solves the problems of insufficient extraction of hand features and poor adaptability to complex gestures in existing three-dimensional hand gesture estimation methods.The method uses a convolutional neural network to extract local features of a hand image, combines feature enhancement and attention fusion structure to obtain global semantic information of the image, and finally improves two-dimensional key point estimation to a three-dimensional gesture heat map through a three-dimensional gesture decoupling regression module, so that high-precision three-dimensional estimation of hand key points is realized.The application proposes a dynamic snake-shaped pixel discarding data enhancement strategy for a hand depth map, randomly discards pixels in a foreground area, introduces dynamic spatial disturbance, simulates the change characteristics of a real hand under the conditions of movement, occlusion and depth noise, and effectively improves the perception ability of the model to uncertainty and detailed structures.The application is also applicable to the field of computer vision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image pose estimation, and particularly relates to a three-dimensional hand pose estimation method based on feature enhancement. BACKGROUND

[0002] In recent years, with the development of deep learning convolutional neural networks, the task of hand pose estimation has also developed rapidly. In the past decade, many effective methods have been proposed. These methods can be divided into regression methods, detection methods and hybrid methods. Regression methods predict hand poses by directly regressing three-dimensional joint coordinates, such as the TriHorn-Net model proposed in the prior art, which improves the model's ability to capture joint positions and depth information by decomposing tasks and using complementary attention maps. Detection methods predict by generating joint heat maps, and the prior art uses convolutional neural networks (CNN) to estimate PCA coefficients of joints. Although the accuracy is improved, the computational overhead is large. Hybrid methods combine the advantages of regression and detection, and the A2J method in the prior art improves the accuracy and robustness of estimation by combining the advantages of the two methods. In addition, the prior art also demonstrates the feasibility of directly regressing 3D joint coordinates from RGB images, but due to the spatial information advantage of depth images, methods based on depth images are still more commonly used. Despite the progress, hand pose estimation still faces challenges such as hand self-occlusion, joint similarity and finger interaction, especially in handling complex scenes and dynamic interactions.

[0003] With the deepening of research, the field of hand pose estimation gradually focuses on the model's ability to express hand features, as well as its stability and generalization performance in complex environments. The high-resolution network (HRNet) described in the prior art achieves accurate fusion of cross-scale information for pose estimation by maintaining multi-resolution feature flow; and by introducing a regional feature integration strategy, local features of the palm and fingers are modeled and optimized in a partitioned manner, effectively improving the network's ability to express complex gesture structures. In addition, existing U-Net networks and feature pyramid structures (FPN) provide a classic architectural basis for multi-scale feature fusion and are widely used in hand pose estimation tasks. These network structures and fusion mechanisms based on feature enhancement effectively improve the model's ability to jointly model the palm and fingers, making hand pose estimation more stable in complex poses, occlusions and dynamic interactions.

[0004] From the above research, in an ideal case, a hand gesture capture method capable of realizing real-time human-computer interaction and other applications should realize direct feedback to the user under low-cost conditions. Using a low-cost depth image sensor can not only effectively reduce the cost and power consumption of image acquisition, but also achieve high estimation accuracy and low delay under the premise of low computational complexity, thereby completing high-precision, real-time three-dimensional hand gesture estimation. However, the three-dimensional hand gesture estimation task itself has high complexity and challenges, especially in the depth estimation process, the coupling relationship between the prediction stage and the feature level will interfere with the estimation of the two-dimensional hand gesture, making it difficult to fully utilize the key features, and thus affecting the accurate prediction of the three-dimensional gesture. In addition, a three-dimensional gesture estimation network structure with high precision usually accompanies higher computational complexity, and in the case of limited hardware computing power, its actual performance and application effect will be significantly affected. Therefore, establishing a network model based on feature enhancement to realize high-precision three-dimensional hand gesture key point estimation while maintaining low computational complexity is the research focus in the field of three-dimensional hand gesture estimation. SUMMARY

[0005] The present application is to overcome the problems in the prior art three-dimensional hand gesture estimation method, such as the difficulty in fully extracting hand features and the lack of adaptability to complex gestures. Therefore, a three-dimensional hand gesture estimation method based on feature enhancement is proposed.

[0006] To solve the above technical problems, the present application is realized by the following technical scheme:

[0007] Scheme one, the present application proposes a three-dimensional hand gesture estimation method based on feature enhancement, which comprises the following steps:

[0008] Step 1, input a single-channel depth image, use a 7x7 convolution layer to extract a preliminary feature map F 0;

[0009] Step 2, normalize the preliminary feature map F 0, enhance the nonlinearity through a ReLU activation function to obtain a primary feature map F a ;

[0010] Step 3, input the primary feature map F a to a residual bottleneck module for feature extraction, output a feature map and perform a 2x2 max pooling operation, output a feature map ;

[0011] Step 4, input the feature map output by the max pooling operation to an hourglass module, fuse multi-scale information, and obtain a feature map ;

[0012] Step 5, input the feature map to a two-dimensional joint estimation branch to generate a feature map , and obtain a heat map after normalization processing hmp , which is used to represent the joint distribution;

[0013] Step 6, use the coordinate matrix to weight the heat map hmp , calculate the two-dimensional joint coordinates UV0;

[0014] Step 7, input the feature map to an attention enhancement branch to obtain spatial attention weights F S ;

[0015] Step 8, multiply the spatial attention weights F S with the channel attention weights, and obtain an enhanced feature map through convolution F e1 ;

[0016] Step 9, linearly weighted fusion F out and the enhanced feature map F e1 to obtain a fused feature map and generate a fused attention map ;

[0017] Step 10, input the fused attention map to perform depth estimation to obtain a depth feature map F d ;

[0018] Step 11, perform weighted pooling on the depth feature map F d to obtain a weighted depth feature ;

[0019] Step 12, input the depth feature to perform depth value regression to output the depth coordinates D0 of each joint;

[0020] Step 13, concatenate the two-dimensional joint coordinates UV0 and the depth coordinates D0 to generate three-dimensional key point coordinates UVD, and complete the three-dimensional pose estimation.

[0021] Further, a preferred embodiment is provided, in step 4, the Hourglass module takes the bottleneck structure as the basic unit, each recursive layer contains three bottleneck sub-modules, each bottleneck sub-module is composed of 1x1, 3x3, 1x1 three convolution layers, and is matched with batch normalization and ReLU activation function, the input and output of each bottleneck sub-module are added element by element in the channel dimension to form a residual mapping.

[0022] Further, a preferred embodiment is provided, in step 4, the feature map output by the max-pooling operation is input into the Hourglass module to fuse multi-scale information and obtain a feature map F h The method is:

[0023] The feature map output by the max-pooling operation is subjected to a 2x2 max-pooling operation with a step of 2 to generate a down-sampling feature tensor F low1 The tensor F low1 is sequentially input into two bottleneck modules to obtain a tensor F low2 If the current recursive depth n>1, the tensor F low2 is passed as input to the Hourglass sub-module of the next layer of recursion; if n=1, the output tensor F low3 is directly obtained by passing through the third bottleneck module; the tensor F low3 is subjected to bottleneck operation to obtain a tensor F low4 In the bottom-up decoding path, the tensor F low4 is subjected to bilinear interpolation up-sampling with a factor of 2 to output a tensor F up The output corresponding to the encoding path is denoted as F up1 The tensors F up and F up1 are added element by element, and after four-layer recursion calculation, the Hourglass module outputs a feature map F h .

[0024] Further, a preferred embodiment is provided, in step 5, the calculation method of the heat map hmp obtained after normalization processing is:

[0025]

[0026] wherein, represents the normalized response value of the i-th joint at the spatial position (p, q), is the corresponding channel output value, and s is a learnable smoothing coefficient.

[0027] Further, a preferred embodiment is provided, wherein step 5 further comprises the steps of respectively imposing a supervised constraint on the two-dimensional heat map prediction and the three-dimensional coordinate regression result, and optimizing the network parameters using a weighted joint loss function.

[0028] Further, a preferred embodiment is provided, wherein step 9 calculates the linear weighted fusion F out and the enhanced feature map F e1 to obtain the fused feature map and generate the fused attention map The method is as follows:

[0029]

[0030] wherein, represents the corresponding element multiplication operation.

[0031] Further, a preferred embodiment is provided, wherein step 11 performs weighted pooling on the depth feature map F d to obtain the weighted depth feature The method is as follows:

[0032]

[0033] wherein, D represents the depth feature dimension.

[0034] Further, a preferred embodiment is provided, wherein step 13 concatenates the two-dimensional joint coordinate UV0 and the depth coordinate D0 to generate the three-dimensional key point coordinate UVD.

[0035]

[0036]

[0037] wherein, , respectively represent the horizontal and vertical coordinates of the i-th joint in the image plane.

[0038] Scheme three, a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by the processor to realize the steps of the method of scheme one.

[0039] In scheme four, a computer device comprises a memory and a processor, the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the method in scheme one.

[0040] The present application has the advantages of:

[0041] The feature enhancement-based three-dimensional hand pose estimation method provided by the present application introduces a feature enhancement and hierarchical fusion mechanism in the network structure design, and realizes efficient multi-scale feature modeling. Through the combination of the residual bottleneck module and the hourglass structure, the local details and global semantic features of the hand are fully captured. And in the feature extraction process, the channel and spatial joint attention mechanism is added to realize the adaptive weighting of key areas and the suppression of redundant information. At the same time, the depth features are dynamically enhanced by combining the local pixel enhancement processing, which significantly improves the estimation accuracy of the model in complex lighting, occlusion and deformation scenes.

[0042] The present application proposes a dynamic snake-shaped pixel dropout (DS-PixDropout) data enhancement strategy for hand depth maps. By randomly dropping pixels in the foreground area and introducing dynamic spatial disturbance, the changing characteristics of real hands under motion, occlusion and depth noise conditions are simulated, thereby effectively improving the model's perception ability of uncertainty and detailed structure. This strategy significantly enhances the generalization performance and application adaptability of the network, and has high engineering realizability and popularization value.

[0043] The present application is also applicable to the fields of computer vision and deep learning. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 The flowchart of the feature enhancement-based three-dimensional hand pose estimation method according to embodiment one.

[0045] Figure 2 The flowchart of the dynamic snake-shaped pixel dropout (DS-PixDropout) method according to embodiment one.

[0046] Wherein, (a) is a selection diagram, (b) is an offset diagram, and (c) is a deletion diagram. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all embodiments of the present application.

[0048] Embodiment one, the embodiment proposes a three-dimensional hand gesture estimation method based on feature enhancement, which specifically comprises the following steps:

[0049] Step 1: First, input the hand depth image into a 7x7 convolutional layer, and the input size is Bx1xHxW, where the input batch size B=64, the height H and the width W of the input depth image are 128. The convolution calculation performs a pixel-by-pixel convolution operation on the input single-channel depth image, generates a set of feature responses on each output channel, and outputs an initial feature map .

[0050] After convolution, the initial feature map is input into a batch normalization layer (the input size is Bx64xHxW, where the input batch size B=64, the height H and the width W of the input feature map are 128). The mean and variance standardization processing is independently performed on each channel, the scaling coefficient is initialized to 1.0, and the offset coefficient is initialized to 0. The batch normalization operation keeps as a numerical stability constant during calculation. After normalization, the tensor passes through a ReLU activation function, and the obtained primary feature map .

[0051] Step 2: The primary feature map is input into three residual bottleneck blocks for feature extraction (the input size is Bx64xHxW, where the input batch size B=64, the height H and the width W of the input feature map are 128). Each bottleneck block contains three convolutional operations, and the structure is 1x1 convolutional layer, 3x3 convolutional layer and 1x1 convolutional layer in turn. Each layer of convolutional operation is matched with batch normalization and ReLU activation function. The feature map obtained after passing through the three-layer convolutional operation is element-wise added to the primary feature map input into the module in the channel dimension to form a residual connection. The above structure is repeated three times to build three bottleneck modules, and each bottleneck block is connected in an output-input manner. The output feature map of the third bottleneck block is denoted as . After the calculation of the third bottleneck block is completed, a 2x2 max-pooling operation is performed on the feature map , the step is 2, and the padding is 0. The max-pooling operation selects the maximum value in each 2x2 window as the output, and the obtained feature map is .

[0052] Step 3: The feature map is input into a 1x1 convolutional layer (the input size is , where the input batch size B = 64, the height H and width W of the input feature map are 128) inputs the Hourglass module with a depth of 4 layers. The module is composed of a top-down encoding path and a bottom-up decoding path with four levels of recursion, and the network depth parameter depth = 4. The Hourglass module is implemented based on the Bottleneck structure as a basic unit. Each recursive layer contains three Bottleneck sub-modules, and each Bottleneck sub-module is composed of 1x1, 3x3, 1x1 three convolutional layers, and is matched with batch normalization and ReLU activation function. The input and output of each Bottleneck sub-module are added element by element in the channel dimension to form a residual mapping. In the top-down encoding path, the input feature map is subjected to a 2x2 max-pooling operation with a step size of 2 to generate a down-sampling feature tensor F low1 ; the F low1 is sequentially input into two Bottleneck modules to obtain a tensor F low2 ; if the current recursive depth n > 1, the F low2 is passed to the next layer of recursive Hourglass sub-module as input; if n = 1, the output tensor F low3 is directly obtained by passing through the third Bottleneck module; the F low3 is subjected to Bottleneck operation to obtain a tensor F low4 ; in the bottom-up decoding path, the F low4 is bilinearly interpolated and up-sampled with a factor of 2 to output a tensor F up , and the output of the encoding path is denoted as F up1 ; the F up is added element by element with the F up1 , and after four levels of recursion calculation, the Hourglass module outputs the feature map F h .

[0053] Step 4: The feature map extracted by the Hourglass module F h (input size The input batch size is B=64, and the input is a two-dimensional joint estimation branch. The two-dimensional joint estimation branch consists of two residual modules and a 1×1 convolutional layer. Each residual module contains three convolutional operations, with convolutional structures of 1×1, 3×3, and 1×1 respectively. F h A 1×1 convolutional layer is used for channel mapping. The number of convolutional kernels is set to J, with a value of J=14, corresponding to 14 hand key points. The convolutional output undergoes batch normalization and ReLU activation to generate feature maps. F out .

[0054] Step 5: Process the feature map F out (Input dimensions are) An adaptive spatial softmax operation is performed (with an input batch size B=64). The response values ​​for all spatial locations are normalized within each channel (corresponding to each joint). The softmax layer contains a trainable smoothing parameter s, initially set to 1.0. The softmax calculation within each channel is performed according to expression EQ1. Let represent the normalized response value of the i-th joint at spatial position (p, q). Here, s represents the output value of the corresponding channel, and s is a learnable smoothing coefficient. A tensor is generated after Softmax processing. hmp .

[0055] Expression EQ1:

[0056]

[0057] Step 6: Predefine the coordinate matrix when calculating the two-dimensional joint coordinates. , The size is 1×(h·w), where . The horizontal coordinate matrix elements are in the range [0, 63]. The vertical coordinate matrix elements are represented as [0, 63]. Within each joint channel, the heatmap and coordinate matrix are weighted and summed to calculate the X and Y coordinates of each joint, as shown in expressions EQ2 and EQ3. , Let X and Y represent the x and y coordinates of the i-th joint in the image plane, respectively. The X and Y coordinates of each joint are concatenated to form a two-dimensional joint coordinate matrix UV0. The output tensor UV0 is stored in 32-bit floating-point format and used for subsequent two-dimensional coordinate scaling operations and depth coordinate concatenation.

[0058] Expression EQ2:

[0059] Expression EQ3:

[0060] Step 7: Feature maps extracted by the Hourglass module F h Input attention enhancement branch, input size is (input size is , where the input batch size B = 64). The attention enhancement branch is composed of a channel attention module, a spatial attention module, and a channel adjustment convolution layer in sequence. This structure corresponds to the Convolutional Block Attention Module (CBAM) module in the implementation.

[0061] Step 8: In the channel attention module, the feature map F h (input size is , where the input batch size B = 64) is sequentially passed through an adaptive average pooling layer and an adaptive maximum pooling layer, and the output size of each pooling layer is B x 256 x 1 x 1. The two pooling results are input into a 1 x 1 convolutional network for feature mapping. The first convolutional layer has an input channel number of 256 and an output channel number of 16 (ratio = 16). The convolutional output is element-wise nonlinearly mapped by a ReLU activation function and then input into a second 1 x 1 convolutional layer. The second convolutional layer has an input channel number of 16 and an output channel number of 256. The average pooling channel output tensor F avg1 and the maximum pooling channel output tensor F max1 are calculated respectively. F sum The channel weighted feature tensor F C is added in the channel dimension, and the Sigmoid activation function is input for normalization to obtain the channel attention weight tensor

[0062] Step 9: In the spatial attention module, the channel weighted feature map (input size is , where the input batch size B = 64) is input into an adaptive spatial attention structure. The average pooling and maximum pooling results are calculated in the spatial direction to obtain tensors F avg2 and F max2 . Along the channel dimension, the tensors F avg2 and F max2 are concatenated to obtain the tensor F concat . The tensor F concatThe input is a convolutional layer with a kernel size of 7×7. The convolutional output is then processed by a sigmoid activation function to obtain the spatial attention weight tensor. F S .

[0063] Step 10: Convert the spatial attention weight tensor F S (Input dimensions are) The input batch size (B=64) and the channel attention weight tensor (Input dimensions are) The feature map is obtained by multiplying the elements (with a batch size of B=64) element by element, resulting in a dual-weighted feature map of channel and space. F SC = F S ⊙ F C ⊙ represents element-wise multiplication. After channel attention and spatial attention calculations are completed, the double-weighted feature map is... F SC The input channels of the adjusted convolutional layer are matched to the number of channels in subsequent modules. This convolutional layer has a kernel size of 1×1, 256 input channels, and J (J=14) output channels, enhancing the feature map through convolution. F e1 .

[0064] Step 11: Use a 1x1 convolution to... F e1 The number of channels J changes from 14 to 256, making it consistent with... F out The number of channels is consistent. Set the learnable parameter tensor. (Dimensions are 1×J×1×1), calculated using expression EQ4, for the branch output estimated through two-dimensional joints. F out (Input dimensions are) (where the input batch size B=64) and the output of the attention enhancement branch F e1 (Input dimensions are B×J×64×64, where the batch size B=64 and the number of channels J=256) Linear weighted fusion is performed to obtain the fused feature map. Where ⊙ represents element-wise multiplication. The merged features... The input is normalized using an adaptive spatial Softmax layer to obtain the fused attention map. .

[0065] Expression EQ4:

[0066] Step 12: Extract the feature map using the Hourglass module.F h (Input dimensions are B×256×64×64, where the input batch size B=64) Input depth estimation branch. The depth estimation branch consists of two residual modules, a 1×1 convolutional layer, and a batch normalization layer. The output depth feature tensor is defined as... F d For depth feature tensors F d Perform local pixel enhancement processing. First, convert the depth feature tensor... F d (Input size is B×64×64×64, where input batch size B=64) Obtain attention-weighted feature maps using the Channel-Spatial Attention Module (CBAM). F e2 Then, the feature map is processed in the spatial dimension. F e2 Region segmentation enhancement is performed. The feature map is divided into two sub-regions along the height direction and two sub-regions along the width direction, each sub-region having a size of 32×32. Channel attention and spatial attention are calculated sequentially for each sub-region, and the processed sub-regions are concatenated along the width dimension to form a local combined tensor. F x1 and F x2 Then, they are spliced ​​together in the height dimension to form a tensor. F x3 Based on this, the input depth feature tensor F d Perform channel and spatial attention calculations again to obtain global augmented features. F x4 .Will F x3 and F x4 Element-wise addition generates the final enhanced depth augmentation feature map. .

[0067] Step 13: Using a fused attention map (Dimensions are B×64×64×64, where the input batch size B=64) are weights, for (Input dimensions are B×64×64×64, where the input batch size B=64) Perform weighted pooling. Calculate the weighted pooling using the expression EQ5. and The weighted depth feature map is obtained by multiplying element-wise along the spatial dimension and summing along the h×w dimension. .

[0068] Expression EQ5:

[0069] Step 14: The weighted depth feature map The input (input size B x J x D, input batch size B = 64, channel number J = 14, where the depth feature dimension D = 256) is input into a shared linear layer for depth value regression. The shared linear layer is composed of a fully connected layer, and the weight matrix of the fully connected layer is defined as W e R^ (D x 1), and the bias term is defined as b e R^ (1 x 1). The bias parameter b is initialized to 0, and the weight matrix and the bias term participate in the gradient update in the training stage. The fully connected layer will perform one-dimensional weighted summation and bias offset operation on the depth feature vector of each joint, and the output tensor D 0 (input size B x J x 1, input batch size B = 64, channel number J = 14), and each channel element represents the estimated coordinate value of a joint in the relative depth direction.

[0070] Step 15: The two-dimensional joint coordinate tensor UV0 (input size B x J x 2, input batch size B = 64, joint number J = 14) is subjected to a scale mapping operation to match the original spatial resolution of the input depth image. Each joint contains two plane coordinate components. Each element of the two-dimensional coordinate tensor UV0 corresponds to the horizontal coordinate U (b, j) and the vertical coordinate V (b, j) of the joint in the image plane, respectively. The original network feature map space size is 64 x 64, while the spatial resolution of the input image is 128 x 128, so a fixed scale factor scale factor = 2 is set. After calculation, the scaled two-dimensional joint coordinate tensor UV'0 is formed, and each element represents the plane position of the joint in the original coordinate system of the input image.

[0071] Step 16: The mapped two-dimensional joint coordinate tensor UV'0 (input size B x J x 2, input batch size B = 64, joint number J = 14) is concatenated with the depth coordinate tensor D0 e R^ (B x J x 1) along the last dimension to form a three-dimensional key point coordinate tensor UVD. Each sample contains the three-dimensional spatial coordinates of 14 hand key points.

[0072] Step 17: The three-dimensional key point coordinate tensor UVD (input size B x J x 3, input batch size B = 64, joint number J = 14) is taken as the final output result of the network. Each sample contains the three-dimensional coordinates of 14 key points, where the first dimension is the batch index, the second dimension is the key point number, and the three elements of the third dimension correspond to the (X, Y, Z) three-dimensional coordinate components in turn. The output result is used to represent the pose structure of the hand in three-dimensional space.

[0073] In the model analysis stage, in order to facilitate visualization and performance evaluation, the enhanced feature map F e1 , the fused attention feature map and weighted depth feature map .

[0074] Step 18: During the training phase, supervised constraints are applied to the 2D feature map prediction and 3D coordinate regression results, and a weighted joint loss function is used to optimize the network parameters. To enhance the robustness of the model under occlusion, illumination changes, and noise interference scenarios, a dynamic snake-like pixel dropout (DS-PixDropout) data augmentation strategy is introduced during training.

[0075] DS-PixDropout randomly selects a set of pixels from the effective region of the hand image based on the set perturbation ratio parameter c∈[0,1]. The number of pixels selected is shown in expression EQ6. The selected hand region is denoted as set s, and N... f N represents the total number of pixels that can be disturbed. s N represents the number of pixels selected for enhancement. For the set, N is... s For each selected pixel (x, y), see [link to details]. Figure 2 (a) The selection diagram shows that random offsets Δx and Δy are applied in the horizontal and vertical directions, respectively. The offset values ​​follow a uniform distribution U(-a,a). See details. Figure 2 The offset diagram described in (b) illustrates this. To ensure all pixels remain within the image boundary, if the offset coordinates exceed the image range [0, W-1] or [0, H-1], a cropping operation is performed, truncating the excess portion to the image boundary. The set of cropped coordinates is denoted as... This indicates the effective region after the disturbance; see details below. Figure 2 (c) The deletion diagram is shown. For s belonging to For pixels in s that do not belong to the same pixel value, their pixel values ​​remain unchanged; for pixels in s that do not belong to the same pixel value, their pixel values ​​remain unchanged. The pixels are replaced with a constant background value V. The enhanced image is defined as follows: The calculation is performed according to expression EQ7. Here, p represents the pixel corresponding to any coordinate, and V represents the constant value assigned to the background (non-hand) pixels. Figure 2 The blue arrow in the image represents N. f The background color is filled in areas that have not been offset.

[0076] Expression EQ6:N s =c×N f

[0077] Expression EQ7:

[0078] The application realizes three-dimensional hand posture estimation according to feature enhancement network structure, through effective fusion of global and local information, the method can better capture the details of hand posture, especially in the case of processing complex gesture changes and self-occlusion, showing superior robustness. Combined with the DS-PixDropout data enhancement method, the adaptability of the network to slender structures such as fingers is further enhanced, and the diversity and generalization ability of the hand depth map data set are improved.

[0079] Those skilled in the art can understand that the above description is only preferred embodiments of the present application, and the features described in each embodiment and / or technical solution of the present disclosure can be combined or combined, even if such combination or combination is not explicitly described in the present disclosure. Not for limiting the present application, although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent replacement for part of the technical features, any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0080] Although the preferred embodiments of the present application have been described, those skilled in the art can make further changes and modifications to these embodiments once they know the basic creative concept. Therefore, the appended technical solutions are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application. Obviously, those skilled in the art can make various modifications and changes to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and changes of the present application fall within the scope of the present application and its equivalent technology, the present application also intends to include these modifications and changes.

Claims

1. A feature enhancement based three-dimensional hand pose estimation method, characterized in that, The method comprises the following steps: Step 1, input single-channel depth image, use 7x7 convolutional layer to extract preliminary feature map F 0; Step 2, normalizing the primary feature map F 0, enhancing nonlinearity by a ReLU activation function, to obtain a primary feature map F a ; Step 3, the primary feature map is obtained F a The input residual bottleneck module performs feature extraction, and outputs a feature map and performs a 2x2 max pooling operation, and outputs a feature map ; Step 4, the feature map output by the max pooling operation inputting an hourglass module, fusing multi-scale information, and obtaining a feature map ; Step 5, generating a feature map inputting a two-dimensional joint estimation branch to generate a feature map , and obtaining a heat map after normalization processing hmp , for representing joint distribution; Step 6, using the coordinate matrix to weight the heat map hmp , calculate the two-dimensional joint coordinates UV0; Step 7, feature maps input attention enhancement branch, to obtain spatial attention weights F S ; Step 8, multiply the spatial attention weight F S with the channel attention weight, and obtain the enhanced feature map by convolution F e1 ; Step 9, linearly weighted fusion F out with enhanced feature maps F e1 to obtain a fused feature map and generate a fused attention map ; Step 10, input fusion attention map perform depth estimation to obtain a depth feature map F d ; Step 11, depth feature map F d weighted pooling is performed to obtain a weighted depth feature ; Step 12, input depth features Carrying out depth value regression, output depth coordinates D0 of each joint; Step 13, splice the two-dimensional joint coordinates UV0 and the depth coordinates D0 to generate three-dimensional key point coordinates UVD, and complete the three-dimensional pose estimation.

2. The feature enhancement based three-dimensional hand pose estimation method of claim 1, wherein, The Hourglass module in step 4 takes the bottleneck structure as a basic unit, each recursive layer contains three bottleneck sub-modules, each bottleneck sub-module is composed of 1x1, 3x3, 1x1 three layers of convolution, and is matched with batch normalization and ReLU activation function, the input and output of each bottleneck sub-module are added in the channel dimension to form a residual mapping.

3. The feature enhancement based three-dimensional hand pose estimation method of claim 1, wherein, The feature map output by the max pooling operation in step 4 An input hourglass module fuses multi-scale information to obtain a feature map F h The method is as follows: feature maps output by the max-pooling operation perform a 2x2 max-pooling operation with a stride of 2 to generate a down-sampled feature tensor F low1 ; input the tensor F low1 to two Bottleneck modules in sequence to obtain a tensor F low2 ; if the current recursion depth n > 1, input the tensor F low2 to the Hourglass sub-module of the next layer of recursion; if n = 1, directly pass the tensor through a third Bottleneck module to obtain an output tensor F low3 ; perform a Bottleneck operation on the tensor F low3 to obtain a tensor F low4 ; in the bottom-up decoding path, perform bilinear interpolation up-sampling on the tensor F low4 with a factor of 2 to output a tensor F up ; the output corresponding to the encoding path is denoted as F up1 ; add the tensors F up and F up1 element by element, and after four layers of recursion are calculated, the Hourglass module outputs a feature map F h .

4. The feature enhancement based three-dimensional hand pose estimation method of claim 1, wherein, The heat map obtained after normalization in step 5 hmp The calculation method is: wherein, represents the normalized response value of the i-th joint at the spatial position (p, q), is the corresponding channel output value, and s is a learnable smoothing coefficient.

5. The feature enhancement based three-dimensional hand pose estimation method of claim 1, wherein, Step 5 also includes the steps of respectively imposing supervision constraints on the two-dimensional heat map prediction and the three-dimensional coordinate regression result, and adopting a weighted joint loss function to optimize the network parameters.

6. The feature enhancement based three-dimensional hand pose estimation method of claim 1, wherein, The linearly weighted fusion is calculated in step 9 F out The enhanced feature map is obtained F e1 , and a fusion attention map is generated The method is as follows: Wherein, represents the corresponding element multiplication operation.

7. The feature enhancement based three-dimensional hand pose estimation method of claim 1, wherein, The method for performing weighted pooling on the deep feature map in step 11 F d The method for performing weighted pooling on the deep feature map in step 11 is as follows: Wherein, D represents the depth feature dimension.

8. The feature enhancement based three-dimensional hand pose estimation method of claim 1, wherein, The method for splicing the two-dimensional joint coordinates UV0 and the depth coordinates D0 to generate three-dimensional key point coordinates UVD in step 13 is: wherein, , respectively denote the horizontal and vertical coordinates of the i-th joint in the image plane.

9. A computer storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to realize the method of any one of claims 1-7.

10. A computer device, comprising: Comprise: Memory, processor and computer program stored on the memory and executable on the processor, the processor executes the program to realize the method of any one of claims 1-7.

Images