A monocular 3D hand gesture estimation method based on fully convolutional neural network
Semantic features are extracted through a fully convolutional neural network and restored high-resolution feature map, the mesoscale uncertainty and network complexity problems of monocular three-dimensional gesture estimation are solved, and efficient three-dimensional gesture estimation and real-time operation are achieved.
Patent Information
- Application Number
- CN202210397216.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-15
- Publication Date
- 2024-12-27
- Estimated Expiration
- 2042-04-15
AI Technical Summary
The existing monocular three-dimensional gesture estimation methods have insufficient accuracy and generalization capabilities, especially the color image-based methods are easily affected by scale uncertainty, and the neural network structure is complex and difficult to achieve real-time operation in low-resource scenarios.
Using a monocular three-dimensional gesture estimation method based on a fully convolutional neural network, semantic rich features are extracted through continuous convolution and downsampling, and combined with upsampling to restore high-resolution feature maps, predicting the two-dimensional coordinates and relative depth of hand key points , simplify network design and adapt to different resource conditions.
It effectively improves the accuracy and generalization capabilities of three-dimensional gesture estimation, simplifies network design, realizes real-time operation in low-resource scenarios, and reduces the difficulty of algorithm maintenance.
Smart Images
Figure CN114882524B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and in particular to a monocular three-dimensional hand gesture estimation method based on a fully convolutional neural network. Background Art
[0002] The high degree of freedom and natural friendliness of hands make them an important research object for intelligent human-computer interaction. The visual gesture interaction system can free users from the dependence on human-computer interaction middleware and directly use gestures to interact with real or virtual scenes and operate smart terminals, greatly improving the user's consistency and convenience. In recent years, technology giants such as Google, Microsoft and Facebook have invested a lot of resources in the research of intelligent interactive wearable terminals based on augmented reality (AR), virtual reality (VR) and mixed reality (MR), and visual gesture interaction is one of the core technologies.
[0003] Hand gesture estimation refers to the process of predicting the position of the key points of the hand from the hand image. Among them, monocular 3D hand gesture estimation requires predicting the 3D position of the key points of the hand in the imaging space from a single hand color image or depth map, which is a very challenging task. As an important part of visual gesture interaction, hand gesture estimation can help computers capture the relative position relationship between the hand and other real objects or virtual objects in the scene, so as to analyze and predict the changes in the real scene, or give corresponding feedback to the scene in the virtual world. Monocular 3D hand gesture estimation has broad application prospects and practical value because of its low requirements on input images and high flexibility of the overall method.
[0004] The prior art closest to the present invention:
[0005] Monocular 3D gesture estimation: The monocular 3D gesture estimation task requires predicting the 3D spatial coordinates of the key points of the hand from a single input hand image, which is an advanced visual understanding task. The high degree of freedom of hand posture, self-occlusion characteristics, and the 3D scale uncertainty of a single image make this task very difficult. Existing monocular 3D gesture estimation methods based on deep learning can be divided into color image-based methods and depth map-based methods based on the input image type. Among them, the depth map-based method can utilize the depth information of the hand surface, which alleviates the scale uncertainty problem to a certain extent and achieves good results. The hand color image-based method can be divided into learning-based methods and model-based methods according to the implementation process. The former is trained by designing a complex neural network and combining a large amount of image data with precise 3D key point annotations, so that the neural network can directly regress the coordinates of the key points from the image; while the model-based method introduces a parameterized hand model, uses a neural network to regress the parameters of the hand model, and performs semi-supervised training through hand images, which can achieve a higher upper limit of accuracy.
[0006] Disadvantages of existing technology:
[0007] 1. Although the existing monocular 3D gesture estimation method based on depth map has high overall accuracy, it relies on high-quality depth map. However, the imaging conditions of depth map are harsh and easily affected by scene interference, which greatly limits the application scenarios of monocular gesture estimation.
[0008] 2. The existing learning-based 3D hand pose estimation methods for monocular color images use a relatively complex neural network structure, some operators are not compatible with the mainstream deployment framework, and are highly dependent on the training of a large amount of accurately labeled data. The results are easily affected by scale uncertainty and have poor generalization ability. (Yang L, Li S, Lee D, et al. Aligning latent spaces for 3D hand pose estimation [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2019: 2335-2343.).
[0009] 3. Existing model-based 3D hand gesture estimation methods from monocular color images rely on neural networks to regress the parameters of the hand model and perform indirect supervision through hand color images, which makes the overall method difficult to train and the training effect is easily affected by the model parameter initialization and image indirect supervision. This increases the difficulty of developing practical applications and is not conducive to application expansion (Boukhayma A, Bem R, Torr PH S. 3D hand shape and pose from images in the wild [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019: 10843-10852.).
[0010] 4. The existing monocular color image 3D gesture estimation method only considers the improvement of accuracy in network structure design, but does not consider the operation requirements under different resource conditions, making it difficult to achieve real-time operation in low-resource scenarios. Summary of the invention
[0011] In order to solve the problems that the monocular 3D gesture estimation method based on hand color image is easily affected by scale uncertainty factors, the 3D gesture estimation neural network structure is complex and has poor scalability, and the existing 3D gesture estimation neural network has low real-time operation efficiency, the present invention proposes a monocular 3D gesture estimation method based on a fully convolutional neural network. The network of the present invention completes the task of predicting the image coordinates and relative depth values of hand key points from the input hand image. The input image is first subjected to continuous convolution and downsampling to obtain a feature map with rich semantics but low spatial resolution, and then the semantic features are further refined through continuous convolution and upsampling to restore the high spatial resolution of the feature map. Finally, the output unit predicts the final two-dimensional coordinates of the key points and their relative depth from the high-resolution feature map.
[0012] The present invention is achieved by at least one of the following technical solutions.
[0013] A monocular 3D hand gesture estimation method based on a fully convolutional neural network comprises the following steps:
[0014] S1, obtaining a hand image and preprocessing the image;
[0015] S2. construct a fully convolutional 3D gesture estimation network and train the fully convolutional 3D gesture estimation network;
[0016] S3, input the preprocessed image into the fully convolutional 3D gesture estimation network to predict the final 2D coordinates of the key points and the relative depth of each key point;
[0017] S4. Post-process the predicted two-dimensional coordinates of the key points and the relative depth to calculate the three-dimensional coordinates of the hand key points.
[0018] Furthermore, the preprocessing includes image scaling and image filling.
[0019] Furthermore, the fully convolutional 3D gesture estimation network includes the following units:
[0020] An input compression unit, used to extract local rough features in the input image while reducing the input image resolution;
[0021] Basic convolution unit, used to extract basic features;
[0022] The downsampling unit is used to spatially downsample the feature map to improve the receptive field of the feature map;
[0023] The upsampling unit is used to spatially upsample the feature map, and cooperate with the lateral connection output from the downsampling unit to restore the spatial information of the feature map and further enrich the semantic features;
[0024] The output unit is used to extract information from the final high-resolution feature map and predict the final two-dimensional coordinates of the hand key points and the normalized relative depth.
[0025] Furthermore, the basic convolution unit comprises:
[0026] 1) 7×7 channel-by-channel convolution: channel-by-channel convolution with a large convolution kernel, keeping the number of input feature map channels single;
[0027] 2) Layer normalization operation: convert the mean and variance of all pixel values of the input feature map into parameters involved in training;
[0028] 3) 1×1 convolution: used to enlarge or reduce the number of feature map channels by 4 times;
[0029] 4) Activation function: The GELU function is used to calculate the output activation value of the convolutional layer;
[0030] 5) Residual connection: Add the unit input and the activated output as the final unit output.
[0031] Furthermore, the output unit includes:
[0032] 1) 2D coordinate prediction branch: A 3×3 convolution kernel is used to extract features from the feature map to obtain a heat map of the key points of the hand, and the highest response position is obtained through a differentiable maximum index operation (soft-argmax), which is the predicted 2D coordinate of the key point.
[0033] 2) Relative depth prediction branch: A 3×3 convolution kernel is used to extract features from the feature map to obtain the hidden depth map of the hand key points, and the normalized relative depth value of each key point is obtained through global spatial average pooling (mean).
[0034] Furthermore, the training of the fully convolutional 3D gesture estimation network includes the following steps:
[0035] 21) Image preprocessing: scaling and padding the input images involved in training, and performing data augmentation;
[0036] 22) Annotation preprocessing: During the training process, the processing of the hand key point annotation mainly includes: modifying the corresponding key point two-dimensional annotation in accordance with the image rotation data enhancement method, and converting the absolute depth annotation into a normalized relative depth annotation. The specific operation is to subtract the depth annotation of all key points from the depth annotation of the root key point, and divide the result by the hand reference length;
[0037] 23) Network forward reasoning: Input the processed image into the fully convolutional 3D gesture estimation network to obtain the predicted 2D coordinates of the key points and relative depth;
[0038] 24) Loss calculation: Input the output of network forward reasoning and the preprocessed annotations into the loss function to obtain the loss function value;
[0039] 25) Gradient back propagation: Calculate the gradient of the loss function value relative to the network parameters of the fully convolutional 3D gesture estimation network, and use the back propagation algorithm to update the network parameters;
[0040] 26) Iterative training: Repeat the above steps for the input images in batches until the loss function value no longer decreases, and the training of the fully convolutional 3D gesture estimation network is completed.
[0041] Furthermore, the fully convolutional 3D gesture estimation network training process uses the Smooth-L1 loss function to calculate the error between the two-dimensional coordinates and relative depth of the key points predicted by the network and the annotations, and calculates the gradient of the error relative to each network parameter, and updates the network parameters through the back-propagation algorithm.
[0042] Furthermore, the data enhancement used in the image preprocessing during the training of the fully convolutional 3D gesture estimation network includes:
[0043] 1) Random image rotation: randomly rotate the input image around the center point from -180° to 180°;
[0044] 2) Random image flipping: randomly flip the input image horizontally and vertically;
[0045] 3) Random color transformation: The HSV channel of the input image is scaled by random values in the range of 75% to 125%, 50% to 150%, and 50% to 150% respectively;
[0046] 4) Random noise enhancement: A Gaussian noise with a mean of 0 and a variance of 0 to 0.1 is applied to each position in the input image with a probability of 50%.
[0047] Furthermore, the post-processing includes:
[0048] 1) Convert the normalized relative depth of key points in the output of the fully convolutional 3D gesture estimation network into the absolute depth of key points;
[0049] 2) Use the two-dimensional coordinates of the key points in the image output by the fully convolutional three-dimensional gesture estimation network and the absolute depth of the key points after transformation to calculate the three-dimensional coordinates of the key points.
[0050] Furthermore, the specific process of calculating the three-dimensional coordinates of the key points is:
[0051]
[0052] Among them, (X i ,Y i ,Zi ) is the three-dimensional coordinate of the i-th key point, (u i ,v i ) is the 2D coordinate of the i-th key point output by the fully convolutional 3D gesture estimation network, z i is the absolute depth value of the i-th key point, and K is the intrinsic parameter matrix of the imaging device.
[0053] Compared with the prior art, the present invention has the following beneficial effects:
[0054] 1. The present invention can effectively deal with the scale uncertainty problem in monocular 3D gesture estimation by decoupling the hand scale information from the depth prediction process of the hand key points by the neural network. In practical applications, by obtaining accurate prior scale information, the present invention can accurately restore the actual depth of the hand key points in the scene relative to the imaging device, effectively improving the upper limit of the accuracy of the 3D gesture estimation method and the generalization ability of the scene.
[0055] 2. The present invention constructs a 3D gesture estimation network based on a full convolution structure through basic convolution operations, simplifies the network design process of 3D gesture estimation applications based on deep learning, and supports the optimization and acceleration of the network by most existing deployment frameworks. At the same time, the network can proportionally adjust the number of its own modules and the channels of the convolution layer to adapt to application scenarios with different resource conditions, reducing the difficulty of subsequent maintenance and optimization of the algorithm.
[0056] 3. The present invention introduces the commonly used lightweight design into the 3D gesture estimation network structure, which greatly improves the operating efficiency of the 3D gesture estimation network, enabling the present invention to achieve real-time 3D gesture estimation in low-resource scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] Figure 1 This is a structural diagram of the fully convolutional 3D gesture estimation network in this embodiment;
[0058] Figure 2 This is a structural diagram of an input compression unit in a fully convolutional 3D gesture estimation network of this embodiment;
[0059] Figure 3 : is a structural diagram of a basic convolution unit in a fully convolutional 3D gesture estimation network of this embodiment;
[0060] Figure 4 This is a structural diagram of an upsampling unit in a fully convolutional 3D gesture estimation network in this embodiment;
[0061] Figure 5 This is a structural diagram of a downsampling unit in a fully convolutional 3D gesture estimation network in this embodiment;
[0062] Figure 6: is a structural diagram of an output unit in a fully convolutional three-dimensional gesture estimation network of this embodiment;
[0063] Figure 7 This is a training and reasoning flowchart of the fully convolutional 3D gesture estimation network in this embodiment. DETAILED DESCRIPTION
[0064] The present invention will be further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.
[0065] Example 1
[0066] The fully convolutional 3D gesture estimation network proposed in the present invention can complete the implementation process of monocular 3D gesture estimation application, which is mainly divided into the training process and reasoning process of the neural network. Among them, the neural network can be used for testing and reasoning after the training is completed. Figure 7 As shown, a monocular 3D gesture estimation method based on a fully convolutional neural network includes the following steps:
[0067] S1. Construct a fully convolutional three-dimensional gesture estimation network and train the fully convolutional three-dimensional gesture estimation network; the fully convolutional three-dimensional gesture estimation network includes the following units:
[0068] Input compression unit: Its composition is as follows Figure 2 As shown in the figure, the input image is first processed through a convolution layer with a convolution kernel size of 4×4 and a stride of 4 to obtain a compressed feature map, and then the mean and variance of the compressed feature map are adjusted to learnable parameters through an intra-layer normalization operation.
[0069] Basic convolution unit: The basic unit for extracting features, which is composed of Figure 3 As shown in the figure, it first uses a 7×7 channel-by-channel convolution to process the input C channel feature map, then undergoes in-layer normalization, and then uses a 1×1 convolution to increase the number of feature map channels to 4C, and then processes it through the GELU activation function, and finally uses a 1×1 convolution to restore the number of feature map channels to C, and then sums it with the unit input and outputs it.
[0070] Downsampling unit: responsible for spatial downsampling of feature maps, its structure is as follows Figure 5 As shown in the figure, the input feature map is first normalized within the layer, and then processed by a convolution layer with a convolution kernel size of 2×2 and a stride of 2 to obtain a feature map with half the spatial resolution.
[0071] Upsampling unit: responsible for spatial upsampling of feature maps, its structure is as follows Figure 4 As shown in the figure, the input feature map is first normalized within the layer, and then deconvolution is performed with a convolution kernel size of 2×2 and a step size of 2 to increase the spatial resolution to twice the original, thereby achieving spatial upsampling of the feature map.
[0072] Output unit: responsible for extracting information from the final high-resolution feature map to predict the final two-dimensional coordinates and relative depth of the hand key points. Its structure is as follows: Figure 6 As shown in the figure, one branch converts the feature map into a heat map of each key point of the hand through the same convolution operation with a convolution kernel size of 3×3 and a step size of 1, and obtains the highest response value in the heat map through a differentiable maximum index operation (soft-argmax), which is the predicted two-dimensional coordinate of the key point; the other branch converts the feature map into a latent depth map of each key point through the same convolution operation with a convolution kernel size of 3×3 and a step size of 1, and then obtains the normalized relative depth value of each key point through global spatial average pooling (mean).
[0073] like Figure 1 As shown in Figure 1, in terms of overall structural design, the fully convolutional 3D gesture estimation network is mainly divided into a downsampling stage and an upsampling stage. In the downsampling stage, the input image is first compressed using an input compression unit, and then processed by three serially connected downsampling feature extraction modules. The downsampling feature extraction module consists of N i The number of feature channels is C i The basic convolution unit and a downsampling unit are connected in series, and the spatial resolution of the feature map i=1,2,3 is sorted from large to small. The processed feature map is processed by N 4 The number of channels is C 4 The basic convolution unit is further processed and then enters the upsampling stage. In the upsampling stage, it first passes through N 4 The image is processed by three basic convolution units and then by three upsampling feature extraction modules in series. The upsampling feature extraction module consists of N i The number of feature channels is C i The basic convolution unit and an upsampling unit are connected in series, which also receives and sums the output of the "downsampling feature extraction module" in the downsampling stage with the same spatial resolution, and finally obtains the predicted two-dimensional coordinates of the hand key points and the normalized relative depth value after being processed by the input unit. 1 、N 2 、N 3 、N 4 , C 1 , C 2 , C 3 , C 4 Taking 1, 1, 1, 1, 48, 96, 192, and 384 respectively, the obtained fully convolutional 3D gesture estimation network is suitable for application scenarios with high requirements on operating efficiency.
[0074] The training of a fully convolutional 3D hand pose estimation network consists of the following steps:
[0075] 1) Image preprocessing: The input images to be trained are padded to 1:1 and scaled to 256×256, and data augmentation operations including random image rotation, random image flipping, random color transformation, and random noise augmentation are performed.
[0076] 2) Annotation preprocessing: During the training process, the processing of the hand key point annotation mainly includes: modifying the corresponding key point 2D annotation in accordance with the image rotation data augmentation method, and converting the absolute depth annotation into a normalized relative depth annotation. The specific operation is to subtract the depth annotation of all key points from the depth annotation of the root key point, and divide the result by the hand reference length;
[0077] 3) Network forward reasoning: The processed image is input into the fully convolutional 3D gesture estimation network to obtain the predicted 2D coordinates of the key points and relative depth.
[0078] 4) Loss calculation: The output of the network forward reasoning and the preprocessed annotations are input into the loss function SmoothL1 function to calculate the loss function value.
[0079] 5) Gradient back propagation: Calculate the gradient of the loss function value relative to the network parameters of the fully convolutional 3D gesture estimation network, and use the back propagation algorithm to update the network parameters.
[0080] 6) Iterative training: Repeat the above steps for the input images in batches until the loss function value no longer decreases, and the training of the fully convolutional 3D gesture estimation network is completed.
[0081] S2. Obtain a hand image and preprocess the image, specifically padding the input image to 1:1 and scaling it to 256×256 resolution.
[0082] S3. Input the preprocessed image into the trained fully convolutional 3D gesture estimation network to obtain the 2D coordinates of the hand key points and the normalized relative depth values.
[0083] S4, post-process the 2D coordinates of the hand key points and the relative depth values of each key point predicted by the full convolutional 3D gesture estimation network, calculate the 3D coordinates of the key points, and complete the gesture estimation task. The post-processing process includes:
[0084] 1) Convert the normalized relative depth to absolute depth. First, use the provided hand reference length (which can be obtained by other methods in the scene in practice and is related to scene factors) to substitute into the following equation:
[0085]
[0086] Among them, l ref represents the reference length of the hand, K is the intrinsic parameter matrix of the imaging device, z ris the absolute depth of the root key point, (u r ,v r ) is the two-dimensional coordinate of the root key point predicted by the network, (u m ,v m ) is the two-dimensional coordinate of the middle finger palm node predicted by the network (i.e., another joint point related to the reference length of the hand), d m is the normalized relative depth value of the middle finger palm node predicted by the network. Solving the above equations, we can get the absolute depth z of the root key point r , the absolute depth of all key points can be restored by combining the normalized relative depth values of each key point predicted by the network:
[0087]
[0088] in, represents the normalized relative depth value of the i-th key point predicted by the network, z i is the corresponding absolute depth value.
[0089] 2) Calculate the three-dimensional coordinates of key points:
[0090]
[0091] Among them, (Z i ,Y i ,Z i ) is the three-dimensional coordinate of the i-th key point, (u i ,v i ) is the two-dimensional coordinate of the i-th key point output by the network, z i is the absolute depth value of the i-th key point, and K is the intrinsic parameter matrix of the imaging device.
[0092] The fully convolutional 3D gesture estimation network has the following characteristics: according to the application scenario requirements, the number of series connections N of the basic convolutional units and the intra-layer channels C of the basic convolutional units can be adjusted, thereby changing the depth and width of the fully convolutional 3D gesture estimation network, affecting the performance of the overall method, and thus being deployed in application scenarios with different resource conditions.
[0093] Example 2
[0094] Change N in the network structure design parameters 1 、N 2 、N 3 、N 4 , C 1 , C 2 , C 3 , C 4They are 2, 2, 2, 2, 96, 144, 216, and 324 respectively. The obtained fully convolutional 3D gesture estimation network is suitable for application scenarios with high requirements on operating efficiency. The rest of the process is the same as that in Example 1, which is suitable for scenarios with relatively balanced requirements on operating efficiency and accuracy.
[0095] Example 3
[0096] Change N in the network structure design parameters 1 、N 2 、N 3 、N 4 , C 1 , C 2 , C 3 , C 4 The values are 3, 2, 3, 2, 96, 192, 384, and 768 respectively. The obtained fully convolutional 3D gesture estimation network is suitable for application scenarios with high requirements for operation efficiency. The rest of the process is the same as that of Example 1, which is suitable for application scenarios with high requirements for operation accuracy and low requirements for efficiency.
[0097] The above embodiments are only used for detailed explanation to help understand the technical solutions of the present invention. For those skilled in the art, any improvements and substitutions made without departing from the principles of the present invention shall fall within the protection scope of the present invention.
Claims
1. A monocular 3D hand gesture estimation method based on a fully convolutional neural network, characterized in that: The following steps are involved: S1, obtaining a hand image and preprocessing the image; S2. construct a fully convolutional 3D gesture estimation network and train the fully convolutional 3D gesture estimation network; S3, input the preprocessed image into the fully convolutional 3D gesture estimation network to predict the final 2D coordinates of the key points and the relative depth of each key point; S4, post-processing the predicted two-dimensional coordinates of the key points and the relative depth to calculate the three-dimensional coordinates of the hand key points; The post-processing includes: 1) Convert the normalized relative depth of the key points in the output of the fully convolutional 3D gesture estimation network to the absolute depth of the key points. First, substitute the provided hand reference length into the following equation: Among them, l ref represents the reference length of the hand, K is the intrinsic parameter matrix of the imaging device, z r is the absolute depth of the root key point, (u r , v r ) is the two-dimensional coordinate of the root key point predicted by the network, (u m , v m ) is the two-dimensional coordinate of the middle finger palm node predicted by the network, d m is the normalized relative depth value of the middle finger palm node predicted by the network; Solving the above equation can obtain the absolute depth z of the root key point r , the absolute depth of all key points can be restored by combining the normalized relative depth values of each key point predicted by the network: in, represents the normalized relative depth value of the i-th key point predicted by the network, z i is the corresponding absolute depth value; 2) Calculate the 3D coordinates of the key points in the image using the 2D coordinates of the key points in the output of the fully convolutional 3D gesture estimation network and the absolute depth of the key points after transformation; The specific process of calculating the three-dimensional coordinates of the key points is: Among them, (X i , Y i , Z i ) is the three-dimensional coordinate of the i-th key point, (u i , v i ) is the 2D coordinate of the i-th key point output by the fully convolutional 3D gesture estimation network, z i is the absolute depth value of the i-th key point, and K is the intrinsic parameter matrix of the imaging device.
2. According to claim 1, a monocular 3D gesture estimation method based on a fully convolutional neural network is characterized in that: The preprocessing includes image scaling and image filling.
3. The method for monocular 3D gesture estimation based on a fully convolutional neural network according to claim 1, characterized in that: The fully convolutional 3D gesture estimation network includes the following units: An input compression unit, used to extract local features in an input image while reducing the resolution of the input image; Basic convolution unit, used to extract basic features of the image; The downsampling unit is used to spatially downsample the feature map to improve the receptive field of the feature map; The upsampling unit is used to spatially upsample the feature map, and cooperate with the lateral connection output from the downsampling unit to restore the spatial information of the feature map and further enrich the semantic features; The output unit is used to extract information from the final high-resolution feature map and predict the final two-dimensional coordinates of the hand key points and the normalized relative depth.
4. The method for monocular 3D gesture estimation based on a fully convolutional neural network according to claim 3, characterized in that: The basic convolution unit comprises: 1) 7×7 channel-by-channel convolution: channel-by-channel convolution with a large convolution kernel, keeping the number of input feature map channels single; 2) Layer normalization operation: convert the mean and variance of all pixel values of the input feature map into parameters involved in training; 3) 1×1 convolution: used to enlarge or reduce the number of feature map channels by 4 times; 4) Activation function: The GELU function is used to calculate the output activation value of the convolutional layer; 5) Residual connection: Add the unit input and the activated output as the final unit output.
5. The method for monocular 3D gesture estimation based on a fully convolutional neural network according to claim 3, characterized in that: The output unit comprises: 1) 2D coordinate prediction branch: A 3×3 convolution kernel is used to extract features from the feature map to obtain the heat map of the key points of the hand, and the highest response position is obtained through a differentiable maximum index operation, which is the predicted 2D coordinate of the key point; 2) Relative depth prediction branch: Use a 3×3 convolution kernel to extract features from the feature map to obtain the hidden depth map of the hand key points, and obtain the normalized relative depth value of each key point through global spatial average pooling.
6. The method for monocular 3D gesture estimation based on a fully convolutional neural network according to claim 1, characterized in that: The training of a fully convolutional 3D hand pose estimation network consists of the following steps: 21) Image preprocessing: scaling and padding the input images involved in training, and performing data augmentation; 22) Annotation preprocessing: During the training process, the processing of the hand key point annotation mainly includes: modifying the corresponding key point two-dimensional annotation in accordance with the image rotation data enhancement method, and converting the absolute depth annotation into a normalized relative depth annotation. The specific operation is to subtract the depth annotation of all key points from the depth annotation of the root key point, and divide the result by the hand reference length; 23) Network forward reasoning: Input the processed image into the fully convolutional 3D gesture estimation network to obtain the predicted 2D coordinates of the key points and relative depth; 24) Loss calculation: Input the output of network forward reasoning and the preprocessed annotations into the loss function to obtain the loss function value; 25) Gradient back propagation: Calculate the gradient of the loss function value relative to the network parameters of the fully convolutional 3D gesture estimation network, and use the back propagation algorithm to update the network parameters; 26) Iterative training: Repeat the above steps for the input images in batches until the loss function value no longer decreases, and the training of the fully convolutional 3D gesture estimation network is completed.
7. The method for monocular 3D gesture estimation based on a fully convolutional neural network according to claim 6, characterized in that: The fully convolutional 3D gesture estimation network training process uses the Smooth-L1 loss function to calculate the error between the two-dimensional coordinates and relative depth of the key points predicted by the network and the annotations, and calculates the gradient of the error relative to each network parameter, and updates the network parameters through the back-propagation algorithm.
8. The method for monocular 3D gesture estimation based on a fully convolutional neural network according to claim 6, characterized in that: The data enhancement used in image preprocessing during the training of the fully convolutional 3D gesture estimation network includes: 1) Random image rotation: randomly rotate the input image around the center point from -180° to 180°; 2) Random image flipping: randomly flip the input image horizontally and vertically; 3) Random color transformation: The HSV channel of the input image is scaled by random values in the range of 75% to 125%, 50% to 150%, and 50% to 150% respectively; 4) Random noise enhancement: A Gaussian noise with a mean of 0 and a variance of 0 to 0.1 is applied to each position in the input image with a probability of 50%.