Three-dimensional hand posture estimation method and device, medium and product

Abstract

The invention discloses a three-dimensional hand posture estimation method and device, a medium and a product, and relates to the field of hand posture estimation. Determining a 2.5 D attitude key point of the hand RGB image to be estimated by using the 2.5 D attitude key point prediction model; wherein the 2.5 D attitude key point prediction model is obtained by training an initial model by using a training data set; the initial model comprises an S-HGNet module, a multi-scale feature fusion module and a 2.5 D attitude conversion module; the S-HGNet module is an improved hourglass network; according to the improved hourglass network, a Basicblock module in the hourglass network is replaced by a feeling enhancement module. And determining three-dimensional key point coordinates of the to-be-estimated hand RGB image by using an inverse perspective projection method according to the 2.5 D attitude key point of the to-be-estimated hand RGB image. According to the invention, the precision of hand posture estimation is improved.

Description

Technical Field

[0001] The present application relates to the field of hand posture estimation, and in particular to a three-dimensional hand posture estimation method, device, medium and product. Background Art

[0002] Hands are the primary and intuitive tool for humans to interact with the world, the main means of manipulating objects, and play a key role in communication. Accurate 3D hand pose estimation has become a key technology in fields such as augmented reality (AR), virtual reality (VR), gesture recognition, and human-computer interaction. It can help humans communicate with machines in a more natural and diverse way. However, hand pose estimation remains challenging due to problems such as high joint flexibility, local self-similarity, and severe self-occlusion. In recent years, RGB-D (RGB+DepthMap, depth image)-based 3D hand pose estimation algorithms have attracted attention due to their reliable results. However, such algorithms usually require additional RGB-D sensor support, which increases the deployment cost and therefore cannot be widely used. Other algorithms use multiple RGB cameras located at different viewpoints to solve the occlusion problem, but the multi-view acquisition system requires expensive hardware and strict environment configuration and cannot run in unconstrained scenes. Compared with the above algorithms, estimating hand pose directly from a single RGB image is more applicable and has received widespread attention in recent research.

[0003] In recent years, many hand pose estimation algorithms based on RGB images have been proposed. Compared with the human body, the hand always covers a smaller area in the whole RGB image, which makes hand pose estimation more challenging due to the high degree of freedom of hand joints, similar skin color and self-occlusion. In addition, due to the lack of depth information and sensitivity to factors such as illumination, existing hand pose estimation algorithms based on RGB images still need to locate occluded key points with effective accuracy. Zimmermann et al. trained a CNN-based algorithm with a strategy of piecewise independent sub-networks. However, their 3D hand pose estimation only relies on 2D key points, which to some extent affects the hand pose estimation performance due to the lack of 3D annotations of large-scale datasets of real images and the neglect of rich texture features and latent space information of RGB images. Cai et al. proposed an estimation algorithm based on weakly supervised deep learning, which converts the 3D hand pose estimated from RGB images into a depth map by introducing a deep regularizer. Then, the estimated depth map is used to perform weak supervision on the 3D hand pose regression. Although this algorithm can effectively overcome the difficulty of obtaining real 3D annotations, it is heavily dependent on paired RGB and depth images. Iqbal et al. used a 2.5D heat map formula to encode the 2D joint position together with the depth information, which greatly improved the accuracy of hand pose estimation. Spurr et al. used hand structural features such as the length of the human hand bones, the joint rotation angle, and the shape of the palm as constraints, and realized 3D hand pose estimation through weakly supervised training, while also using 2.5D features. The above algorithms usually focus on single isolated hand pose detection and can only estimate the left hand pose or the right hand pose, which lacks versatility in practice. Moon et al. proposed an Internet-based algorithm to predict 3D single-hand or interactive hand poses, using two symmetrical branches to simultaneously estimate 3D interactive hand poses. However, since the local detail information of the finger edges is not considered, the algorithm has low accuracy in the case of occluded gestures. Summary of the invention

[0004] The purpose of this application is to provide a three-dimensional hand posture estimation method, device, medium and product to improve the accuracy of hand posture estimation.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a three-dimensional hand posture estimation method, comprising:

[0007] Get the RGB image of the hand to be estimated;

[0008] The 2.5D posture key point prediction model is used to determine the 2.5D posture key points of the hand RGB image to be estimated; wherein the 2.5D posture key point prediction model is obtained by training the initial model using a training data set; the training data set includes the training hand RGB image and the corresponding 2.5D posture key points; the initial model includes an S-HGNet module, a multi-scale feature fusion module and a 2.5D posture conversion module; the S-HGNet module is connected to the multi-scale feature fusion module and the 2.5D posture conversion module respectively; the multi-scale feature fusion module is also connected to the 2.5D posture conversion module; the S-HGNet module is an improved hourglass network; the improved hourglass network is to replace the Basicblock module in the hourglass network with a perception enhancement module; the perception enhancement module includes a first 3×3 convolutional layer, a first BN layer, an activation function layer, a second 3×3 convolution layer, second BN layer, dilated convolution layer, third BN layer and CBAM attention mechanism module, the first 3×3 convolution layer, the first BN layer and the activation function layer are connected in sequence, the activation function layer is also connected to the second 3×3 convolution layer and the dilated convolution layer respectively, the second 3×3 convolution layer is connected to the second BN layer, the dilated convolution layer is connected to the third BN layer, and the second BN layer and the third BN layer are both connected to the CBAM attention mechanism module; the S-HGNet module is used to extract feature maps of different scales from the hand RGB image; the multi-scale feature fusion module is used to determine the two-dimensional joint point heat map of the hand based on the feature maps of different scales; the 2.5D posture conversion module is used to determine the root relative depth, the depth of the left hand root relative to the right hand root, the existence of left and right hands and 2.5D posture key points based on the feature map and the two-dimensional joint point heat map of the hand;

[0009] According to the 2.5D posture key points of the hand RGB image to be estimated, the three-dimensional key point coordinates of the hand RGB image to be estimated are determined by using an inverse perspective projection method.

[0010] Optionally, the multi-scale feature fusion module includes an upsampling layer, a downsampling layer, a global attention mechanism module and a Soft-Attention mechanism module.

[0011] Optionally, the 2.5D posture conversion module includes a first convolutional layer, a second convolutional layer, a first multi-layer perceptron network and a second multi-layer perceptron network.

[0012] Optionally, based on the feature map and the two-dimensional hand joint point heat map, determining the root relative depth, the depth of the left hand root relative to the right hand root, and the existence of the left and right hands specifically includes:

[0013] Determining a potential depth map based on the feature map using the first convolutional layer;

[0014] Determine a depth map according to the potential depth map and the two-dimensional hand joint point heat map;

[0015] Determining the root relative depth according to the depth map;

[0016] Determine a potential vector according to the feature map using the second convolutional layer;

[0017] Determine, based on the latent vector, the depth of the left-hand root relative to the right-hand root using the first multi-layer perceptron network;

[0018] The presence of the left and right hands is determined based on the latent vector using the second multi-layer perceptron network.

[0019] Optionally, according to the 2.5D posture key points of the hand RGB image to be estimated, using an inverse perspective projection method, determining the coordinates of the three-dimensional key points of the hand RGB image to be estimated specifically includes:

[0020] Using the formula Determine the three-dimensional key point coordinates of the hand RGB image to be estimated; where, is the coordinate of the left-hand 3D key point; is the coordinate of the right-hand 3D key point; ∏ is the camera back-projection operation; T -1 is the inverse affine transformation; It is the 2.5D posture key point of the left hand; is the 2.5D posture key point of the right hand; Z L is the absolute depth of the left hand root; Z R It is the absolute depth of the right hand root.

[0021] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any one of the above-described three-dimensional hand posture estimation methods.

[0022] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any one of the above-described three-dimensional hand posture estimation methods.

[0023] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the three-dimensional hand posture estimation methods described above.

[0024] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0025] The present application provides a three-dimensional hand posture estimation method, device, medium and product, which obtains an RGB image of a hand to be estimated; uses a 2.5D posture key point prediction model to determine the 2.5D posture key points of the RGB image of the hand to be estimated; wherein the 2.5D posture key point prediction model is obtained by training an initial model using a training data set; the initial model includes an S-HGNet module, a multi-scale feature fusion module and a 2.5D posture conversion module; the S-HGNet module is respectively connected to the multi-scale feature fusion module and the 2.5D posture conversion module; the multi-scale feature fusion module is also connected to the 2.5D posture conversion module; the S-HGNet module et module is an improved hourglass network; the improved hourglass network replaces the Basicblock module in the hourglass network with a perception enhancement module; the perception enhancement module includes the first 3×3 convolution layer, the first BN layer, the activation function layer, the second 3×3 convolution layer, the second BN layer, the dilated convolution layer, the third BN layer and the CBAM attention mechanism module, the multi-scale feature fusion module is used to determine the two-dimensional joint point heat map of the hand based on feature maps of different scales; the 2.5D posture conversion module is used to determine the root relative depth, the depth of the left hand root relative to the right hand root, the existence of the left and right hands and the 2.5D posture key points based on the feature map and the two-dimensional joint point heat map of the hand. According to the 2.5D posture key points of the hand RGB image to be estimated, the three-dimensional key point coordinates of the hand RGB image to be estimated are determined by using the inverse perspective projection method. This application improves the accuracy of hand posture estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A schematic diagram of a flow chart of a three-dimensional hand posture estimation method provided in one embodiment of the present application;

[0028] Figure 2 This is the overall framework diagram of the 2.5D posture key point prediction model;

[0029] Figure 3 This is a schematic diagram of the Basicblock module structure;

[0030] Figure 4 It is a schematic diagram of the SEM module structure;

[0031] Figure 5 Schematic diagram of the global attention mechanism module structure;

[0032] Figure 6 Schematic diagram of the Soft-Attention mechanism module structure;

[0033] Figure 7 Schematic diagram of experimental results;

[0034] Figure 8 Schematic diagram of the two-dimensional PCK curve on the InterHand2.6M dataset;

[0035] Fig. 9 Schematic diagram of the three-dimensional PCK curve on the RHD dataset;

[0036] Fig.10 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0038] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0039] In an exemplary embodiment, Figure 1 As shown, a three-dimensional hand posture estimation method is provided, comprising the following steps:

[0040] S1: Obtain the RGB image of the hand to be estimated.

[0041] S2: Determine the 2.5D posture key points of the hand RGB image to be estimated by using a 2.5D posture key point prediction model; wherein the 2.5D posture key point prediction model is obtained by training the initial model using a training data set; the training data set includes the training hand RGB image and the corresponding 2.5D posture key points; the initial model includes an S-HGNet module, a multi-scale feature fusion module and a 2.5D posture conversion module; the S-HGNet module is connected to the multi-scale feature fusion module and the 2.5D posture conversion module respectively; the multi-scale feature fusion module is also connected to the 2.5D posture conversion module; the S-HGNet module is an improved hourglass network; the improved hourglass network replaces the Basicblock module in the hourglass network with a perception enhancement module; the perception enhancement module includes a first 3×3 convolutional layer, a first BN layer, an activation function layer, The second 3×3 convolution layer, the second BN layer, the dilated convolution layer, the third BN layer and the CBAM attention mechanism module, the first 3×3 convolution layer, the first BN layer and the activation function layer are connected in sequence, the activation function layer is also connected to the second 3×3 convolution layer and the dilated convolution layer respectively, the second 3×3 convolution layer is connected to the second BN layer, the dilated convolution layer is connected to the third BN layer, and the second BN layer and the third BN layer are both connected to the CBAM attention mechanism module; the S-HGNet module is used to extract feature maps of different scales from the hand RGB image; the multi-scale feature fusion module is used to determine the two-dimensional joint point heat map of the hand based on the feature maps of different scales; the 2.5D posture conversion module is used to determine the root relative depth, the depth of the left hand root relative to the right hand root, the existence of left and right hands and 2.5D posture key points based on the feature map and the two-dimensional joint point heat map of the hand.

[0042] As an optional implementation, the multi-scale feature fusion module includes an upsampling layer, a downsampling layer, a global attention mechanism module and a Soft-Attention mechanism module.

[0043] As an optional implementation, the 2.5D posture conversion module includes a first convolutional layer, a second convolutional layer, a first multi-layer perceptron network and a second multi-layer perceptron network.

[0044] As an optional implementation, based on the feature map and the two-dimensional hand joint point heat map, determining the root relative depth, the depth of the left hand root relative to the right hand root, and the existence of the left and right hands specifically includes:

[0045] A potential depth map is determined based on the feature map using the first convolutional layer.

[0046] A depth map is determined according to the potential depth map and the two-dimensional hand joint heat map.

[0047] The root relative depth is determined according to the depth map.

[0048] According to the feature map, a potential vector is determined using the second convolutional layer.

[0049] The first multi-layer perceptron network is used to determine the depth of the left-hand root relative to the right-hand root based on the latent vector.

[0050] The presence of the left and right hands is determined based on the latent vector using the second multi-layer perceptron network.

[0051] S3: According to the 2.5D posture key points of the hand RGB image to be estimated, the three-dimensional key point coordinates of the hand RGB image to be estimated are determined by using an inverse perspective projection method.

[0052] As an optional implementation manner, according to the 2.5D posture key points of the hand RGB image to be estimated, using the inverse perspective projection method, determining the coordinates of the three-dimensional key points of the hand RGB image to be estimated specifically includes:

[0053] Using the formula Determine the three-dimensional key point coordinates of the hand RGB image to be estimated; where, is the coordinate of the left-hand 3D key point; is the coordinate of the right-hand 3D key point; ∏ is the camera back-projection operation; T -1 is the inverse affine transformation; It is the 2.5D posture key point of the left hand; is the 2.5D posture key point of the right hand; Z L is the absolute depth of the left hand root; Z R It is the absolute depth of the right hand root.

[0054] In practical applications, with the rapid development of deep learning and computer vision technology, multi-scale feature fusion is widely used in the field of human body and hand gesture estimation, because different parts of the human body and hand require feature maps of different scales to represent the key point positions. Therefore, the multi-scale network structure considers the information exchange and feature extraction between feature maps of different scales at the beginning of design, and then fuses the features of different scales to obtain the final output. For example, the Feature Pyramid Networks (FPN) proposed by Lin et al. can fuse features of different scales together through downsampling and upsampling operations, so that the network can simultaneously obtain rich semantic information and detail information, while maintaining a balance between expression ability, speed and resource consumption. Sun et al. proposed a high-resolution network (HRNet), which adopts a branch-merge structure to simultaneously capture features at different levels in the image. Through this design, HRNet can efficiently fuse multi-scale features and always maintain the high-resolution feature representation of the image throughout the entire processing process. This feature enables HRNet to generate effective high-resolution output, thereby more effectively using multi-scale information to predict key points. The Hourglass Network (HGNet) constructed by Newell et al., also known as the hourglass network, was originally used for human key point detection. It is a highly respected architecture for estimating human posture and has since been widely used in other fields, mainly because of its outstanding ability to balance the requirements of large receptive field and high resolution. The algorithm adopts a top-down processing strategy, first performing residual block processing on the input image to reduce its resolution, thereby generating a low-resolution feature map. Then, after reaching the lowest resolution, the network begins to gradually upsample the feature map and fuse features across scales to generate a heat map that accurately predicts the location of key points. HandyPose proposed by Divyansh et al. uses a multi-level waterfall attribute space pooling module for multi-scale representation, and maintains a larger field of view by connecting multiple levels of features of networks at different levels in series, achieving high-precision estimation of hand posture. Therefore, this application draws some inspiration from the above algorithm, allowing the network to pay more attention to deep feature learning and use deep features to assist shallow features in key point mining.

[0055] To this end, this application proposes a new RGB image 3D hand posture estimation method based on attention mechanism and multi-scale feature fusion. The main innovations of this method are:

[0056] 1) The Sensory Enhancement Module (SEM) is proposed to replace the Basic block in the Hourglass Network architecture, and the Hourglass Network is improved to the S-HGNet module. SEM combines the dilated residual block and the CBAM attention mechanism to maintain spatial perception while expanding the network receptive field, and can effectively extract key information at each stage.

[0057] 2) A multi-scale feature fusion module (MSFF) is proposed. This module introduces spatial attention mechanism and soft-attention mechanism. By combining higher semantic information and more global spatial scale information, it can accurately obtain the two-dimensional key points of the hand in RGB images.

[0058] 3) To address the problem of spatial loss caused by direct regression calculation of three-dimensional posture information by two-dimensional key point coordinates, this application introduces a 2.5D posture conversion module, which simultaneously estimates the left and right hands, the relative root node depths of the left and right hands, and the 2.5D left and right hand postures. Finally, the 2.5D posture is used to obtain the three-dimensional coordinates, which significantly improves the extraction of three-dimensional posture information of the key points of the hands.

[0059] The goal of 3D hand pose estimation is to estimate the 3D position of each key point of the hand based on a single RGB image. There is no unified standard for the number of key points of the hand. Different data sets may use different numbers of key points. Currently, the mainstream algorithms for hand key point annotation mainly include 14 key points, 16 key points, and 21 key points. In order to represent hand pose information, this application decided to use the 21 key point algorithm proposed by OpenPose.

[0060] The overall network framework of this application is as follows Figure 1 As shown, it includes the S-HGNet module, the multi-scale feature fusion module MSFF and the 2.5D posture conversion module.

[0061] Most deep learning networks rely on residual networks and use basic blocks. In order to accurately estimate occluded, distorted and overlapped keypoints, the encoding process usually adopts repeated downsampling operations to obtain a large receptive field. However, it reduces the spatial resolution, which leads to the loss of key local details required for accurate estimation. Therefore, the trade-off between global features and detail features should be considered. Inspired by the dilated convolution and CBAM modules, a new receptive enhancement module (SEM) is proposed, which can not only maintain the high-resolution spatial structure of the hand but also obtain a large receptive field.

[0062] In the HGNet architecture, Figure 3As shown in , the Basicblock structure contains two consecutive 3×3 convolutional layers, which are connected by an identity mapping that acts as a shortcut connection. This application replaces the Basicblock module with the SEM module. The SEM module structure is as follows Figure 4 As shown in Figure 2. In addition to the 3×3 convolutional layer of the basic block, the improved SEM block also includes an additional parallel dilated convolutional layer. The output is then obtained from the CBAM block. Both the input and output are feature maps. The dilated convolution is mathematically defined as:

[0063]

[0064] Among them, f out Represents the value of the output feature map at the spatial position [i, j], which is used for further calculation and processing of the feature map by subsequent algorithms. inp is the input feature map. [p,q] represents the position index of the expanded convolution kernel, where w represents the kernel size and b represents the expansion rate. In the algorithm proposed in this application, the SEM module is used to construct the backbone network algorithm to improve the ability to extract hand features.

[0065] In traditional algorithms, high-resolution feature maps contain more details and can accurately locate key points, but due to the small receptive field, distorted or overlapping key points may not be recognized. Low-resolution representations contain more contextual semantic information and can handle difficult scenes (for example: occluded key points). However, the level of spatial detail is low. The reduction in spatial details reduces the ability of the algorithm to accurately locate key points. How to effectively aggregate multi-level features remains a challenging problem for pose estimation. In general, element summation and channel cascading are the most commonly used algorithms for fusing multi-level features. Aggregating multiple levels of features by element summation is averaged, but this is not a learnable process. Attention mechanisms are widely used to address this limitation and enhance spatial details. SPCNet proposes a selective information module, which accumulates multiple features and allows adaptive fusion of information. Therefore, on this basis, this application uses the spatial attention mechanism and the Soft-Attention attention mechanism to fuse multi-scale features to obtain more accurate two-dimensional key points of the hand. Specifically, Figure 2 As shown in the figure, first, the multi-scale features (i.e., 8, 16, 32, and 64 pixels) from the improved HGNet are downsampled or upsampled to the same resolution multiple times and then fused. The global attention mechanism enables the features to adapt to the feature map, where the fusion is performed based on the global spatial weights and subsequent outputs. Subsequently, the high-resolution and low-resolution features are fused through the Soft-Attention mechanism for three-level cross-scale features and produce classification outputs for each key point.

[0066] The global attention module is as follows Figure 5As shown, a 1×1 convolutional layer is used to combine multiple fusion feature channels (denoted as F 1 ~F 4 , with 64, 128, 256, and 512 channels, respectively) are adjusted to 256 channels. Then batch normalization and ReLU are performed in sequence. Subsequently, the multi-scale features are upsampled to a size of 64×64, and the obtained feature is represented as G 1 ~G 4 To achieve this, a 1×1 convolution filter is used to project the feature maps of multiple levels into four single-channel features, which are then concatenated as U = U 1 ~U 4 , corresponding to G 1 ~G 4 . Then the softmax function is applied to U along the channel dimension to generate the pixel-level adaptive weight W corresponding to the multi-level features, as follows:

[0067]

[0068] in, is the index of the reference single channel feature map element. The corresponding trainable weight parameters are expressed as Next, the global learning weight W n and G n Multiply them together and use a 1×1 convolutional layer to resize the resulting product to match F n This process enables the final fused features to collect multi-level information from multiple scales and enhance the algorithm's global information perception ability.

[0069] In the feature fusion stage, the Soft-Attention mechanism is combined, such as Figure 2 As shown. Soft-Attention is a differentiable attention mechanism. The algorithm trained with Soft-Attention can implicitly learn to suppress irrelevant areas in the input image while highlighting salient features that are useful for specific tasks. Soft-Attention has the characteristics of plug-and-play and is widely used in network improvements. This application uses Soft-Attention to merge high-resolution features with second-high-resolution features. Through this strategy, information of adjacent scales is fused so that the output information can represent features of more scales. The specific structure of Soft-Attention is shown in Figure 6 shown.

[0070] To estimate the final 3D hand pose of both hands, learn the function F:H→P 2.5D , and maps the obtained 2D key point feature map to 2.5D posture. 2.5D posture is represented by P 2.5DIndicated by each 2.5D joint (x i ,y i ,z i ), where (x i ,y i ) is a three-dimensional joint (X i ,Y i ,Z i ) is a two-dimensional projection of i =Z i -Z root(i) , the symbol root(i) represents the hand root of joint i, which indicates the relative depth of the hand. During the reasoning process, the depth estimate Z i In (x i ,y i ) to restore the 3D pose. The algorithm of this application estimates the dominant hand (h L ,h R ), h L Indicates the presence of the left hand, h R Indicates the presence of the right hand, 2.5D left and right hand posture P 2.5D , and the relative depth z of the right hand relative to the left hand root R→L , where L and R represent the left and right hands. Since this algorithm aims to predict 2.5D joints (x i ,y i ,z i ), and the conversion from 2.5D to 3D involves the process of inverse perspective projection, so it must be done through Z i =z i +Z root(i) Estimate the depth Z of joint i i , to complete this conversion. When both hands are present, the relative depth of the right hand relative to the left hand root is z R→L Used to obtain left hand root depth.

[0071] 2.5D hand pose estimation specifically includes: first inputting the RGB image into the improved HGNet network, outputting the feature map through the improved HGNet network, and then using the obtained feature map to predict the potential depth map of the root node relative depth using convolution. In order to concentrate the depth value on the joint position, the 2D key point map is element-wise multiplied with the potential depth map to obtain the depth map. Finally, the values ​​on each slice of the depth map are added to obtain the root relative depth z of joint i i .

[0072] At the same time, the output feature map detects the presence of two hands and measures the depth of the left hand root relative to the right hand root. The feature map is repeatedly convolved and downsampled into a latent vector x, which is used to estimate the presence of two hands and measure the depth of the left root relative to the right root through two independent multi-layer perceptron (MLP) networks.

[0073] Finally, the 2.5D posture key points are mapped to three-dimensional coordinates through the following two formulas.

[0074]

[0075] Among them, ∏ and T -1 It is the camera back-projection operation and the inverse affine transformation (i.e. cropping and resizing). Projection requires the left and right root Z L and Z R The absolute depth (written as a vector), Z L and Z R The calculation formula is as follows:

[0076]

[0077] Z R =[0,0,z R ] T .

[0078] Among them, Z L and Z R is the absolute depth of the left and right hand roots. Estimating the absolute depth of the left and right hand roots can avoid the problem of spatial loss caused by directly regressing the 2D keypoint coordinates to calculate the 3D posture information.

[0079] The loss function used for the training algorithm in this application is:

[0080] L=L 2D +L 2.5D +L z .

[0081] L 2D It represents the difference between the extracted 2D heat map features and the 2D posture estimation heat map. The calculation formula is as follows:

[0082]

[0083] Among them, H i and They represent the predicted 2D heat map and the real 2D heat map of the i-th key point respectively. Finally, the Frobenius norm is used to sum the matrix differences to obtain the loss value of the 2D posture.

[0084] L 2.5D It represents the 2.5D hand pose estimation loss, which is calculated as follows:

[0085]

[0086] in, represents a true three-dimensional Gaussian heat map, Represents the predicted 3D Gaussian heatmap. If the input image does not contain one of the right or left hands, the corresponding loss is set to zero. The true 3D Gaussian heatmap is obtained by calculating the Gaussian blob.

[0087] L z It represents the depth loss of the right hand relative to the left hand, and the calculation formula is as follows:

[0088] L z =|z R→L -z R→L* |.

[0089] Among them, z R→L* Indicates the true value of the depth of the right hand root relative to the left hand root, z R→L represents the predicted depth of the right hand root relative to the left hand root. When the input image contains only one hand, the loss becomes zero.

[0090] This application is based on two datasets, InterHand2.6M and RHD, for experiments. The InterHand2.6M dataset is used for network pre-training, and the RHD dataset is used for fine-tuning and testing. All experimental data are labeled with the three-dimensional coordinates of 21 joint points.

[0091] InterHand2.6M dataset: This dataset contains a large number of real RGB images, consisting of hand gestures of single hands and interactive hand sequences. The dataset includes 1,361,062 training images and 849,160 test images. It contains semi-automatically annotated 3D coordinates of 42 hand joints, mainly used for network pre-training.

[0092] RHD dataset (Rendered Handpose Dataset): This dataset is a synthetic dataset containing two independent hand data. The dataset includes 43,986 hand images, including 41,258 training images and 2,728 validation images. Since the background of this dataset is an outdoor scene, this application uses this dataset to approximate the generalization ability of the algorithm under field conditions.

[0093] For the InterHand2.6M dataset, this application uses AP hThe gesture prediction results are evaluated using three evaluation indicators: mean relative-root position error (MRRPE), mean per joint position error (MPJPE). For the RHD dataset, this application uses EPE (end point error) and AUC (area under the PCK curve) for evaluation, with a threshold range of 20mm to 50mm, where PCK refers to the percentage of correct three-dimensional joints.

[0094] All experiments were performed on a machine equipped with an NVIDIA RTX2080Ti graphics card, and the neural network was implemented using the PyTorch framework. The input hand images were directly cropped and resized to 256×256 resolution. The batch_size was set to 32, and the network algorithm was trained using the Adam optimizer. The InterHand2.6M and RHD datasets were trained for a total of 42 epochs.

[0095] The experimental results are as follows Figure 7 As shown in Figure 1, the hand postures with finger self-occlusion and hand-to-hand mutual occlusion are shown. Compared with the InterNet algorithm of Moon et al., the algorithm proposed in this application performs better in solving such occlusion problems. Specifically, some problems can be observed in the estimation results of the InterNet algorithm: For example, the left hand posture prediction is inaccurate ( Figure 7 The second row and the first column), some finger joints of the right hand are wrongly predicted on the left hand; the predicted positions of the left hand joints are chaotic, and some finger joints are predicted on the right hand ( Figure 7 The predicted positions of the right hand joints are also confusing ( Figure 7 The second row and third column); and the overall predicted position of the hand joints is confusing ( Figure 7 After adopting the algorithm of this application, these problems have been significantly improved: the left hand gesture estimation is more accurate ( Figure 7 The positions of different finger joints are more clearly distinguished; the two hands can be better distinguished, and the prediction of joint positions is more accurate ( Figure 7 These results fully demonstrate that the algorithm of this application has a better estimation effect when dealing with the occlusion problem in hand posture estimation.

[0096] Table 1 lists the evaluation indicators obtained by comparing with PoseNet, InterNet, DIGIT, MS-FF, and KeypointTransformer algorithms on the InterHand2.6M dataset, where the bold values ​​are the optimal values. It can be observed that compared with PoseNet, InterNet, DIGIT, MS-FF, and KeypointTransformer algorithms, the algorithm of this application outperforms other algorithms in all scenarios. Specifically, compared with PoseNet, InterNet, DIGIT, MS-FF, and KeypointTransformer algorithms, among the evaluation indicators obtained by the algorithm of this application, the mean error of the root node (MRRPE), the mean error of the joint points on one hand (MPJPE (single hand)), and the mean error of the joint points on the interactive hand (MPJPE (two hands)) all obtained the lowest error values, which were 29.57mm, 9.96mm, and 12.32mm, respectively. Compared with the InterNet algorithm, the various indicators of the algorithm of this application are reduced by about 9.2%, 18.2%, and 23.1%. The initial parameter values ​​of the algorithm are randomly generated through normal distribution. After each iteration of optimization, the parameters will be adjusted, resulting in slight differences in the final APh values. h The value has not dropped significantly, indicating that the algorithm performance is relatively stable.

[0097] Table 1 Comparison results of evaluation indicators of different algorithms

[0098]

[0099] On the other hand, benefiting from the proposed multi-scale feature fusion estimation algorithm, the algorithm of this application can obtain more accurate two-dimensional key point results. Figure 8 Comparison results of 2D keypoint prediction between InterNet and the proposed algorithm are shown. Figure 8 (a) and Figure 8 (b) shows the results of single-hand images and interactive hand images on the InterHand2.6M dataset test set. It can be seen that the algorithm proposed in this application not only obtains more accurate 2D single-hand posture prediction, but also greatly improves the prediction of 2D interactive hand posture.

[0100] To prove the universality of the proposed algorithm, this application also compares the performance of the algorithm of this application with PoseNet, Weakly-supervised 3D Hand Pose Regression Network, BMC, dVAE, and DetNet algorithms on the RHD dataset. Fig. 9The changes in the three-dimensional PCK curves of the algorithm in this application are shown with the thresholds of PoseNet, Weakly-supervised 3D Hand PoseRegressionNetwork (WS-HPRNet), BMC, dVAE, and DetNet algorithms in the range of 20-50mm. The final results prove that the algorithm in this application is superior to other algorithms.

[0101] Table 2 further shows the comparison results of the average joint error (EPE) of different algorithms on the RHD dataset. Among them, GTS (ground truth scale) and GTH (groundtruth handness) respectively indicate that the real hand category and size need to be used during testing. Since the algorithm of this application does not use scale parameters, it is more difficult for the algorithm of this application to predict the depth of three-dimensional joints compared with PoseNet, BMC, dVAE, and Knowledge as Priors algorithms. However, the algorithm of this application is more suitable for practical applications because it does not require additional scale parameters of the dataset. As can be seen from Table 2, the algorithm of this application can obtain lower error values ​​even without using additional scale parameters during testing.

[0102] Table 2 Comparison of RHD dataset and existing algorithms

[0103] algorithm GTS GTH EPE / mm PoseNet, yes yes 30.42 dVAE yes yes 19.95 Knowledge as Priors yes yes 20.74 BMC yes yes 19.73 BMC no no 19.73 InterNet no no 20.89 This application algorithm no no 18.21

[0104] This application conducts ablation experiments on the proposed different modules to further demonstrate the effectiveness of each module in the algorithm of this application in the hand posture estimation task. For fair comparison, all experiments are trained and tested on the InterHand2.6M dataset. In the 3D hand posture estimation network based on attention mechanism and multi-scale feature fusion, the SEM module is removed to extract feature images using the basic block HGNet, the MSFF module is removed to predict 2D key points using the improved HGNet, and the 2.5D posture estimation module is removed to directly regress 3D key points. The results are shown in Table 3.

[0105] Table 3 Ablation experiment data analysis table

[0106]

[0107] It can be seen that after removing the DDRB module, the performance evaluation indicators of the algorithm of this application decreased by 0.63mm, 1.44mm and 1.96mm respectively, proving that the module expands the receptive field, enhances the sensitivity to spatial information, and can effectively extract the key information of each stage. After removing the multi-scale feature fusion module, the performance evaluation indicators of the algorithm of this application decreased by 3.2mm, 4.7mm and 4.92mm respectively, which is the largest difference compared with the experimental results of the overall algorithm. This shows that the fusion of spatial detail information and global information of different feature maps is crucial for estimating hand joints. Especially in terms of gesture regression, the multi-scale feature fusion module can significantly improve the accuracy. After removing the 2.5D posture estimation module, the performance evaluation indicators of the algorithm of this application decreased by 2.36mm, 3.52mm and 2.94mm respectively, which shows that the 2.5D posture estimation module solves the problem of spatial loss caused by direct regression calculation of three-dimensional posture information by two-dimensional key point coordinates, and therefore plays a relatively important role. The experimental results show that the three-dimensional hand posture estimation network based on the attention mechanism and multi-scale feature fusion achieves the best overall performance under the synergy of the three modules. This fully verifies the effectiveness of each module and provides a more accurate and stable solution for hand posture estimation.

[0108] This application proposes a 3D hand posture estimation network based on attention mechanism and multi-scale feature fusion for a single RGB image, which effectively improves the accuracy of 3D hand posture estimation in the presence of occlusion, scale and depth ambiguity. Among them, the S-HGNet module adopts a new dilated residual block to enhance the receptive field, while retaining the spatial information perception ability, and effectively extracts the key information of each stage. Then, the global attention mechanism and the Soft-Attention attention mechanism are used to fuse multi-scale features to ensure the trade-off between global features and detail features. Through two attention strategies, the algorithm can obtain accurate two-dimensional key points. At the same time, the 2.5D posture conversion module estimates the left and right hands, the relative root node depth of the left and right hands, and the 2.5D left and right hand postures, and finally regresses to obtain the three-dimensional posture, thereby effectively avoiding the problem of space loss when directly regressing the three-dimensional posture information from the two-dimensional key point coordinates. Compared with the existing algorithms, the algorithm of this application has good generalization performance, and the experiments on the InterHand2.6M and RHD datasets have proved the effectiveness and superiority of the algorithm of this application, which can well estimate the postures of single hands and interactive hands. Although the current algorithm performs well in hand pose estimation in single-hand and interactive hand scenarios, it still has certain limitations in hand-object interaction scenarios. Therefore, the next work will focus on solving the complex hand pose estimation problem in hand-object interaction scenarios to further improve the adaptability and accuracy of the algorithm.

[0109] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store hand RGN image data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a three-dimensional hand posture estimation method is implemented.

[0110] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above-mentioned three-dimensional hand posture estimation method when executing the computer program.

[0111] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the above-mentioned three-dimensional hand posture estimation method when executed by a processor.

[0112] In an exemplary embodiment, a computer program product is provided, including a computer program, which implements the above-mentioned three-dimensional hand posture estimation method when executed by a processor.

[0113] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0114] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0115] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0116] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0117] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A three-dimensional hand posture estimation method, characterized in that: include: Get the RGB image of the hand to be estimated; The 2.5D posture key point prediction model is used to determine the 2.5D posture key points of the hand RGB image to be estimated; wherein the 2.5D posture key point prediction model is obtained by training the initial model using a training data set; the training data set includes the training hand RGB image and the corresponding 2.5D posture key points; the initial model includes an S-HGNet module, a multi-scale feature fusion module and a 2.5D posture conversion module; the S-HGNet module is connected to the multi-scale feature fusion module and the 2.5D posture conversion module respectively; the multi-scale feature fusion module is also connected to the 2.5D posture conversion module; the S-HGNet module is an improved hourglass network; the improved hourglass network is to replace the Basicblock module in the hourglass network with a perception enhancement module; the perception enhancement module includes a first 3×3 convolutional layer, a first BN layer, an activation function layer, a second 3×3 convolution layer, second BN layer, dilated convolution layer, third BN layer and CBAM attention mechanism module, the first 3×3 convolution layer, the first BN layer and the activation function layer are connected in sequence, the activation function layer is also connected to the second 3×3 convolution layer and the dilated convolution layer respectively, the second 3×3 convolution layer is connected to the second BN layer, the dilated convolution layer is connected to the third BN layer, and the second BN layer and the third BN layer are both connected to the CBAM attention mechanism module; the S-HGNet module is used to extract feature maps of different scales from the hand RGB image; the multi-scale feature fusion module is used to determine the two-dimensional joint point heat map of the hand based on the feature maps of different scales; the 2.5D posture conversion module is used to determine the root relative depth, the depth of the left hand root relative to the right hand root, the existence of left and right hands and 2.5D posture key points based on the feature map and the two-dimensional joint point heat map of the hand; According to the 2.5D posture key points of the hand RGB image to be estimated, the three-dimensional key point coordinates of the hand RGB image to be estimated are determined by using an inverse perspective projection method.

2. The three-dimensional hand posture estimation method according to claim 1, characterized in that: The multi-scale feature fusion module includes an upsampling layer, a downsampling layer, a global attention mechanism module and a Soft-Attention mechanism module.

3. The three-dimensional hand posture estimation method according to claim 1, characterized in that: The 2.5D posture conversion module includes a first convolutional layer, a second convolutional layer, a first multi-layer perceptron network and a second multi-layer perceptron network.

4. The three-dimensional hand posture estimation method according to claim 3, characterized in that: Based on the feature map and the two-dimensional hand joint point heat map, determining the root relative depth, the depth of the left hand root relative to the right hand root, and the existence of the left and right hands, specifically includes: Determine a potential depth map based on the feature map using the first convolutional layer; Determine a depth map according to the potential depth map and the two-dimensional hand joint point heat map; Determining the root relative depth according to the depth map; Determine a potential vector according to the feature map using the second convolutional layer; Determine, based on the latent vector, the depth of the left-hand root relative to the right-hand root using the first multi-layer perceptron network; The presence of the left and right hands is determined based on the latent vector using the second multi-layer perceptron network.

5. The three-dimensional hand posture estimation method according to claim 1, characterized in that: According to the 2.5D posture key points of the hand RGB image to be estimated, the coordinates of the three-dimensional key points of the hand RGB image to be estimated are determined by using an inverse perspective projection method, specifically including: Using the formula Determine the three-dimensional key point coordinates of the hand RGB image to be estimated; where, are the coordinates of the left-hand 3D key points; is the coordinate of the right-hand 3D key point; ∏ is the camera back-projection operation; T -1 is the inverse affine transformation; It is the 2.5D posture key point of the left hand; is the 2.5D posture key point of the right hand; Z L is the absolute depth of the left hand root; Z R It is the absolute depth of the right hand root.

6. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the three-dimensional hand posture estimation method described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the three-dimensional hand posture estimation method described in any one of claims 1 to 5 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the three-dimensional hand posture estimation method described in any one of claims 1 to 5 is implemented.

Images