A 3D gesture estimation method and system based on attention mechanism and graph convolution

Abstract

The application belongs to the technical field of computer vision, and particularly relates to a 3D gesture estimation method and system based on an attention mechanism and a graph convolution, comprising: performing feature extraction on a hand RGB image by using a 2D feature extraction module to obtain a 2D heat map and a contour map; performing feature enhancement on the 2D heat map, inputting the 2D heat map after feature enhancement and the contour map into a feature mutual injection module, injecting 2D joint position information into the contour map to obtain a first feature map, and simultaneously injecting global information related to a predicted joint coordinate in the contour map into features of the 2D heat map to obtain a second feature map; after the first and second feature maps are spliced, the spliced first and second feature maps are input into a 3D gesture prediction network after a skip link of the 2D feature extraction module, and 3D hand joint position is predicted. The application uses an attention-based module to complete feature mutual injection between different modalities, so that hand joint position information and global information are fully utilized, and 3D gesture estimation precision is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, and particularly relates to a 3D gesture estimation method and system based on an attention mechanism and a graph convolution. BACKGROUND

[0002] The statements in this section merely provide background technology related to the present application and do not necessarily constitute the prior art.

[0003] The hand is an important organ and communication tool of human beings, participating in various actions of the human body, including various subtle actions. Therefore, estimating 3D gestures from monocular RGB images has wide applications in various scientific fields, especially virtual and augmented reality, human-computer interaction, action recognition and robot simulation. In recent years, researchers have tried to use various methods to estimate 3D gestures more accurately and quickly. With the revolutionary progress of deep learning in machine learning and computer vision, many methods use learning methods to predict 3D gestures. Generally, these methods use convolutional neural networks to implicitly encode and decode joint information to complete gesture prediction.

[0004] Although convolutional neural networks perform well in estimating 3D gestures, they only focus on specific features, such as joint features, while ignoring other features. Therefore, some researchers have begun to explore the application of Transformers in this field, including Epipolar Transformer, METRO, Keypoint Transformer and other models. These models use attention mechanisms to establish connections between joint features or use encoder-decoder architectures to accurately estimate 3D gestures. Some models also incorporate graph convolutional neural networks to enhance the feature representation of joints or predict joint coordinates. However, due to the inherent self-occlusion, motion, depth ambiguity and other problems of monocular RGB images, the current prediction performance still needs to be improved. SUMMARY

[0005] In order to solve the problems of the prior art, the present application provides a 3D gesture estimation method and system based on an attention mechanism and a graph convolution, which uses a graph convolutional neural network-based module to strengthen the connections between hand joints through limited propagation, especially emphasizing the connections between neighboring joints. At the same time, an attention-based module is used to complete the mutual injection of features between different modalities, so as to fully utilize the hand joint position information and global information and improve the accuracy of 3D gesture estimation.

[0006] In order to achieve the above-mentioned purpose, one or more embodiments of the present application provide the following technical solutions:

[0007] The first aspect of the present application provides a 3D gesture estimation method based on an attention mechanism and a graph convolution, comprising:

[0008] extracting features of the obtained hand RGB image by using the 2D feature extraction module to obtain a 2D heat map and a contour of the hand RGB image;

[0009] inputting the 2D heat map into a graph convolutional neural network to strengthen the connection between adjacent joints, and obtaining a 2D hand joint heat map with strengthened features;

[0010] inputting the 2D heat map with strengthened features and the contour map into a feature mutual injection module, injecting 2D joint position information into the contour map through a feature injection operation to obtain a first feature map, and injecting global information related to predicted joint coordinates in the contour map into the features of the 2D heat map to obtain a second feature map; and splicing the first feature map and the second feature map to serve as the output of the feature mutual injection module;

[0011] inputting the output of the feature mutual injection module and the output of the first residual block in the 2D feature extraction module into a 3D gesture prediction network in a skip connection manner to predict 3D hand joint positions and simultaneously predict a depth map as supervision.

[0012] The second aspect of the application provides a 3D gesture estimation system based on an attention mechanism and a graph convolution, comprising:

[0013] a 2D feature extraction module configured to extract features of the obtained hand RGB image by using the 2D feature extraction module to obtain a 2D heat map and a contour of the hand RGB image;

[0014] a feature strengthening module configured to input the 2D heat map into a graph convolutional neural network to strengthen the connection between adjacent joints, and obtain a 2D heat map with strengthened features;

[0015] a feature mutual injection module configured to input the 2D heat map with strengthened features and the contour map into the feature mutual injection module, inject 2D joint position information into the contour map through a feature injection operation to obtain a first feature map, and inject global information related to predicted joint coordinates in the contour map into the features of the 2D heat map to obtain a second feature map; and splice the first feature map and the second feature map to serve as the output of the feature mutual injection module;

[0016] a 3D gesture estimation module configured to input the output of the feature mutual injection module and the output of the first residual block in the 2D information extraction module into a 3D gesture prediction network in a skip connection manner to predict 3D hand joint positions and simultaneously predict a depth map as supervision.

[0017] The third aspect of the present application provides a computer readable storage medium, which stores a program, and the program is executed by a processor to implement the steps of the 3D gesture estimation method based on attention mechanism and graph convolution as described in the first aspect of the present application.

[0018] The fourth aspect of the present application provides an electronic device, which comprises a memory, a processor, and a program stored in the memory and executable on the processor, and the processor implements the steps of the 3D gesture estimation method based on attention mechanism and graph convolution as described in the first aspect of the present application when executing the program.

[0019] The above one or more technical solutions have the following beneficial effects:

[0020] (1) The present application designs a feature mutual injection (Mutual-Injection Transformer, MIT) module, which injects global information related to predicted joint coordinates in the contour map into the features of the 2D heat map, and at the same time injects more accurate joint position information in the 2D heat map into the features of the contour map, thereby improving the feature representation of the contour map and the heat map, and providing more sufficient and stable clues for the subsequent estimation of 3D gestures.

[0021] (2) The present application designs a Key-Neighbor-based GCN module, which uses the feature updating rule and limited propagation of the graph convolutional neural network to strengthen the connection between neighbor joints, specifically, an initial undirected graph is defined according to the connection of the hand joints in the real world to form an initial adjacency matrix, then two layers of graph convolutional neural network are deployed to make each joint obtain the features of its neighbor joints, and the parameters in the adjacency matrix are updated by back propagation to learn the implicit relationship between the joints.

[0022] The advantages of the additional aspects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be understood by the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0023] The drawings accompanying the specification of the present application form a part thereof, serve to provide further understanding of the present application, and together with the description of the exemplary embodiments of the present application and their description serve to explain the present application, and do not constitute improper limitations on the present application.

[0024] Figure 1 The method flowchart of the first embodiment.

[0025] Figure 2 The functional principle block diagram of the feature mutual injection MIT module of the first embodiment.

[0026] Figure 3PCK curves of the MIT model with feature mutual injection for the first embodiment compared with other models on the STB dataset.

[0027] Figure 4 PCK curves of the MIT model with feature mutual injection for the first embodiment compared with other models on the RHD dataset.

[0028] Figure 5 Visualization of the prediction results of the MIT model with feature mutual injection for the first embodiment on the STB dataset.

[0029] Figure 6 Visualization of the MIT model with feature mutual injection for the first embodiment on the RHD dataset.

[0030] Figure 7 (a) and (b) are the comparison charts of the 3D coordinates of the joints of the monocular RGB hand images predicted by the model and actually captured on different datasets, respectively. DETAILED DESCRIPTION

[0031] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0032] It should be noted that the terms used herein are only intended to describe specific embodiments and are not intended to limit the exemplary embodiments according to the present application.

[0033] The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0034] Embodiment One

[0035] As shown in Figure 1 The present embodiment discloses a 3D gesture estimation method based on attention mechanism and graph convolution, comprising the following steps:

[0036] Step 1, using a 2D feature extraction module to extract features from the acquired hand RGB image, obtaining a 2D heat map and a contour of the hand RGB image;

[0037] Step 2, inputting the 2D heat map into a graph convolutional neural network to strengthen the connection between neighboring joints, obtaining a 2D hand joint heat map with enhanced features;

[0038] Step 3, inputting the 2D heat map with enhanced features and the contour map into a feature mutual injection module, and completing the mutual feature injection between the 2D heat map features and the contour map features through the feature mutual injection mechanism;

[0039] Step 4, the output of the feature mutual injection module is input into the 3D gesture prediction network after being jump linked with the output of the first residual block in the 2D feature extraction module, and the 3D hand joint position is predicted, and the depth map is also predicted as supervision.

[0040] In step 1, the 2D feature extraction module includes an encoder and two decoders, and the structures of the encoder and the decoders are the same as those of the hourglass network. The hand RGB image is first input into the encoder, and the purpose is to extract the features of the image and retain the high-resolution spatial information. The encoder includes five residual blocks, each of which includes a convolution operation, a residual connection and a down-sampling. The five residual blocks successively extract features and obtain feature maps with resolutions of 64x64, 32x32, 16x16, 8x8 and 4x4, and the number of channels of each feature map is 256. The 4x4 feature map is input into two decoders for transposed convolution operation, and two feature maps with resolutions of 8x8, 16x16, 32x32 and 64x64 are obtained. The two decoders are jump linked with the feature maps with the same dimensions in the encoder, and after training, a 2D heat map and a contour map of the hand image with a resolution of 64x64 are obtained. Since the 2D heat map and the contour map are not homogeneous, in order to predict both of them, a two-part network structure is adopted in this part.

[0041] Preferably, the loss function of the 2D heat map is:

[0042]

[0043] wherein, represents the predicted value, represents the Gaussian distribution of the true value;

[0044] Preferably, the loss function of the contour map is:

[0045]

[0046] wherein, represents the confidence of the pixel belonging to the hand, represents whether the pixel belongs to the hand (if yes, then otherwise ).

[0047] In step 2, the Key-Neighbor-based GCN part regards each joint heat map in step 1 as the feature of the joint, and defines an initial undirected graph according to the connection of the hand joints in the real world to form an initial adjacency matrix. Then, the features of the neighbor joints are obtained by using limited propagation, and the parameters in the adjacency matrix are updated by using back propagation to learn the implicit relationship between the joints.

[0048] Specifically, the node feature update rule is as follows:

[0049]

[0050] Where σ is a nonlinear activation function, It is the degree matrix of the key points. F is the sum of the adjacency matrix of the hand joints in the real world and the identity matrix. n W represents the key feature and W represents the neural network weight.

[0051] The enhanced joint heatmap features and contour map features are input into the MIT module to complete the mutual injection between features.

[0052] In step 3, the structure of the feature mutual injection module MIT is as follows: Figure 2 As shown, by utilizing the attention mechanism and adopting a complementary information approach, heatmaps and contour maps are regarded as primary and secondary features, respectively. Information from the primary features is injected into the secondary features to enhance their feature representation.

[0053] Specifically, taking the process of injecting heatmap features into contour map features as an example: 1×1 convolution is used to extract the query matrix Q from the contour map. sil Extract the key-value matrix K from the heatmap. heat V heat Q sil With K heat The transpose of the matrix is ​​multiplied, and after passing through the softmax function, the first attention matrix is ​​obtained. To avoid unwanted high correlation, a sigmoid-based attention mechanism is used, extracting an additional query matrix Q from the contour map using 1×1 convolution. sig At the same time, an additional key matrix K is extracted from the heatmap. sig Q sig With K sig The transpose of the matrix is ​​multiplied, and after passing through the sigmoid function, the second attention matrix is ​​obtained. The first and second attention matrices are then multiplied element-wise to obtain the final attention matrix. This attention matrix is ​​then compared with the value matrix V from the heatmap. heat After performing matrix multiplication, the feature map obtained by injecting heatmap features into contour map features is obtained after passing through a fully connected layer and residual connections. This feature map is denoted as the first feature map.

[0054] Similarly, the process of injecting contour map features into heatmap features is as follows:

[0055] Extract the key matrix K from the contour map sil and the value matrix V sil Extracting the query matrix Q from the heatmap heat ;

[0056] The matrix Q heat is multiplied by the transpose of the matrix K sil , and the third attention matrix is obtained after passing through the softmax function. The third attention matrix is multiplied by the matrix V sil element by element, and the second feature map is obtained after passing through the full connection layer and residual connection.

[0057] Preferably, the expressions for extracting the query, key and value matrices from the heat map are as follows:

[0058] Q heat = X heat C Q1 K heat = X heat C K1 V heat = X heat C V1

[0059] The expressions for extracting the query, key and value matrices from the contour map are as follows:

[0060] Q sil = X sil C Q2 K sil = X sil C K2 V sil = X sil C V2

[0061] Wherein, X represents the output of the previous step, and C represents a 1x1 convolution layer.

[0062] Preferably, the MIT part includes a Softmax-based attention mechanism, a Sigmoid-based attention mechanism, a multi-head attention mechanism, and an MLP layer. The attention mechanism of the MIT part is represented as:

[0063]

[0064]

[0065] Wherein, d k represents the dimension of K heat , the query matrix Q heat is exchanged with Q sil .

[0066] In order to avoid the high correlation that is not desired, the sigmoid-based attention mechanism is used in this part.

[0067] The preferred sigmoid-based attention mechanism is represented as:

[0068]

[0069] The corresponding attention matrix Atten is further obtained corr is represented as:

[0070]

[0071] wherein, represents the multiplication of the corresponding position elements.

[0072] The output of the MIT part is preferably represented as:

[0073] F MIT = C(Cat(F H , F S ))

[0074] F H = F heat + Atten corr V heat + MLP(LN(Atten corr V heat ))

[0075] F S = F sil + Atten sil V sil + MLP(LN(Atten sil V sil ))

[0076] wherein, C represents a 1x1 convolution layer, Cat represents concatenating feature maps in the first dimension, F H represents injecting 2D heat map features into the contour feature process; F S represents injecting contour map features into the 2D heat map feature process; LN represents layer normalization. Through the MIT part, the 2D heat map feature contains global features from the contour map, and the contour map contains joint position information from the heat map.

[0077] In step 4, the 3D gesture estimation part maps the 2D joint coordinates to the three-dimensional space, while predicting the depth map as supervision. Due to the difference in granularity between 3D coordinates and depth maps, this part adopts the same binary hourglass network structure as the 2D feature extraction module.

[0078] The objective function of the 3D gesture estimation network is:

[0079] L 3D = λ D L D+λ J L J Here, L D represents the loss function of the depth map, and the smooth L1 loss is used to calculate the difference between the predicted depth map and the real depth map; L J is the loss function of 3D joint coordinates, and the mean square error is used to calculate the error between the predicted joint position and the real joint position; λ D and λ J respectively represent the weights of the depth map loss and the joint coordinate loss.

[0080] Preferably, the loss function of 3D joint coordinates is:

[0081]

[0082] wherein, represents the predicted value, represents the real value;

[0083] Preferably, the loss function of the depth map is:

[0084]

[0085] wherein, represents the predicted depth value, represents the real depth value.

[0086] The performance of the model proposed in the embodiment will be verified by experiments:

[0087] (I) In order to better illustrate the excellent gesture estimation performance of the model proposed in the present application, comparative experiments were carried out with the existing advanced gesture estimation model, and the results are shown in Table 1.

[0088] Here, the data set used and the common evaluation index in 3D gesture estimation are briefly introduced.

[0089] Stereo Tracking Benchmark data set: This data set is a real data set, which contains two subsets STB-BB and STB-SK, and the resolution of the image is 640x480. Each hand image has complete 2D and 3D labels of 21 joints, as well as the corresponding segmentation map and depth map. 15000 samples are selected as the training set, and the remaining 3000 samples are selected as the test set.

[0090] Rendered hand pose Dataset: This dataset is composed of synthetic images, performed by 20 characters for 39 hand poses, including 41,258 training images and 2,728 test images. In addition to containing the 2D / 3D labels of each hand joint, the dataset also provides the corresponding depth map and hand segmentation map. Although it is synthetic, this dataset is more challenging than the STB dataset because the different viewpoints, different hands of multiple characters introduce huge visual diversity. In addition, some poses even have serious self-occlusion, which makes accurate prediction more difficult.

[0091] Evaluation index: PCK index is usually used to evaluate the accuracy and stability of various key point detection algorithms. In gesture estimation, PCK represents the proportion of correctly estimated joints. If the distance between the predicted joint and the real joint position is within a certain threshold, the predicted joint position is considered correct. AUC index represents the area under the PCK curve, whose value ranges from 0 to 1. When the value is 0, the prediction effect is the worst, and when the value is 1, the prediction effect is the best.

[0092] Table 1

[0093]

[0094] As can be seen from Table 1, the AUC evaluation index of the MIT provided in the embodiment on the STB dataset and the RHD dataset is 0.998 and 0.954, respectively, which exceeds the performance of the previous SOTA model. The PCK curve of the MIT model compared with other models on the STB dataset is shown in Figure 3 , and the performance of the MIT model at a threshold of 20mm-50mm is significantly better than that of other models. The PCK curve of the MIT model compared with other models on the RHD dataset is shown in Figure 4 , and the performance of the MIT model at a threshold of 20mm-50mm is significantly better than that of other models.

[0095] (II) Qualitative analysis of the MIT model on the benchmark dataset;

[0096] Select several groups of samples in the STB and RHD datasets respectively, predict the 3D coordinates of the hand joints and compare them with the true values. In order to more clearly show the results, the embodiment selects two different perspectives to show the prediction results of the MIT model, Figure 5 is the visualization of the prediction results of the MIT model on the STB dataset, Figure 6 is the visualization of the MIT model on the RHD dataset. Obviously, the MIT model provided in the embodiment can stably predict 3D hand joint coordinates under low light, self-occlusion and various complex hand pose conditions.

[0097] To evaluate the generalization ability of the MIT model, 3D coordinates of joints from network and real captured monocular RGB hand images were predicted, as shown in Figure 7 These hand images have no correct labels and are more difficult to predict than images in the benchmark dataset, such as dim light, motion blur when shooting, etc. Qualitative results show that the MIT model also has certain reasoning ability and potential to handle images not in the dataset.

[0098] (III) In addition to comparing with other models, further comparative experiments were set up to verify the role of each module of the MIT model (Table 2). The experimental objects include:

[0099] (I) Remove the Key-Neighbor-based GCN module and the MIT module in the MIT model.

[0100] (II) Remove the MIT module in the MIT model.

[0101] (III) Remove the Key-Neighbor-based GCN module in the MIT model.

[0102] (IV) Complete MIT model.

[0103] Table 2

[0104]

[0105] As can be seen from the first row of Table 2, the lack of Key-Neighbor-based GCN module and MIT module means that the 2D heat map and contour map features are directly connected to the 3D gesture estimation network without further enhancing the features after splicing, so the prediction of this model is far from the prediction result of the complete model of the application, which shows that the Key-Neighbor-based GCN module and the MIT module of the application can improve the accuracy and robustness of the prediction.

[0106] The MIT model uses the MIT module to complete the mutual injection between the 2D heat map features and the contour map features. If the MIT module is missing, the model cannot effectively inject features between the two, so the features of the two cannot be fully utilized. For specific results, see the second row of Table 2. In addition, the model without MIT performs much worse than the complete model, which shows that the MIT module can complete the task of mutual complementation between structurally heterogeneous features and improve the accuracy of prediction and model robustness. The MIT module can effectively alleviate the problems of self-occlusion, motion blur, etc. in 3D gesture estimation.

[0107] The MIT model uses a Key-Neighbor-based GCN module to enhance the connection between hand joints, especially between adjacent joints. If the Key-Neighbor-based GCN module is missing, the accuracy of the model will be significantly reduced, as shown in Table 2, line 3. The related PCK curve also shows the gap in performance. These results show that the Key-Neighbor-based GCN module can further enhance the feature representation of the hand during the inference stage, improving the prediction accuracy by emphasizing the relationship between adjacent nodes.

[0108] Embodiment Two

[0109] The embodiment discloses a 3D gesture estimation system based on an attention mechanism and a graph convolution, comprising:

[0110] The 2D feature extraction module is configured to perform feature extraction on the obtained hand RGB image using the 2D feature extraction module to obtain a 2D heat map and an outline of the hand RGB image.

[0111] The feature enhancement module is configured to input the 2D heat map into a graph convolutional neural network to enhance the connection between adjacent nodes, and obtain a 2D heat map after feature enhancement.

[0112] The feature mutual injection module is configured to input the 2D heat map after feature enhancement and the outline map into the feature mutual injection module, inject 2D node position information into the outline map through a feature injection operation to obtain a first feature map, and inject global information related to the predicted node coordinates in the outline map into the features of the 2D heat map to obtain a second feature map; the first feature map and the second feature map are spliced and output as the output of the feature mutual injection module.

[0113] The 3D gesture estimation module is configured to input the output of the feature mutual injection module and the output of the first residual block of the 2D information extraction module into the 3D gesture prediction network after jumping link, predict the 3D hand node position, and predict the depth map as supervision.

[0114] Embodiment Three

[0115] The purpose of the embodiment is to provide a computer-readable storage medium.

[0116] The computer-readable storage medium stores a computer program, which is executed by a processor to implement the steps of the medical image segmentation method based on self-attention and information bottleneck according to Embodiment 1 of the present disclosure.

[0117] Embodiment Four

[0118] The purpose of the embodiment is to provide an electronic device.

[0119] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, and the processor implements the steps in the method for denoising a medical image feature map based on a diffusion model according to Embodiment 1 of the present disclosure when executing the program.

[0120] The steps involved in the devices of the above embodiments two, three, and four correspond to the method of Embodiment 1, and the specific implementation can be referred to the relevant description part of Embodiment 1. The term “computer readable storage medium” should be understood to include a single medium or multiple media that store one or more sets of instructions; it should also be understood to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by a processor and causing the processor to perform any one of the methods of the present disclosure.

[0121] Those skilled in the art should understand that each module or step of the present disclosure described above can be implemented by a general-purpose computer device, and alternatively, they can be implemented by program codes executable by a computing device, so that they can be stored in a storage device for execution by a computing device, or they can be respectively manufactured into individual integrated circuit modules, or a plurality of modules or steps among them can be manufactured into a single integrated circuit module to implement. The present disclosure is not limited to any specific combination of hardware and software.

Claims

1. A 3D gesture estimation method based on attention mechanism and graph convolution, characterized in that, The method comprises the following steps: The 2D feature extraction module is configured to perform feature extraction on the obtained hand RGB image to obtain a 2D heat map and a contour of the hand RGB image; The feature enhancement module is configured to input the 2D heat map into a graph convolutional neural network to strengthen the connection between adjacent joints, and obtain a 2D hand joint heat map with enhanced features; The feature mutual injection module is configured to input the 2D heat map with enhanced features and the contour into the feature mutual injection module, inject 2D joint position information into the contour through a feature injection operation to obtain a first feature map, and inject global information related to the predicted joint coordinates in the contour into the features of the 2D heat map to obtain a second feature map; the first feature map and the second feature map are spliced to serve as the output of the feature mutual injection module; The output of the feature mutual injection module is linked to the output of the first residual block of the 2D feature extraction module in a jump connection mode, and is input into the 3D gesture prediction network to predict 3D hand joint positions and a depth map as supervision; The 2D joint position information is injected into the contour, which comprises: extracting the query matrix from the profile map and the matrix extracting the key matrix in the heat map , and the value matrix ; matrix and the transpose matrix of the matrix are multiplied to obtain the first attention matrix after the softmax function The matrix is multiplied by the transpose of the matrix , and the second attention matrix is obtained after the sigmoid function; the first attention matrix and the second attention matrix are multiplied element by element, and the matrix is multiplied, and finally the first feature map is obtained after the full connection layer and the residual connection. The global information related to the predicted joint coordinates in the contour is injected into the features of the 2D heat map, which comprises: extracting a key matrix from a profile graph and a value matrix extracting a query matrix in a heat map ; matrix matrix matrix matrix 2. The 3D gesture estimation method based on attention mechanism and graph convolution of claim 1, wherein, The 2D feature extraction module comprises an encoder and two decoders, and the structures of the encoder and the decoders are the same as those of a sandglass network; The hand RGB image is input into the encoder, the encoder comprises five residual blocks, the five residual blocks sequentially extract features and obtain image features with different resolutions; the features extracted by the last residual block are input into two decoders for transposed convolution operation, each decoder sequentially extracts features, and the two decoders and the feature maps with the same dimension in the encoder are linked in a jump connection mode to simultaneously obtain a 2D heat map and a contour of the hand RGB image.

3. The 3D gesture estimation method based on attention mechanism and graph convolution of claim 1, wherein, The 2D heat map is input into the graph convolutional neural network to strengthen the connection between adjacent joints, which comprises: The heat map of each joint is regarded as the feature of the joint, an undirected graph is constructed according to the relationship between the hand joints in the real world, and then an initial adjacency matrix is formed; Limited propagation is used to enable the current node to obtain the features of its neighbor nodes, and the parameters in the adjacency matrix are updated according to back propagation to learn the implicit features between the joints; two layers of graph convolution are deployed to complete the limited propagation, thereby strengthening the feature representation of the joints.

4. The 3D gesture estimation method based on attention mechanism and graph convolution of claim 3, wherein, The node feature update rule is: wherein, is a non-linear activation function, is a degree matrix of the joint nodes, is an adjacency matrix of the hand joints in the real world plus an identity matrix, represents the joint node features, denotes the neural network weights.

5. The 3D gesture estimation method based on attention mechanism and graph convolution of claim 1, wherein, The target function of the 3D gesture prediction network is: In the formula, represents the loss function of the depth map, and the smooth L1 loss is used to calculate the difference between the predicted depth map and the real depth map; represents the loss function of the 3D joint coordinates, and the mean square error is used to calculate the error between the predicted joint position and the real joint position; and respectively represent the weights of the depth map loss and the joint coordinate loss.

6. An attention mechanism and graph convolution based 3D gesture estimation system, characterized in that: It comprises: The 2D feature extraction module is configured to perform feature extraction on the obtained hand RGB image to obtain a 2D heat map and a contour of the hand RGB image; The feature enhancement module is configured to input the 2D heat map into a graph convolutional neural network to strengthen the connection between adjacent joints, and obtain a 2D heat map with enhanced features; The feature mutual injection module is configured to: input the feature-enhanced 2D heat map and the contour map into the feature mutual injection module, inject the 2D joint position information into the contour map through a feature injection operation to obtain a first feature map, and inject global information related to a predicted joint coordinate in the contour map into features of the 2D heat map to obtain a second feature map; and splice the first feature map and the second feature map to serve as an output of the feature mutual injection module; The 3D gesture estimation module is configured to: input the output of the feature mutual injection module and the output of the first residual block in the 2D information extraction module in a skip connection to the 3D gesture prediction network to predict 3D hand joint position and predict a depth map as supervision; The injection of the 2D joint position information into the contour map includes: extracting the query matrix from the profile map and the matrix extracting the key matrix in the heat map , and the value matrix ; matrix matrix multiplication with the transpose matrix of the matrix , and the first attention matrix is obtained after the softmax function The matrix is multiplied by the transpose of the matrix , and the second attention matrix is obtained after passing through the sigmoid function; the first attention matrix and the second attention matrix are multiplied element by element, and the matrix is multiplied, and finally the first feature map is obtained after passing through the full connection layer and the residual connection. The injection of the global information related to the predicted joint coordinate in the contour map into the features of the 2D heat map includes: extracting a key matrix from a profile graph and a value matrix extracting a query matrix in a heat map ; The matrix is multiplied by the transpose of the matrix , and the third attention matrix is obtained after passing through a softmax function. The third attention matrix is multiplied element by element with the matrix , and the second feature map is obtained after passing through a fully connected layer and a residual connection.

7. A computer-readable storage medium having stored thereon a program, characterized in that, The program is executed by the processor to implement the steps in the 3D gesture estimation method based on the attention mechanism and the graph convolution according to any one of claims 1-5.

8. An electronic device comprising a memory, a processor, and a program stored in the memory and capable of running on the processor, characterized by The processor executes the program to implement the steps in the 3D gesture estimation method based on the attention mechanism and the graph convolution according to any one of claims 1-5.

Images