A gesture reconstruction method based on image point cloud multi-modal fusion and joint guidance

Abstract

The present application belongs to the field of human-computer interaction and computer vision, and particularly relates to a gesture reconstruction method based on image point cloud multi-modal fusion and joint guidance, which comprises taking a hand depth map and a three-dimensional point cloud as input, using a feature extraction network for a two-dimensional depth image and a three-dimensional point cloud to extract key point features of the depth map and the three-dimensional point cloud respectively; then using a multi-modal feature and global feature fusion module to provide additional global information for the multi-modal feature; then using a joint coordinate guided feature fusion iteration module to further optimize and update the fusion features, so as to improve the reconstruction accuracy of the three-dimensional gesture, and at the same time, through multiple iteration updates, the accurate estimation of each key point in the hand posture is realized. The present application effectively combines the depth image information and the point cloud spatial geometric structure features, and at the same time, uses the key point features to aggregate the feature information of each modal, reduces the redundant interaction of invalid features, and improves the efficiency of multi-modal fusion.

Description

Technical Field

[0001] This invention belongs to the fields of human-computer interaction and computer vision, and specifically relates to a gesture reconstruction method based on multimodal fusion of image point clouds and joint guidance. Background Technology

[0002] With the rapid development of computer vision and artificial intelligence technologies, gesture recognition and 3D gesture reconstruction have become important research directions in fields such as human-computer interaction, virtual reality (VR), and augmented reality (AR). Many of these more challenging applications demand higher accuracy in 3D hand pose estimation. For example, augmented reality applications involve frequent hand movements and occluded hand-object interactions. In 3D gesture reconstruction tasks, accurately predicting the coordinates of key hand points can effectively analyze user hand movements and behaviors, forming the foundation for efficient human-computer interaction.

[0003] With the advancement of deep learning technology and the widespread availability of low-cost depth cameras, significant progress has been made in 3D hand pose estimation based on depth images. However, some scenarios still present challenging problems, mainly including the following: (1) Early gesture reconstruction methods were mostly based on depth images. For example, the method proposed in patent application CN102262783A treats depth data as a single-channel two-dimensional image, inevitably ignoring the three-dimensional properties of the depth data; (2) In dynamic gesture recognition tasks, some methods directly extract local hand features using hand point cloud data, such as the method proposed in patent application CN11. Patent applications 8887736A and CN118202987A, due to their disordered and unstructured characteristics, require very complex feature extractors. Furthermore, since their task objective is recognition and the focus is not on three-dimensional gesture reconstruction, the accuracy of the obtained hand joint coordinates is low. (3) At present, there are very few methods that utilize both depth images and point cloud data. For example, patent application CN117351573A only uses depth images to obtain two-dimensional hand joint coordinates, ignoring the ability of depth images to describe local contours and reflect local depth changes. It does not fully explore and utilize the features of depth images and hand point clouds. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention proposes a gesture reconstruction method based on multimodal fusion of image point clouds and joint guidance. A gesture reconstruction model is constructed, and this model is used to reconstruct gestures from depth images. The reconstruction process includes the following steps:

[0005] 101. Acquire hand depth images and their point cloud data. The image feature extraction network extracts local and global image features from the depth images, and the point cloud feature extraction network extracts global point cloud features from the point cloud data.

[0006] 102. The global image features and global point cloud features are concatenated together, and a bias embedding vector is generated for each key point through an induced bias layer. The vectors of all key points constitute the global key point embedding.

[0007] 103. Global keypoint embedding: The high-dimensional embedding is projected onto the three-dimensional space through linear transformation to obtain the initial three-dimensional joint coordinates, and local point cloud features are obtained based on the three-dimensional joints.

[0008] 104. Update the coordinates of the 3D joints using a graph convolutional network to obtain the updated global features. Then, concatenate the global features with the local image features and the local point cloud features to obtain the image joint features and the point cloud joint features.

[0009] 105. Based on the attention mechanism, the joint features of the image and the joint features of the point cloud are weighted respectively, and then the weighted features are updated using a graph convolutional network.

[0010] 106. Predict 3D joint coordinates using the updated image joint features and point cloud joint features from the graph convolutional network. Determine if the iteration limit has been reached. If not, update the local point cloud features based on the predicted 3D joints and return to step 104. Otherwise, use the currently predicted 3D joint coordinates as the reconstructed structure.

[0011] Furthermore, the process by which the image feature extraction network extracts local and global image features from depth images includes:

[0012] The depth image is processed by an autoencoder network to obtain a two-dimensional local feature map F. 2d ∈R C×H×W Where C, H, and W are two-dimensional local feature maps F, respectively. 2d The number of channels, height, and width;

[0013] Two-dimensional local feature map F 2d The input is fed into a convolutional residual network to generate a heatmap for each key point. The two-dimensional UV coordinates of the joints are estimated based on normalized heatmaps, and local image features are obtained by aggregating joint image features from local feature maps based on the heatmaps. Where J is the number of hand joints in the depth image, and d 2d Image feature dimensions corresponding to each joint;

[0014] Two-dimensional local feature map F 2d A global image feature is output through global pooling.

[0015] Furthermore, the process by which the point cloud feature extraction network extracts local and global point cloud features from point cloud data includes:

[0016] The three-dimensional local geometric features F are extracted from the point cloud data through point set convolutional layers. 3d ∈R N×d Where N represents the number of points in the point cloud, and d represents the feature dimension of the point cloud corresponding to each point;

[0017] Three-dimensional local geometric features F 3d ∈R N×d After passing through a one-dimensional convolutional layer and a max pooling layer, global point cloud features are obtained.

[0018] The two-dimensional local feature map F 2d The point cloud feature F is then projected onto the feature space of the 3D local geometric features and then stitched together with the 3D local geometric features to generate the fused point cloud feature F. fuse ;

[0019] Using the initialized 3D joint coordinates as the starting position, the point cloud feature F is further guided. fuse Aggregate the data to generate the final local point cloud features. Where d 3d This represents the feature dimension of the point cloud corresponding to each joint.

[0020] Furthermore, the process of weighting image joint features and point cloud joint features based on the attention mechanism includes:

[0021] By mapping image joint features and point cloud joint features through two linear modules respectively, query vectors, key vectors and value vectors of image joint features and point cloud joint features are obtained;

[0022] The query vectors of image joint features and point cloud joint features are concatenated together, the key vectors are concatenated together, and the value vectors are concatenated together.

[0023] The concatenated query vector, key vector, and value vector are input into the attention module to obtain attention weights. These attention weights are then used to weight the image joint features and point cloud joint features.

[0024] Compared with existing technologies, this invention effectively combines depth image information with point cloud spatial geometric features, and uses keypoint features to aggregate feature information from each modality, reducing redundant interactions of invalid features and improving the efficiency of multimodal fusion. Furthermore, this invention deeply mines complementary information between multiple modalities through a joint coordinate-guided strategy, which helps reduce interference from ambiguous information in cross-modal feature fusion, significantly improving the accuracy and robustness of gesture 3D reconstruction. Moreover, the proposed method has been experimentally validated on challenging public gesture datasets such as NYU and DexYCB, demonstrating that this method significantly outperforms previous methods in 3D gesture reconstruction. Attached Figure Description

[0025] Figure 1 This is a flowchart of the three-dimensional gesture reconstruction method based on image point cloud multimodal fusion and joint guidance of the present invention;

[0026] Figure 2 Figure (a) shows the input data for two different modalities in this invention, where Figure (b) is a deep depth image and Figure (a) is hand point cloud data.

[0027] Figure 3 Figure (a) shows the reconstruction result on a single-hand dataset, and Figure (b) shows the reconstruction result on a hand-object interaction dataset.

[0028] Figure 4 This invention provides a network framework for a three-dimensional gesture reconstruction method based on image point cloud multimodal fusion and joint guidance. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] This invention proposes a gesture reconstruction method based on multimodal fusion of image point clouds and joint guidance. A gesture reconstruction model is constructed, and this model is used to reconstruct gestures from depth images, such as... Figure 1 The reconstruction process includes the following steps:

[0031] 101. Acquire hand depth images and their point cloud data. The image feature extraction network extracts local and global image features from the depth images, and the point cloud feature extraction network extracts global point cloud features from the point cloud data.

[0032] 102. The global image features and global point cloud features are concatenated together, and a bias embedding vector is generated for each key point through an induced bias layer. The vectors of all key points constitute the global key point embedding.

[0033] 103. Global keypoint embedding: The high-dimensional embedding is projected onto the three-dimensional space through linear transformation to obtain the initial three-dimensional joint coordinates, and local point cloud features are obtained based on the three-dimensional joints.

[0034] 104. Update the coordinates of the 3D joints using a graph convolutional network to obtain the updated global features. Then, concatenate the global features with the local image features and the local point cloud features to obtain the image joint features and the point cloud joint features.

[0035] 105. Based on the attention mechanism, the joint features of the image and the joint features of the point cloud are weighted respectively, and then the weighted features are updated using a graph convolutional network.

[0036] 106. Predict 3D joint coordinates using the updated image joint features and point cloud joint features from the graph convolutional network. Determine if the iteration limit has been reached. If not, update the local point cloud features based on the predicted 3D joints and return to step 104. Otherwise, use the currently predicted 3D joint coordinates as the reconstructed structure.

[0037] In this embodiment, a specific implementation process of a gesture reconstruction method based on image point cloud multimodal fusion and joint guidance is proposed, which includes the following steps:

[0038] Step 1: Using different feature extraction networks, extract the corresponding multimodal features, global features, and initialized 3D key point coordinates from the input hand depth image and 3D point cloud, respectively. The depth image feature extraction network mainly captures the local details and depth change information of the hand, while the point cloud feature extraction network mainly obtains the 3D structural information and spatial features of the hand. In this embodiment, the multimodal features include depth image features and 3D point cloud features.

[0039] Step 2: Use the feature fusion module to fuse multimodal features with global features to obtain multimodal keypoint features that have been fused with global information. By fusing global information with keypoint features from different modalities, the loss of global features can be compensated for, and the error caused by occlusion in keypoint prediction can be reduced.

[0040] Step 3: Use the initialized 3D keypoint coordinates to guide the keypoint features of the depth map and the keypoint features of the point cloud to further interact and fuse, thereby improving the semantic and spatial consistency of the features. This step uses the initial keypoint coordinates as a guide to achieve further coupling and optimization of the depth map and point cloud data through feature interaction.

[0041] Step 4: Based on the idea of ​​iterative correction, the multimodal features are updated. The gestures are gradually corrected using two-dimensional visual features and three-dimensional geometric structure information, and finally an accurate and robust three-dimensional gesture model is obtained. Through the iterative optimization mechanism, the joint positions are gradually approximated and the model is accurately reconstructed.

[0042] like Figure 4This invention extracts two-dimensional local feature maps and three-dimensional local features from depth maps and point cloud data using an image encoder and a point cloud encoder, respectively. Global features are obtained based on the two-dimensional local feature maps and three-dimensional local features. The two-dimensional local feature maps and three-dimensional local features are further processed to obtain point cloud joint features and depth joint features. The point cloud joint features and depth joint features are concatenated with the global features and then feature fusion is performed. The joint regressor predicts the three-dimensional joint coordinates based on the input fused features.

[0043] In this embodiment, the hand depth image is a cropped image focusing on the hand region. It can be a single hand image or a hand-object interaction image. The hand depth image is acquired by a depth camera. Figure 2 (a) Provide an example of a hand depth image; the 3D point cloud is derived from the depth image using camera parameters, i.e., the point cloud data is generated by backprojecting the depth map. Figure 2 (a) Depth map projection generation Figure 2 (b) The corresponding point cloud data, depth image and point cloud data are processed by image processing branch and point cloud processing branch respectively to extract the corresponding local features and global information, and the coordinates of three-dimensional key points are initialized at the same time.

[0044] In the image processing branch, the depth image is first processed by an autoencoder network to obtain a two-dimensional local feature map F. 2d ∈R C×H×W Subsequently, the local feature map F 2d The input is fed into a convolutional residual network to generate a heatmap for each key point. Furthermore, based on the heatmap, the image features of the joints are aggregated from the local feature maps. In particular, the two-dimensional UV coordinates of the joints can be estimated through heatmaps, which are supervised by the real UV coordinates, ensuring the accuracy and robustness of image feature extraction.

[0045] To optimize the predicted keypoint UV coordinates (U' j V j ') and the actual joint UV coordinates (U j V j The error between (i.e., Ground Truth, GT) is supervised by the Smooth L1 loss function in this embodiment, and its loss formula is as follows:

[0046]

[0047] Where J is the number of hand joints in the depth image; L(x) is the standard Smooth L1 loss function, defined as:

[0048]

[0049] This loss function is robust. When the prediction error is small, the loss is a quadratic function, providing a strong gradient signal. When the error is large, the loss is linear, avoiding the excessive influence of outliers on the training process, thus improving the stability of network training. By optimizing this loss function, the deviation between the predicted UV coordinates and the true coordinates can be effectively reduced, providing higher accuracy for subsequent 3D gesture reconstruction.

[0050] In the point cloud processing branch, three-dimensional local geometric features F are extracted from point cloud data through point set convolutional layers (such as PointNet or PointNet++). 3d ∈R N×d To enhance the multimodal interaction between depth maps and point clouds, the two-dimensional local feature map F is integrated into the three-dimensional space. 2d The feature F obtained by projection proj With point cloud local features F 3d The points are stitched together to generate the fused point cloud features F. fuse Furthermore, using the initialized 3D keypoint coordinates as the starting position, local feature aggregation is further guided to generate the final point cloud joint features.

[0051] Specifically, the process of guiding local feature aggregation involves, for each joint, using the sphere query algorithm to find K neighboring points within a radius r around that location in the point cloud data, centered on the joint's 3D coordinates. The geometric features of these points are then used as the initial point cloud features for the joint. This step ensures that the point cloud features effectively capture the local geometric information near each joint, laying the foundation for subsequent fusion processing.

[0052] Meanwhile, after the depth image and point cloud data are processed through their respective feature extraction branches, not only are the corresponding local features obtained, but also image global vectors G containing global information are generated. img and point cloud global vector G pc Next, the two global vectors are concatenated and copied J times to provide to the induced bias layer, generating an independent bias embedding vector for each joint. This operation is equivalent to generating a learnable position embedding for each joint, which helps to improve the localization accuracy of the key points. After generating the global key point embedding, the high-dimensional embedding is projected onto the three-dimensional space through a linear transformation to obtain the initial three-dimensional joint coordinate position.

[0053] To improve the accuracy of initial 3D joint coordinate prediction and provide more accurate geometric references and spatial guidance information for subsequent fusion and interaction, a loss function for the initial 3D joint positions is introduced here. This loss function adopts the form of Smooth L1 loss and is defined as follows:

[0054]

[0055] in, Let be the three-dimensional joint position coordinates of the j-th joint predicted by the network. Let be the true 3D coordinates of the j-th joint.

[0056] In this invention, global features refer to the keypoint embeddings generated in the first step. These keypoint embeddings are further processed by a graph convolutional network (GCN) to capture the geometric relationships between keypoints. One of the core features of a GCN is a learnable adjacency matrix. Traditional GCNs typically rely on predefined adjacency matrices; however, in multimodal feature processing and global feature extraction tasks, predefined adjacency relationships may struggle to fully adapt to the actual data distribution. To overcome this limitation, this invention introduces a dynamic adjacency matrix. A dynamic adjacency matrix is ​​a learnable kinematic matrix that adaptively adjusts the connection strength between nodes based on the features of the input data, thereby more accurately modeling the semantic and geometric relationships between keypoints. By dynamically adjusting adjacency relationships, GCN can not only capture local geometric relationships but also uncover global semantic relationships, giving the model a higher representational ability for complex relationships between keypoints. This provides a more accurate semantic and geometric foundation for subsequent feature-level connections between image and point cloud features. Simultaneously, its adaptive nature ensures the model's generalization ability to the input data, improving the overall performance of the multimodal feature fusion and interaction modules.

[0057] After updating the global features, they are concatenated with features extracted from the image branch and the point cloud branch, respectively, to achieve effective fusion of global features and multimodal features. Specifically, this includes the following steps:

[0058] Joint image features extracted from image branches Integrate with the updated keypoint embedding in the following ways:

[0059]

[0060] Where Concat represents the feature-level connection operation, updated_embedding is the keypoint embedding; the generated image joint features F' j It not only preserves the characteristics of image modalities, but also integrates global joint semantics and geometric relationships;

[0061] Joint point cloud features extracted from point cloud branches It is also combined with the updated keypoint embedding in a similar way:

[0062]

[0063] Among them, the generated point cloud joint features F” j While preserving the modal characteristics of point clouds, supplementary global information is introduced.

[0064] Based on their respective modal characteristics, the image joint features and point cloud joint features of this invention introduce keypoint embedding containing global information, making the features of each modality more consistent in spatial geometry and semantic information. The fused multimodal features have stronger expressive power and provide rich feature representations for subsequent multimodal feature interaction modules.

[0065] This invention guides further interaction and fusion between depth map keypoint features and point cloud keypoint features by using initialized 3D keypoint coordinates. Through the aforementioned feature fusion module, joint image features and joint point cloud features incorporating global information are generated. To further enhance the semantic relevance and spatial consistency among these features, 3D joint coordinates are introduced as additional guiding information to ensure precise spatial alignment of features and facilitate depth interaction between different modalities.

[0066] The fusion of joint image features and point cloud features is achieved through an attention mechanism. Specifically, image features and point cloud features are fed into a multi-head attention module, where the relationship between each joint feature and other joint features is dynamically adjusted by calculating its attention weight. In this way, the correlation between features is significantly enhanced, while the influence of irrelevant information is suppressed.

[0067] To ensure accurate spatial alignment of image and point cloud features, initialized 3D joint coordinates are introduced into the attention mechanism as guiding information. Specifically, the 3D coordinates influence attention computation through an adaptive adjustment mechanism, ensuring that feature interaction depends not only on the semantic information of the features themselves but also on their spatial positional relationships. In implementation, depth map joint features and point cloud joint features are passed to two different linear blocks. During the computation of each linear block, the 3D joint coordinates serve as conditional inputs, generating a set of adjustment parameters. These parameters control the behavior of the multi-head self-attention and multilayer perceptron modules within the attention mechanism, thereby adjusting the feature interaction method.

[0068] During the calculation, the depth map joint features F' and the point cloud joint features F” are processed separately and then...

[0069] jj

[0070] The integration of attention mechanisms includes the following steps:

[0071] First, F is tested using two linear modules. j 'and F j "The features are linearly transformed to generate three vectors: query (Q), key (K), and value (V);"

[0072] Next, the query, key, and value vectors of the image and point cloud features are concatenated to form a joint query, key, and value matrix—q, k, v;

[0073] By concatenating the features of these two different modalities, the model can simultaneously focus on the relationship between the two modalities. The joint query, key, and value matrix is ​​input into the self-attention module AttenBlock to calculate the attention output.

[0074] The attention module is responsible for performing a weighted summation of the features based on the attention weights, thereby obtaining the weighted sum.

[0075] Output F j and F j ;

[0076] Finally, the attention output is updated with weights based on the scaling, offset, and gating signal vectors obtained from the previous two linear blocks. This weighted update ensures that the features can adjust their weights according to the spatial relationship between joints during the interaction with the 3D coordinates.

[0077] After processing by the attention module, the processed features are input into the Graph Convolutional Network (GCN). Here, the GCN utilizes a predefined joint adjacency matrix, constructed based on the spatial relationships and physical connections between joints, capturing the topological structure and geometric constraints between them. The graph convolution operation fuses information from neighboring nodes through the adjacency matrix, ensuring that each node's features contain the semantic and geometric information of other nodes in its neighborhood.

[0078] For the image joint features and point cloud joint features obtained after graph convolutional network processing, this embodiment adopts an iterative update method to further process the feature data. First, new three-dimensional joint coordinates are predicted based on the image joint features and point cloud joint features obtained after graph convolutional network processing. Next, these updated features will guide the entire process to continue feature fusion and interaction. Through gradual correction, the spatial structure and geometric shape of the gesture are further optimized.

[0079] During iterative calibration, the model uses the output features of the previous GCN step to predict new 3D joint coordinates. The updated 3D coordinates consider not only 2D visual features but also 3D geometric information, ensuring a more accurate spatial structure of the gesture. Furthermore, the new 3D joint coordinates replace the initialized 3D joint coordinates during iteration, used to extract more accurate joint point cloud features from the point cloud space. Based on the new 3D joint coordinates, the loss between the prediction and the true 3D coordinates is calculated, and the model is further optimized. This process helps to gradually reduce the error between the predicted and true 3D coordinates. The loss function for the iterative process is expressed as:

[0080]

[0081] Among them, l s P represents the loss function during a single feature iteration. j '(t) represents the predicted 3D joint coordinates of the j-th joint after the t-th iteration update.

[0082] In summary, the total loss of the gesture reconstruction model of this invention is expressed as:

[0083]

[0084] Among them, L total α represents the total loss of the gesture reconstruction model; α, β, and γ are the weight coefficients of the loss function; and T is the number of feature iterations.

[0085] This invention uses multiple iterations to gradually correct the predicted 3D joint coordinates of the gesture reconstruction model. Each iteration updates features and provides error feedback based on the previous iteration. The optimization process provides the model with more spatial information and geometric constraints, ensuring that the final 3D gesture prediction is more accurate and robust. After several iterations of correction, the final 3D joint coordinate prediction is obtained. At this point, the gesture reconstruction model not only accurately reflects the spatial layout of the gesture, but also has strong robustness and can adapt to different postures and external interference.

[0086] This embodiment also presents a performance comparison of three-dimensional gesture reconstruction methods based on image point cloud multimodal fusion and joint guidance, as follows:

[0087] 1) Dataset

[0088] The NYU dataset contains 8252 test frames and 72,757 training frames, including captured RGBD data and ground truth annotations for hand poses. This dataset includes multiple viewpoints and annotations, suitable for training and evaluating gesture estimation models. Each frame in the dataset contains RGBD data from three Kinect cameras, providing one frontal view and two side views. The training set data comes from a single user, while the test set includes samples from two users. Furthermore, the dataset's annotations include the positions of 36 joints, of which we selected 14 for training and evaluation.

[0089] 2) Performance Comparison

[0090] Figure 3 (a) A rendering of the reconstructed single hand is provided, where blue lines represent actual joints and red lines represent joints predicted by this invention. Figure 3 (b) A reconstructed structure of the object is also provided. To verify the effectiveness of the proposed method, this embodiment compares the proposed method with current reconstruction methods based on depth images or point clouds in detail, including methods combining depth images and point clouds. The performance comparison between the present invention and the prior art is shown in Table 1.

[0091] Table 1. Comparison of the gesture reconstruction method of the present invention with other reconstruction methods in terms of objective performance.

[0092] method Input data MPJPE DepthPrior++ Depth 12.24 A2J Depth 8.61 AWR Depth 7.48 HandPointNet Point 10.54 Hand Folding Point 8.58 HandR2N2 Point 7.27 IPNET Depth & Point 7.17 Hand DAGT Depth & Point 7.12 Method of the present invention Depth & Point 7.05

[0093] In Table 1, the input data Depth represents the depth image, Point represents the point cloud, and Depth&Point indicates the presence of both modalities. This embodiment uses the mean joint error (MPJPE) to measure the error between the predicted and actual values. Table 1 details a comparison of the present invention's 3D gesture reconstruction method based on image point cloud multimodal fusion and joint guidance, highlighting its performance on the mainstream indicator of mean joint error. Experiments show that the present invention's 3D gesture reconstruction method based on image point cloud multimodal fusion and joint guidance exhibits smaller experimental errors compared to other reconstruction methods, and its objective performance surpasses that of current mainstream gesture reconstruction methods.

[0094] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A gesture reconstruction method based on multimodal fusion of image point clouds and joint guidance, characterized in that, A gesture reconstruction model is constructed, and this model is used to reconstruct gestures from depth images. The reconstruction process includes the following steps:

101. Acquire hand depth images and their point cloud data. The image feature extraction network extracts local and global image features from the depth images, and the point cloud feature extraction network extracts global point cloud features from the point cloud data.

102. The global image features and global point cloud features are concatenated together, and a bias embedding vector is generated for each key point through an induced bias layer. The vectors of all key points constitute the global key point embedding.

103. Global keypoint embedding: The high-dimensional embedding is projected onto the three-dimensional space through linear transformation to obtain the initial three-dimensional joint coordinates, and local point cloud features are extracted from the point cloud data based on the three-dimensional joints.

104. Update the global keypoint embedding through a graph convolutional network to obtain updated global features. Concatenate the global features with local image features and local point cloud features to obtain image joint features and point cloud joint features.

105. Based on the attention mechanism, the joint features of the image and the joint features of the point cloud are weighted respectively, and the weighted features are fed into the graph convolutional network to make full use of the topological information for further feature fusion and updating.

106. Predict 3D joint coordinates using the image joint features and point cloud joint features obtained after updating the graph convolutional network. Determine whether the iteration limit has been reached. If not, update the local features of the point cloud based on the predicted 3D joints and return to step 104. Otherwise, use the currently predicted 3D joint coordinates as the reconstruction result.

2. The gesture reconstruction method based on image point cloud multimodal fusion and joint guidance according to claim 1, characterized in that, The process by which an image feature extraction network extracts local and global image features from a depth image includes: The depth image is processed by an autoencoder network to obtain a two-dimensional local feature map F. 2d ∈R C×H×W Where C, H, and W are two-dimensional local feature maps F, respectively. 2d The number of channels, height, and width; Two-dimensional local feature map F 2d The input is fed into a convolutional residual network to generate a heatmap for each key point. The two-dimensional UV coordinates of the joints are estimated based on normalized heatmaps, and local image features are obtained by aggregating joint image features from local feature maps based on the heatmaps. Where J is the number of hand joints in the depth image, and d 2d Image feature dimensions corresponding to each joint; Two-dimensional local feature map F 2d A global image feature is output through global pooling.

3. The gesture reconstruction method based on multimodal fusion of image point clouds and joint guidance according to claim 2, characterized in that, The process by which a point cloud feature extraction network extracts local and global point cloud features from point cloud data includes: The three-dimensional local geometric features F are extracted from the point cloud data through point set convolutional layers. 3d ∈R N×d Where N represents the number of points in the point cloud, and d represents the feature dimension of the point cloud corresponding to each point; Three-dimensional local geometric features F 3d ∈R N×d After passing through a one-dimensional convolutional layer and a max pooling layer, global point cloud features are obtained. The two-dimensional local feature map F 2d The point cloud feature F is then projected onto the feature space of the 3D local geometric features and then stitched together with the 3D local geometric features to generate the fused point cloud feature F. fuse ; Using the initialized 3D joint coordinates as the starting position, the point cloud feature F is further guided. fuse Aggregate the data to generate the final local point cloud features. Where d 3d This represents the feature dimension of the point cloud corresponding to each joint.

4. A gesture reconstruction method based on multimodal fusion of image point clouds and joint guidance according to claim 1 or 3, characterized in that, The process of updating the local features of the point cloud based on the 3D joints includes: for each joint, using the 3D coordinates of the joint as the center, the ball query algorithm is used to find K neighboring points within a radius r around the position in the point cloud data, and the geometric features of these points are used as the local features of the point cloud of the joint.

5. The gesture reconstruction method based on image point cloud multimodal fusion and joint guidance according to claim 1, characterized in that, The bias embedding vector for each joint is generated by an induced bias layer. The process includes: concatenating global image features and global point cloud features, then copying them J times to obtain J feature maps. Each feature map is then fed into a one-dimensional convolutional layer, and a learnable position embedding is added to each feature map for each joint. The learnable position embedding provides an independent bias for each joint to generate a bias embedding vector.

6. The gesture reconstruction method based on multimodal fusion of image point clouds and joint guidance according to claim 1, characterized in that, The process of weighting image joint features and point cloud joint features based on the attention mechanism includes: By mapping image joint features and point cloud joint features through two linear modules respectively, query vectors, key vectors and value vectors of image joint features and point cloud joint features are obtained; The query vectors of image joint features and point cloud joint features are concatenated together, the key vectors are concatenated together, and the value vectors are concatenated together. The concatenated query vector, key vector, and value vector are input into the attention module to obtain attention weights. These attention weights are then used to weight the image joint features and point cloud joint features.

7. The gesture reconstruction method based on multimodal fusion of image point clouds and joint guidance according to claim 1, characterized in that, The loss function of the gesture reconstruction model during training is expressed as: Among them, L total The total loss of the gesture reconstruction model is represented by α, β, and γ, which are the weight coefficients of the loss function. uv The loss function for the image feature extraction network; init The loss function for the point cloud feature extraction network; Let be the loss function at the t-th feature iteration, where T is the number of feature iterations.

8. The gesture reconstruction method based on image point cloud multimodal fusion and joint guidance according to claim 7, characterized in that, The loss function of an image feature extraction network is expressed as: Where J is the number of hand joints in the depth image; L(x) is the standard Smooth L1 loss function; (U' j V j ') represents the UV coordinates of the j-th hand joint calculated using a normalized heatmap. (U j V j ) represents the actual UV coordinates of the j-th joint.

9. The gesture reconstruction method based on multimodal fusion of image point clouds and joint guidance according to claim 7, characterized in that, The loss function of the point cloud feature extraction network is expressed as: Where J is the number of hand joints in the depth image; Let J be the 3D joint position coordinates of the j-th joint predicted by the network. Let be the true 3D coordinates of the j-th joint.

10. The gesture reconstruction method based on multimodal fusion of image point clouds and joint guidance according to claim 7, characterized in that, The loss function during feature iteration is expressed as: Among them, l s The loss function represents the loss function during a single feature iteration; J is the number of hand joints in the depth image; P j '(t) represents the predicted 3D joint coordinates of the j-th joint after the t-th iteration update. Let be the true 3D coordinates of the j-th joint.

Images