Multimodal 3D hand mesh reconstruction method based on diffusion model

Abstract

The present invention belongs to the technical field of human posture recognition, and discloses a multimodal three-dimensional hand mesh reconstruction method based on a diffusion model. The method collects depth maps and point cloud data through a feature extraction model, and extracts hand joint features based on the depth maps and point cloud data; an initialized standard Gaussian distribution noise is input into the diffusion model together with the extracted hand joint features for iterative denoising, and the hand joint positions are gradually restored from the noise through multiple iterations; and the three-dimensional hand mesh is reconstructed through a mesh reconstruction module based on the hand joint positions. Based on color images, the present invention realizes high-precision three-dimensional hand mesh reconstruction by fusing depth maps and three-dimensional point cloud data, and significantly improves the speed and accuracy of hand posture reconstruction.

Description

Technical Field

[0001] The invention belongs to the technical field of human posture recognition and relates to a multimodal three-dimensional hand mesh reconstruction method based on a diffusion model. Background Art

[0002] At present, the technical solutions for hand posture reconstruction mainly focus on deep learning-based regression methods, detection methods, and graph convolutional networks (GCN). The regression method directly predicts the position of the hand joints and has a fast calculation speed, but it is prone to inaccuracy when dealing with complex gestures (such as finger crossing and self-occlusion); the detection method usually locates the joints step by step. Although it can obtain better spatial distribution information, it is difficult to cope with the interference caused by different perspectives and occlusions. In recent years, graph convolutional networks have been used in hand mesh reconstruction. They can capture the spatial relationship between joints, but they usually require a lot of computing resources and have poor adaptability to noise and unseen samples.

[0003] Some patents (such as diffusion models for 3D human pose estimation and hand pose reconstruction based on point clouds) have also proposed methods for fusing multimodal data, but most of them compromise efficiency or accuracy. For example, some solutions focus on generating mesh models from a single depth map or point cloud, and fail to fully utilize the advantages of multimodal data fusion, and thus perform poorly under occlusion. Current solutions generally lack the ability to accurately locate hand joints and handle noise, and have significant accuracy deficiencies in detail reconstruction and complex gesture scenes.

[0004] Current hand pose reconstruction techniques suffer from significant shortcomings in several aspects.

[0005] (1) The accuracy of deep learning regression methods decreases when dealing with complex gestures (such as crossed fingers, palm occlusion, etc.). This is mainly because their direct coordinate prediction method lacks modeling of the dependencies between joint points, which leads to inaccurate estimates in complex postures.

[0006] (2) Although the detection method can locate the joint points step by step, it is unstable under different viewing angles, lighting changes or occlusions, and it is difficult to accurately capture the three-dimensional spatial structure of the hand.

[0007] (3) Although the graph convolutional network-based solution can model the relationship between joints, it has high computational complexity and high requirements on hardware resources. It is not robust enough when dealing with noise and unknown gestures and cannot adapt well to dynamic or high-noise scenarios.

[0008] (4) These shortcomings are mainly due to the high dependence of existing methods on joint point positioning, sensitivity to noise, and relatively single assumptions in spatial relationship modeling, making it difficult to maintain high-precision hand posture reconstruction in complex and changing environments. To address these problems, the diffusion model can be introduced as a key technology to gradually generate and refine the coordinates of hand joint points through a step-by-step denoising process, so that the model can gradually optimize the posture estimation in complex environments with high uncertainty. At the same time, the diffusion model can better restore the details and dependencies between joint points by modeling the relationship between multimodal data (such as depth maps and point clouds), thereby greatly improving the reconstruction accuracy and robustness of hand posture. Summary of the invention

[0009] The purpose of the present invention is to propose a multimodal three-dimensional hand mesh reconstruction method based on a diffusion model. By introducing the diffusion model and based on a single color image, high-precision three-dimensional hand mesh reconstruction is achieved by fusing depth maps and three-dimensional point cloud data, which significantly improves the speed and accuracy of hand posture reconstruction.

[0010] The present invention is implemented by the following technical solution: a multimodal three-dimensional hand mesh reconstruction method based on a diffusion model, the steps are as follows:

[0011] S1: The feature extraction model collects depth maps and point cloud data, and extracts hand joint features based on the depth maps and point cloud data;

[0012] S2: An initialized standard Gaussian distribution noise is input into the diffusion model together with the extracted hand joint features for iterative denoising, and the hand joint positions are gradually restored from the noise through multiple iterations;

[0013] S3: Reconstruct the 3D hand mesh based on the hand joint positions through the mesh reconstruction module.

[0014] Further preferably, the feature extraction model is formed by connecting a first encoder, a local feature encoder, and a joint feature extractor in sequence.

[0015] Further preferably, the first encoder extracts a depth map and three-dimensional point cloud data from the color image.

[0016] Further preferably, the local feature encoder includes an upsampling layer, a ConvNeXt-based encoder and a PointNet++-based encoder. The features output by the first encoder are processed by the upsampling layer and then enter the ConvNeXt-based encoder and the PointNet++-based encoder for feature encoding to obtain two-dimensional features and three-dimensional features.

[0017] Further preferably, the ConvNeXt-based encoder extracts two-dimensional local features and two-dimensional global features from the depth map.

[0018] Further preferably, an encoder based on PointNet++ extracts 3D local features and 3D global features from 3D point cloud data.

[0019] Further preferably, the joint feature extractor is composed of a self-attention layer, a multi-layer perceptron, a convolutional layer, a multi-head self-attention layer, a bias induction layer and an upsampling layer connected in sequence, and the joint feature extractor extracts joint conditions based on two-dimensional features and three-dimensional features; and generates an independent and learnable bias embedding for each joint through the bias induction layer.

[0020] Further preferably, the diffusion model includes a sampling layer, a parallel denoising module structure and a refinement layer, the parallel denoising module structure includes two parallel denoising branches, each branch includes a plurality of denoising modules connected in series, and each denoising module is composed of self-attention, cross-attention, convolution layer, activation function, multi-layer perceptron and pooling layer connected in sequence. The features acquired by the sampling layer enter the parallel denoising module structure for processing, the output of the parallel denoising module structure enters the refinement layer, and the refinement layer outputs the hand joint position.

[0021] Further preferably, the input of the diffusion model is a multimodal feature, including the joint coordinate distribution , Joint conditions , 2D local features, 3D local features, and time step.

[0022] Further preferably, the grid reconstruction module consists of a second encoder, an upsampling layer, a third encoder and a regressor.

[0023] The difference between the present invention and the prior art is mainly reflected in the fusion application of multimodal data and the introduction of diffusion model to improve the accuracy and stability of hand gesture recognition. The prior art usually relies on a single depth map or other conditions for hand gesture recognition, which is limited by complex gestures and self-occlusion, and is prone to problems such as reduced recognition accuracy or inability to track stably. In contrast, the present invention simultaneously collects depth maps and point cloud data, uses diffusion models to gradually denoise and generate accurate hand joint coordinates, and has higher robustness and anti-interference capabilities. The layer-by-layer denoising process of the diffusion model effectively solves the uncertainty caused by noise, allowing the system to maintain high accuracy in adverse environments such as complex gestures, finger crossing and low light. In addition, the present invention combines the generated three-dimensional joint point data with the mesh reconstruction technology to form a complete hand mesh structure, further enhancing the authenticity and detail performance of the hand model. The advantage of this technical solution is that it can accurately reconstruct hand gestures in complex dynamic environments, achieve real-time and natural interactive experience, and provide more stable and delicate hand gesture capture effects in applications such as VR / AR, medical rehabilitation, and intelligent robots.

[0024] The present invention can be applied to product fields that require high-precision hand tracking and recognition, such as augmented reality (AR), virtual reality (VR), human-computer interaction systems, intelligent robots, and medical rehabilitation equipment. In these scenarios, the system needs to accurately capture and reconstruct the user's hand posture in real time to achieve natural and smooth interaction. For example, in VR applications, accurate hand reconstruction allows users to control virtual objects through natural gestures; in medical rehabilitation, precise hand posture tracking can help evaluate and guide the patient's rehabilitation training process. The present invention solves the problem of insufficient reconstruction accuracy in complex hand movements in current technologies, and brings new possibilities for application scenarios that require fine hand capture. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flow chart of the present invention.

[0026] Figure 2 Schematic diagram of the local feature encoder structure.

[0027] Figure 3 Schematic diagram of the joint feature extractor structure.

[0028] Figure 4 Schematic diagram of the diffusion model. DETAILED DESCRIPTION

[0029] The present invention will be further described in detail below with reference to the accompanying drawings.

[0030] like Figure 1 As shown in FIG. 1 , the multimodal 3D hand mesh reconstruction method based on the diffusion model has the following steps:

[0031] S1: The feature extraction model collects depth maps and point cloud data, and extracts hand joint features based on the depth maps and point cloud data;

[0032] S2: An initialized standard Gaussian distribution noise is input into the diffusion model together with the extracted hand joint features for iterative denoising, and the hand joint positions are gradually restored from the noise through multiple iterations;

[0033] S3: Reconstruct the 3D hand mesh based on the hand joint positions through the mesh reconstruction module.

[0034] like Figure 1 As shown in FIG, the feature extraction model is composed of a first encoder, a local feature encoder, and a joint feature extractor connected in sequence. The feature extraction model takes a single color image (RGB image) as input data, and first extracts a depth map from it by the first encoder. and 3D point cloud data , where R represents the dimension, H and W are the height and width of the depth map, respectively. is the number of 3D points. For efficient processing, the depth map is preprocessed to a fixed size of 128×128, and 1024 points are randomly sampled in the 3D point cloud to normalize the data volume. The first encoder can adopt a Transformer-based encoder.

[0035] Then, the depth map and 3D point cloud data are input into the local feature encoder. The local feature encoder makes a detailed analysis of these standardized data and extracts the local features and global features contained therein. Specifically, the local feature encoder extracts 2D features from the depth map and 3D features from the 3D point cloud data to ensure that as much spatial and texture information as possible can be used in the subsequent hand joint position recovery process to improve the final reconstruction accuracy.

[0036] The local feature encoder is Figure 2 As shown, it includes an upsampling layer, an encoder based on ConvNeXt, and an encoder based on PointNet++. The features output by the first encoder are processed by the upsampling layer, and then enter the encoder based on ConvNeXt and the encoder based on PointNet++ for feature encoding to obtain two-dimensional features and three-dimensional features.

[0037] ConvNeXt is an architecture that strengthens convolution operations. By introducing technologies such as feature aggregation and channel attention mechanism, the ConvNeXt-based encoder can effectively capture the details in the depth map and generate high-quality two-dimensional local features. and two-dimensional global features. This feature not only contains pixel-level information, but also incorporates the overall structure of the image, thus providing a more global feature expression. In addition, the ConvNeXt-based encoder uses stacked convolutional layers and channel attention mechanisms to enable the local feature encoder to focus on key spatial information and high-frequency details in the image.

[0038] PointNet++ is known for its efficient 3D point cloud data processing capabilities. It gradually aggregates information in 3D point cloud data through layered sampling and local aggregation operations to extract 3D features with semantic information. This method can not only capture the global pattern of spatial distribution in point clouds, but also generate fine 3D local features through the encoder based on PointNet++. And the three-dimensional global features that contain overall information. The encoder based on PointNet++ has stronger robustness in understanding the three-dimensional spatial structure through multi-level feature aggregation, especially in the face of local noise or point cloud density changes, it can still maintain high-precision feature extraction effects.

[0039] This dual-channel feature extraction strategy enables the local feature encoder to capture the 2D features of the depth map and effectively encode the 3D features in the 3D point cloud data to form a complete feature space. In the subsequent joint position recovery stage, these rich features will be further combined and processed, providing important clues about the position and posture of the hand joints, helping the model to more accurately reconstruct the 3D structure of the hand.

[0040] After completing the local feature encoding, the extracted two-dimensional features and three-dimensional features are input into the joint feature extractor to further mine the feature information related to the hand joint position. Figure 3 As shown in the figure, the joint feature extractor consists of a self-attention layer, a multi-layer perceptron, a convolutional layer, a multi-head self-attention layer, a bias-induced layer, and an upsampling layer connected in sequence. The joint feature extractor introduces a bias-induced layer (BIL) in this process. Its role is to generate an independent and learnable bias embedding for each joint during the feature extraction process, so that the joint feature extractor can more accurately locate the coordinates of each joint in a complex space. Specifically, given the input feature (where N is the number of joints and D is the feature dimension of each joint), the bias induction layer generates a unique bias vector for each joint position , these bias vectors are learned and optimized through model training. The output of the bias layer is calculated by the following formula:

[0041] ;

[0042] Among them, B represents the bias matrix, which is the bias vector The feature vector of each joint is combined with unique bias information to give each joint a unique positioning in the feature space, making the joint feature extractor more discriminative and accurate at different joint positions.

[0043] The core of the bias induction layer is to give the joint feature extractor stronger expressive power in the feature space by introducing joint bias, so that each joint can have a personalized feature offset, thereby providing an independent spatial identifier in the internal feature representation of the joint feature extractor. This is equivalent to generating a specific "marker" for each joint position, so that the position of each joint can be more clearly identified and adjusted during the iterative denoising process. As the training process deepens, these biases can be gradually optimized to adapt to the geometric characteristics and movement patterns of different joints.

[0044] This design is crucial for accurately restoring the position of each joint, especially in the complex geometry of the hand. The positioning embedding of different joints can help the joint feature extractor distinguish and accurately adjust the three-dimensional coordinates of each joint to reduce position deviation. At the same time, through the guidance of the bias, the joint feature extractor can better understand the relative position relationship between joints during the feature learning process, thereby improving the overall reconstruction accuracy.

[0045] like Figure 4 As shown in the figure, the diffusion model includes a sampling layer, a parallel denoising module structure and a refinement layer. The parallel denoising module structure includes two parallel denoising branches, each of which includes multiple denoising modules connected in series. Each denoising module consists of self-attention, cross-attention, convolution layer, activation function, multi-layer perceptron and pooling layer connected in sequence. The features obtained by the sampling layer enter the parallel denoising module structure for processing, and the output of the parallel denoising module structure enters the refinement layer, which outputs the hand joint position.

[0046] The specific process of step S2 in this embodiment is as follows:

[0047] In the input stage of the diffusion model, multimodal feature input is received, including the joint coordinate distribution (representing the three-dimensional coordinates of 21 joints), joint conditions , 2D local features , 3D local features , and the time step , The number of feature channels representing the joint conditions. These inputs are combined into the diffusion model in different ways, gradually guiding the diffusion model to gradually reconstruct the precise hand joint positions from random noise in the iterative denoising process.

[0048] The multimodal feature input is passed to the sampling layer, where K-nearest neighbors sampling is used to distribute the joint coordinates. Select 2D local features and 3D local features in the neighborhood of . Specifically, for each joint coordinate (in ), find the K nearest points in its local neighborhood and the corresponding local features :

[0049] ;

[0050] This neighbor sampling provides a set of local features around each joint position. ,These local features will be used as key reference information for reconstruction in the subsequent ,denoising iterations.

[0051] Next, in order to ensure that the diffusion model can correctly identify the noise characteristics of the joints at each iteration, the diffusion model introduces a time step control mechanism. For example, by adding a sinusoidal position embedding and joint index embedding To achieve this:

[0052] ;

[0053] This joint indicator vector The encoding information of time steps and joint indices is included and added as additional input to the diffusion model to help the model better understand the characteristic changes of joints at different denoising stages. These embeddings also allow the diffusion model to dynamically adjust the prediction of joint positions at each step of denoising, further improving accuracy.

[0054] Then, the local features, joint conditions, joint indication vectors and time steps are embedded into the convolution layer, which can effectively simulate the motion relationship between joints and output the corresponding joint features. , then, the joint features are input into the activation function, which is specifically expressed as follows:

[0055] ;

[0056] in, Represents the activation features of the convolution layer output, representing the feature matrix generated by the joint features after the convolution operation, are trainable weights, is the channel motion correspondence matrix between joints. The diffusion model uses convolutional networks and activation functions to enhance joint conditions, and then combines them with local features and joint indicators to capture the kinematic relationship between hand joints. By aggregating joint conditions and local features, the diffusion model can more accurately predict joint positions and understand the three-dimensional posture of the hand.

[0057] After that, the features and time step embedding output by the convolutional layer and the local features are connected to generate joint features, which are then input into the multi-layer perceptron (MLP) to generate joint local features and some feature priors. The specific representation is as follows:

[0058] ;

[0059] in, is the updated feature representation, and Respectively The joint The coordinates and local features of local neighbor points, Represents each joint output from the convolutional layer After being extracted by MLP, it provides a deeper feature representation for the next denoising operation. PE is a sinusoidal position embedding function.

[0060] In the experiment, the feature extraction capability of a single denoising module is limited and it is difficult to fully handle complex denoising tasks. Therefore, a parallel denoising module structure is adopted to replicate a single denoising module. A maximum pooling layer is introduced between every two denoising modules to gradually update the joint conditions. Specifically, the role of the maximum pooling layer is to aggregate local features so that more representative joint feature representations can be used in subsequent denoising modules. The input of the denoising module defines the pooling operation as:

[0061] ;

[0062] in, Indicates The input of the denoising module, Indicates The feature output of the secondary denoising module, MaxPool represents the maximum pooling operation, obtained by maximum pooling It can provide the next denoising module with more local joint embedding features, thereby gradually enriching the detail capture capability of the diffusion model.

[0063] In addition, cross fusion is performed between every two denoising modules, combining the output of each denoising module with the output of its parallel denoising module to form an enhanced feature representation. This operation helps to integrate the features between different denoising modules to obtain more semantic information and local detail features. Suppose the output of the denoising module is and cross-fusion features , the following feature enhancements can be achieved through cross-connection:

[0064] ;

[0065] in, In order to cross-fuse features, this structure enables the model to obtain richer feature representations in continuous iterations.

[0066] Next, the final outputs of the two parallel denoising modules are cascaded to align features for subsequent processing. Suppose the final outputs of the two parallel denoising branches are and , then the cascade operation can be expressed as:

[0067] ;

[0068] in, is a joint feature;

[0069] Subsequently, this joint feature It will be input into a refinement layer to further optimize the estimation of joint positions. This refinement layer includes multiple linear transformation layers and residual connections to adjust the coordinate positions of the joints so that the diffusion model can gradually approach the true joint positions in each iteration. and bias , the output joint position adjustment can be expressed as:

[0070] ;

[0071] in For the joint coordinate distribution, adding residual connections can help the model maintain a continuous approximation effect during the estimation process.

[0072] Finally, the joint coordinates output by the refinement layer are The diffusion model is input again as the input for the next denoising stage. This step-by-step iterative optimization strategy helps the model refine at each step based on the previous round of estimates, thereby gradually converging to the precise joint coordinates. :

[0073] ;

[0074] This gradual iterative mechanism enables the model to effectively approximate the actual joint positions in multiple rounds of denoising iterations, thereby improving the overall joint prediction accuracy.

[0075] The grid reconstruction module in step S3 is composed of a second encoder, an upsampling layer, a third encoder and a regressor. The second encoder and the third encoder can adopt encoders based on ResNet50.

[0076] First, the second encoder is used to encode the input color image and generate output features These features contain rich hand appearance information such as color and texture, providing initial feature representation for subsequent mesh reconstruction.

[0077] Then, the output features of the second encoder are Processed through the upsampling layer to obtain upsampled features ,in represents the number of feature points, Indicates the number of channels. Upsampling can improve the resolution of features and provide more details for subsequent refinement processing.

[0078] Next, the hand joint features predicted by the diffusion model are With upsampling features Specifically, these features are input into the third encoder and encoded into a one-dimensional hand joint token. By fusing feature information from different sources, the model's ability to perceive the hand shape can be effectively improved.

[0079] In addition, the regressor receives the hand joint tokens output by the third encoder and gradually converts them into predictions of the hand mesh. The regressor gradually increases the number of tokens through multiple stages of upsampling operations, thereby improving the detail resolution of the hand mesh. Let the feature representation of the sth stage be , the feature of the s+1th stage is expressed as , then the staged upsampling operation Upsample can be expressed as:

[0080] ;

[0081] During this process, the regressor will gradually refine the hand mesh structure, making it gradually approach the real hand mesh.

[0082] At each refinement stage of the regressor, the number of input token channels is first adjusted through the dimensionality reduction layer so that its dimension adapts to the requirements of the subsequent processing module. This step ensures that the features can be effectively processed by the Transformer module, thereby obtaining accurate refinement results. Next, the input passes through the Transformer module to capture the correlation between tokens through the self-attention mechanism. Through this mechanism, the Transformer module can deeply explore the interaction between tokens and generate more expressive feature representations. Specifically, the processing of the Transformer module can be expressed as:

[0083] ;

[0084] in, represents the input features, Represents the output features. The features generated after the Transformer module provide rich contextual information.

[0085] With the help of the self-attention mechanism, the token features are refined and then input into the upsampling layer. The main function of the upsampling layer is to gradually increase the number of tokens, so that the resolution of the hand mesh is gradually improved. This layer-by-layer refinement structure allows each stage of upsampling and processing to further improve the refinement of the mesh prediction. Ultimately, after multiple refinement stages, the regressor is able to output a high-resolution, refined hand mesh, providing higher accuracy for the reconstruction of the hand posture.

[0086] During the experiment, we also found that increasing the number of layers of the Transformer module can improve the accuracy of mesh reconstruction to a certain extent, but when the number of layers exceeds a certain critical value, the performance improvement tends to saturate. Therefore, the setting of the number of layers needs to balance performance improvement and computing resources to achieve the best effect.

[0087] The present invention can be widely used in many fields such as virtual reality (VR) interaction, medical rehabilitation, intelligent robot control and human-computer interaction, greatly improving the system's capture accuracy and real-time response capabilities for hand gestures. In VR, this technology allows users to interact with virtual objects smoothly through natural gestures, increasing the sense of immersion; in the field of medical rehabilitation, the system can track and analyze the patient's hand movements, assisting doctors in rehabilitation training and recovery monitoring; in intelligent robots and home control, gesture recognition enables robots to accurately understand the user's intentions and achieve safer and more natural human-computer interaction.

[0088] Taking medical rehabilitation as an example, the present invention can play an important role in the patient's hand activity tracking and rehabilitation training. During rehabilitation training, patients can operate virtual objects through natural gestures, such as grabbing, moving or rotating, and the system can accurately capture and record the changes of every joint and posture of the hand in real time. These precise hand data will generate detailed motion analysis reports, providing doctors with comprehensive motion data to help them evaluate the patient's rehabilitation progress. Doctors can use this data to analyze the patient's movement quality, amplitude and flexibility, and identify possible problems that the patient may have during the rehabilitation process, such as unsmooth movements, insufficient amplitude or insufficient speed.

[0089] At the same time, the present invention can also provide patients with real-time visual feedback, allowing them to adjust their movements immediately during the operation and gradually standardize their postures. In the virtual environment, the system will present a model of the patient's hand, showing the degree of match with the target movement, and the patient can adjust the movement by observing his or her virtual hand. This instant feedback not only improves the efficiency of training, but also allows patients to complete rehabilitation exercises independently in a home environment, reducing dependence on professional equipment and medical staff. Doctors can check the patient's practice at any time through the remote monitoring system and adjust the rehabilitation plan according to the progress, thereby achieving more personalized treatment. This technology makes rehabilitation training more scientific and convenient, helping patients recover hand function faster, while providing doctors with more intuitive data support, realizing the organic combination of precision medicine and remote rehabilitation.

[0090] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0091] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A multimodal 3D hand mesh reconstruction method based on a diffusion model, characterized in that: Here are the steps: S1: The feature extraction model collects depth maps and point cloud data, and extracts hand joint features based on the depth maps and point cloud data; S2: input an initialized standard Gaussian distributed noise together with the extracted hand joint features into a diffusion model for iterative denoising, and gradually recover the hand joint positions from the noise through multiple iterations; the diffusion model includes a sampling layer, a parallel denoising module structure and a refinement layer, the parallel denoising module structure includes two parallel denoising branches, each branch includes a plurality of denoising modules connected in series, and each denoising module is composed of self-attention, cross-attention, convolution layer, activation function, multi-layer perceptron and pooling layer connected in sequence; the features obtained by the sampling layer enter the parallel denoising module structure for processing, the output of the parallel denoising module structure enters the refinement layer, and the refinement layer outputs the hand joint positions; In the input stage of the diffusion model, multimodal feature input is received, including the joint coordinate distribution , Joint conditions , 2D local features , 3D local features , and the time step , The number of feature channels representing joint conditions; The multimodal feature input is passed to the sampling layer, where K-nearest neighbor sampling is used to distribute the joint coordinates. Select two-dimensional local features and three-dimensional local features in the neighborhood of , find the K nearest points in its local neighborhood and the corresponding local features : ； This neighbor sampling provides a set of local features around each joint position. ; Next, in order to ensure that the diffusion model can correctly identify the noise characteristics of the joints in each iteration, the diffusion model introduces a time step control mechanism by adding a sinusoidal position embedding and joint index embedding To achieve this: ； Joint indicator vector It contains the encoding information of time steps and joint indices and is added to the diffusion model as an additional input; Then, the local features, joint conditions, joint indication vectors, and time step embeddings are input to the convolutional layer, which outputs the corresponding joint features. , then, the joint features are input into the activation function, which is specifically expressed as follows: ； in, Represents the activation features of the convolution layer output, representing the feature matrix generated by the joint features after the convolution operation, are trainable weights, is the channel motion correspondence matrix between joints; After that, the features and time step embedding output by the convolutional layer and the local features are connected to generate joint features, which are then input into the multi-layer perceptron MLP to generate joint local features, which are expressed as follows: ； in, is the updated feature representation, and Respectively The joint The coordinates and local features of local neighbor points, Represents each joint output from the convolutional layer The updated features are extracted by MLP, which provides a deeper feature representation for the next denoising operation. PE is the sinusoidal position embedding function. Parallel denoising module structure, a single denoising module is replicated A maximum pooling layer is introduced between every two denoising modules to gradually update the joint conditions; The input of the denoising module defines the pooling operation as: ； in, Indicates The input of the denoising module, Indicates The feature output of the secondary denoising module, MaxPool represents the maximum pooling operation; Between every two denoising modules, cross fusion is performed to combine the output of each denoising module with the output of its parallel denoising module to form an enhanced feature representation. Suppose the output of the denoising module is and cross-fusion features , the following feature enhancements are achieved through cross-connection: ； in, It is a cross-fusion feature; Next, the final outputs of the two parallel denoising modules are cascaded to align features for subsequent processing; the final outputs of the two parallel denoising branches are assumed to be and , then the cascade operation can be expressed as: ； in, is a joint feature; Subsequently, this joint feature This is fed into a refinement layer to further refine the estimate of the joint positions; Finally, the joint coordinates output by the refinement layer are The diffusion model is input again as the input of the next denoising stage, thus gradually converging to the precise joint coordinates. : ； S3: Reconstruct the 3D hand mesh based on the hand joint positions through the mesh reconstruction module.

2. The multimodal three-dimensional hand mesh reconstruction method based on diffusion model according to claim 1, characterized in that: The feature extraction model is formed by sequentially connecting a first encoder, a local feature encoder, and a joint feature extractor.

3. The multimodal three-dimensional hand mesh reconstruction method based on diffusion model according to claim 2, characterized in that: The first encoder extracts a depth map and three-dimensional point cloud data from the color image.

4. The multimodal three-dimensional hand mesh reconstruction method based on diffusion model according to claim 2, characterized in that: The local feature encoder includes an upsampling layer, a ConvNeXt-based encoder and a PointNet++-based encoder. The features output by the first encoder are processed by the upsampling layer and then enter the ConvNeXt-based encoder and the PointNet++-based encoder for feature encoding to obtain two-dimensional features and three-dimensional features.

5. The multimodal three-dimensional hand mesh reconstruction method based on diffusion model according to claim 4, characterized in that: The ConvNeXt-based encoder extracts 2D local features and 2D global features from the depth map.

6. The multimodal three-dimensional hand mesh reconstruction method based on diffusion model according to claim 4, characterized in that: The PointNet++-based encoder extracts 3D local features and 3D global features from 3D point cloud data.

7. The multimodal three-dimensional hand mesh reconstruction method based on diffusion model according to claim 4, characterized in that: The joint feature extractor is composed of a self-attention layer, a multi-layer perceptron, a convolutional layer, a multi-head self-attention layer, a bias induction layer and an upsampling layer connected in sequence, and the joint feature extractor extracts joint conditions according to two-dimensional features and three-dimensional features; An independent and learnable bias embedding is generated for each joint through a bias inducing layer.

8. The multimodal three-dimensional hand mesh reconstruction method based on diffusion model according to claim 1, characterized in that: The input of the diffusion model is multimodal features, including the joint coordinate distribution , Joint conditions , 2D local features, 3D local features, and time step.

9. The multimodal three-dimensional hand mesh reconstruction method based on diffusion model according to claim 1, characterized in that: The grid reconstruction module consists of a second encoder, an upsampling layer, a third encoder and a regressor.

Images