Multi-modal three-dimensional hand mesh reconstruction method and system based on topology-aware transformer

Abstract

The application discloses a kind of multi-modal three-dimensional hand grid reconstruction method and system based on topology perception Transformer, belong to computer vision and intelligent interaction technical field.The application introduces the topology perception hierarchical coding denoising network containing anatomical and topological constraint in the process of three-dimensional hand posture and grid reconstruction;After real hand 3D joint coordinates are projected and forwardly noisy by perspective projection and are carried out feature fusion, it is handled by the topology perception hierarchical coding denoising network formed by hand hierarchical coding module, hierarchical feature aggregation mechanism and topology perception Transformer module, obtains joint feature containing rich structure information, finally realizes the two-stage hand grid vertex construction from rough to fine by joint guided grid reconstruction module.The application significantly improves the precision and robustness of three-dimensional hand posture and grid reconstruction.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and intelligent interaction technology, and in particular to a multimodal 3D hand mesh reconstruction method and system based on topology-aware Transformer. Background Technology

[0002] 3D hand pose reconstruction is a key technology for the development of current technologies in fields such as human-computer interaction, robotics and teleoperation, content creation and entertainment, and medical and health care.

[0003] Current technical solutions mainly focus on 3D hand pose estimation and mesh reconstruction based on deep learning. Existing methods can be broadly divided into two categories: one is optimization methods based on parametric modules, such as structured hand modeling methods represented by the MANO module, which achieve joint estimation of pose and shape by regressing low-dimensional parameters and imposing geometric or physical constraints; the other is end-to-end data-driven methods, including direct regression modules based on convolutional neural networks (CNN), Transformer, or graph neural networks (GNN). These methods typically rely on large amounts of labeled data to learn the mapping relationship from images to 3D joints and meshes, and introduce topological constraints, semantic attention, or hierarchical feature fusion into the network structure to enhance geometric consistency and structural rationality. In recent years, diffusion models have gradually become a new trend in generative modeling. They achieve refined 3D shape or pose reconstruction through a progressive denoising process, demonstrating excellent performance in terms of modeling uncertainty and generating details. However, existing diffusion methods often lack explicit modeling of the hand's topological structure and anatomical constraints, which can easily lead to geometric discontinuities or structural inconsistencies between poses.

[0004] Traditional parametric module-based methods (such as the MANO module) have certain advantages in terms of structural controllability and anatomical plausibility. However, due to the limited module dimensionality, their expressive power is insufficient to cover complex hand deformations and high-frequency geometric details. Furthermore, these methods typically rely on optimization or fitting processes, resulting in high computational costs and sensitivity to initialization and occlusion, making it difficult to achieve stable, high-precision reconstruction in real-world scenes. While end-to-end deep network methods improve the automation and generalization capabilities of estimation, most modules primarily focus on feature correlations, neglecting the inherent topological structure of the hand and the anatomical constraints between joints. This lack of structural priors in feature learning often leads to problems such as local discontinuities, global distortions, or self-intersections in the predicted joint positions and mesh shapes. Moreover, existing methods often rely on large-scale, finely labeled data during training, but real-world hand poses and occlusion variations are complex, and data distributions deviate significantly from the training set, resulting in insufficient robustness of modules across scenes or under extreme conditions.

[0005] Generative approaches based on diffusion models, which have emerged in recent years, have provided new ideas for 3D pose reconstruction, but they also have certain limitations. Diffusion models essentially rely on a progressive denoising generation process, resulting in long inference times, making it difficult to meet the latency-sensitive requirements of real-time interaction or robot control. More importantly, existing diffusion-based hand modeling methods mostly focus on shape generation or pose sampling, lacking explicit modeling of the hand's topology, physical constraints, and semantic consistency. While the generated results are generally realistic, they are prone to problems such as local geometric misalignment, structural inconsistencies, or finger misalignment. The root cause of these shortcomings is that the diffusion process mainly models noise in the feature space, failing to effectively introduce structural constraints or hierarchical priors, resulting in a lack of perception and constraint capabilities regarding the geometric continuity of the hand. Summary of the Invention

[0006] The purpose of this invention is to provide a multimodal 3D hand mesh reconstruction method and system based on topology-aware Transformer. This invention achieves 3D hand pose estimation and mesh reconstruction from 2D joints by constructing a topology-aware hierarchical coding denoising network and utilizing a joint-guided mesh reconstruction module, which significantly improves the accuracy of 3D hand pose estimation and the consistency of mesh reconstruction.

[0007] This invention is achieved through the following technical solution:

[0008] A multimodal 3D hand mesh reconstruction method based on topology-aware Transformer includes the following steps:

[0009] Step 1: Given the real 3D joint coordinates of the hand, perform perspective projection and forward noise addition to obtain the corresponding 2D joint coordinates and noisy joint sequence, and then perform cross-dimensional feature fusion to obtain the fused features;

[0010] Step 2: The fused features are output as joint features containing structural information through a topology-aware hierarchical coding denoising network. The topology-aware hierarchical coding denoising network includes a hand layered coding module, a hierarchical feature aggregation mechanism, and a topology-aware Transformer module. First, the hand layered coding module divides the hand joints into seven layers according to their anatomical structure, from the wrist to the tips of each finger. Then, the spatial topological relationship between joints is modeled using hand reference points learned by several topology-aware Transformer modules to obtain joint features. A hierarchical feature aggregation mechanism is set between adjacent topology-aware Transformer modules to construct hierarchical features through bottom-up information propagation.

[0011] Step 3: After average pooling of the joint features, the global features are obtained. The global features are input into the joint regression head to directly generate the 3D hand joint pose, and the global features are input into the joint-guided mesh reconstruction module to obtain the hand mesh vertices.

[0012] In a further preferred embodiment, the hand layered coding module organizes the hand joints into seven functional groups according to anatomical function: the wrist, the MCP joint, and the five fingers; each functional group is equipped with learnable embedding parameters.

[0013] Further optimization involves recording the mapping relationship between each joint and its child joints in the hierarchical feature aggregation mechanism based on the topological structure of the hand skeleton tree; for a parent joint with child joints, its own features are concatenated with the features of all direct child joints and then averaged; for leaf joints, their original features are used directly.

[0014] Further optimization reveals that the topology-aware Transformer module comprises a baseline topology construction (BTC), an adaptive offset learning network, and a structured relative position encoding (SRPE). One branch processes the input features in the adaptive offset learning network of the topology-aware Transformer, and then, together with the output features of the baseline topology construction (BTC), processes them in the structured relative position encoding to obtain the output features. Another branch processes the input features and hierarchical features in the normalization layer of the topology-aware Transformer module, and then calculates attention weights in the self-attention module to obtain the output features. The output features from the two branches are weighted and fused before entering the normalization layer for normalization, and finally, a multilayer perceptron (MLP) performs feature transformation to obtain the output features of the topology-aware Transformer module.

[0015] Further preferred, the baseline topology construction (BTC) parameterizes the basic topological structure of the hand into four learnable parameters: the relative angular distribution of the five fingers on the palm, the length ratio of each level of bones from proximal to distal, the special angular offset of the thumb, and the range constraint factor of the dynamic offset.

[0016] Further optimization involves using an adaptive offset learning network to learn the structural adjustment of the current gesture relative to a standard topology and applying it to reference point coordinates, ultimately outputting the 2D offset of each joint.

[0017] Further optimization shows that the topology-aware attention weights of the topology-aware Transformer module are obtained by summing the feature similarity attention weights, structured position attention weights, and hierarchical semantic attention weights and then activating them using the Softmax function.

[0018] This invention also provides a multimodal 3D hand mesh reconstruction system based on topology-aware Transformer, comprising:

[0019] The data preprocessing and feature fusion module is used to receive the real 3D joint coordinates of the hand, generate the corresponding 2D joint coordinates and noisy joint sequences through perspective projection and forward noise processing, respectively, and perform cross-dimensional feature fusion to output the fused features.

[0020] A topology-aware hierarchical coding denoising network module, connected to the data preprocessing and feature fusion module, is used to receive the fused features and output joint features containing structural information; this module further includes:

[0021] The hand layered coding unit is used to divide the hand joints into multiple layers from the wrist to the fingertips according to their anatomical structure, and equips each functional group with learnable embedding parameters.

[0022] Hierarchical feature aggregation units are set between adjacent topology-aware Transformer modules. Based on the topological relationship of the hand skeleton tree, hierarchical features are constructed through bottom-up information propagation.

[0023] At least one topology-aware Transformer module is used to model the spatial topological relationships between joints based on hand reference points. It integrates a baseline topology construction subunit, an adaptive offset learning network subunit, and a structured relative position encoding subunit to compute topology-aware attention weights.

[0024] A 3D reconstruction output module, connected to the topology-aware hierarchical coding denoising network module, is used to average pool the joint features into global features, and further includes:

[0025] The joint regression head unit is used to directly regress the global features to generate accurate 3D hand joint poses;

[0026] Joint-guided mesh reconstruction units are used to generate refined hand mesh vertices using the global features as input and a two-stage reconstruction strategy from coarse to fine.

[0027] The present invention also provides a non-volatile computer storage medium storing computer-executable instructions that execute the multimodal three-dimensional hand mesh reconstruction method.

[0028] The present invention also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the multimodal three-dimensional hand mesh reconstruction method.

[0029] The beneficial effects of this invention are:

[0030] By employing a topology-aware hierarchical coding denoising network, the model can output joint features containing rich structural information. Based on this, the joint regression head can directly generate accurate 3D hand joint poses, while the joint-guided mesh reconstruction module can generate highly detailed mesh vertices.

[0031] Traditional methods are prone to problems such as local discontinuities, global distortions, or self-intersections. This invention fundamentally avoids such errors by explicitly modeling hand topological constraints. For example, the hand layered encoding module groups joints according to anatomical function, enabling the model to understand the semantic roles of different joints; the baseline topology construction (BTC) and structured relative position encoding (SRPE) in the topology-aware Transformer module incorporate the physiological structure of the hand (such as finger angle distribution and bone length ratio) as strong priors into the network, ensuring that the generated hand model is anatomically sound.

[0032] This invention successfully combines the powerful generative capabilities of diffusion models, the global relational modeling advantages of Transformers, and the unique anatomical topological priors of the hand, significantly improving the accuracy and robustness of 3D hand pose and mesh reconstruction. Attached Figure Description

[0033] Figure 1 This is a flowchart of the method of the present invention;

[0034] Figure 2 Diagram of hand-based layered coding strategy;

[0035] Figure 3 This is a diagram of the architecture of the topology-aware Transformer module;

[0036] Figure 4 Flowchart for the joint-guided mesh reconstruction module. Detailed Implementation

[0037] The invention will now be explained in further detail with reference to the accompanying drawings.

[0038] like Figure 1 As shown, this embodiment provides a multimodal 3D hand mesh reconstruction method based on topology-aware Transformer, with the following steps:

[0039] Step 1: Provide the actual 3D joint coordinates of the hand (Contains 3D coordinates of 21 joints, where R represents dimension), for the actual 3D joint coordinates of the hand. The corresponding 2D joint coordinates are obtained by perspective projection using camera parameters. (Including 21 joint points in 2D coordinates), for the actual hand 3D joint coordinates A 3D noisy joint sequence is obtained by progressively adding Gaussian noise to a Markov chain. ; For the T-th segment of noise joint features, a spatial patching embedding layer is used to map the 2D joint coordinates and the 3D noise joint sequence to a unified high-dimensional feature space, thereby achieving cross-dimensional feature fusion and obtaining fused features.

[0040] Step 2: Input the fused features into a topology-aware hierarchical coding denoising network for processing, and output joint features containing structural information. The topology-aware hierarchical coding denoising network comprises a hand-layered coding module, a hierarchical feature aggregation mechanism, and a topology-aware Transformer module. First, the hand-layered coding module divides the hand joints into seven layers according to their anatomical structure, from the wrist to the tips of each finger. Then, it models the spatial topological relationships between joints using hand reference points learned by n topology-aware Transformer modules, thereby obtaining joint features. Furthermore, a hierarchical feature aggregation mechanism is set up between adjacent topology-aware Transformer modules to construct hierarchical features through bottom-up information propagation.

[0041] Step 3: Determine joint features Global features are obtained after average pooling. Global features are input into the joint regression head to directly generate accurate 3D hand joint poses. and global features The input joint-guided mesh reconstruction module produces a refined hand mesh vertices. .

[0042] like Figure 1 As shown, this invention includes a forward diffusion process during the training phase and a reverse denoising process during the inference phase, which together constitute a diffusion model. The forward diffusion process involves adding Gaussian noise to the real 3D joint coordinates of the hand. The reverse denoising process uses 2D joint coordinates as conditions, starts with random noise, and generates 3D hand pose and mesh through sampling by the diffusion model.

[0043] Specifically, topology-aware hierarchical coding denoising networks receive 2D joint coordinates. and 3D noise joint sequences The spatial patch embedding layer is used as input to achieve cross-dimensional feature fusion; the fused features are further fused with spatial location information after passing through the hand-partial encoding module. and time step information This forms the final input features. The input is fed into the topology-aware Transformer module, and after processing, the output features of the topology-aware Transformer module are obtained. . The hierarchical feature aggregation mechanism constructs hierarchical features through bottom-up information propagation. . Hierarchical features serve as input to the next topology-aware Transformer module. As auxiliary contextual information, these are collectively fed into subsequent topology-aware Transformer modules for deep feature learning. After processing by n topology-aware Transformer modules, the output is joint features containing structural information. .

[0044] like Figure 2 As shown, the hand-layered encoding module (layered encoding according to finger structure, strategy 1) organizes hand joints into seven functional groups based on anatomical function: wrist, MCP joint, and five finger functional groups (thumb, index finger, middle finger, ring finger, and little finger). Each functional group is equipped with learnable embedding parameters to encode the unique semantic information of that functional group. This design ensures that joints of the same type can share a structural semantic basis, thereby better reflecting the typological organization characteristics of hand joints by capturing common features among joints of the same type.

[0045] In the hierarchical feature aggregation mechanism, this invention records each joint based on the topological structure of the hand skeleton tree. with its sub-joint The mapping relationship is as follows: For a parent joint with child joints, its own features are concatenated with the features of all direct child joints and then average pooled; for leaf joints, their original features are used directly.

[0046] like Figure 3As shown, the topology-aware Transformer module uses the Transformer as its basic framework and designs a topology-aware attention mechanism to enhance structural modeling capabilities. This mechanism, based on traditional self-attention feature similarity calculation, further introduces learnable hand topological priors, enabling attention weights to simultaneously consider feature matching and anatomical constraints. The topology-aware attention mechanism shifts attention computation from an abstract feature space to a physically meaningful structured semantic space. The entire mechanism comprises three core components: baseline topology construction (BTC), adaptive offset learning network, and structured relative position encoding (SRPE). Specifically, one branch: input features... After being processed in the adaptive offset learning network of the topology-aware Transformer, the features are combined with the output features of the baseline topology construction (BTC) and then fed into the structured relative position encoding (SRPE); another branch: input features and hierarchical features After concatenation, the resulting concatenated features are processed through a normalization layer. Then, in the self-attention module, attention weights are calculated using the query, key, and value matrices, and the weighted attention values ​​are used to obtain the output features. The weighted output features from the two branches are then normalized in a normalization layer. Finally, a multilayer perceptron (MLP) performs feature transformation to obtain the output features of the topology-aware Transformer module. Through this design, the attention weights not only reflect feature similarity but also embody the spatial relationships based on the hand's topological structure.

[0047] The baseline topology construction (BTC) parameterizes the basic topology of the hand into four learnable parameters:

[0048] ;

[0049] in:

[0050] This indicates the relative angular distribution of the five fingers on the palm. These represent the relative angles of the five fingers on the palm.

[0051] This indicates the proportional relationship of bone length from proximal to distal. These represent the lengths of the metacarpophalangeal joint, proximal interphalangeal joint, distal interphalangeal joint, and fingertip, respectively.

[0052] This indicates a specific angular offset of the thumb.

[0053] This represents the range limitation factor for dynamic offset.

[0054] These parameters encode the hand's baseline configuration and physiological constraints in a learnable manner. Based on these topologies, hand reference point coordinates can be constructed. (J represents the number of joints), providing an anatomically sound initial position for each joint. Specifically, the distribution of the five metacarpophalangeal joints (MCPs) around the wrist joint is first determined based on angular parameters. For each subsequent joint of the finger, a direction vector from the MCPs to the wrist joint is calculated. Then normalize it. Then, along this direction, the positions of the proximal interphalangeal joint (PIP), distal interphalangeal joint (DIP), and fingertip are obtained by recursively deducing the proportion of bone length.

[0055] For the thumb, considering its unique anatomical structure, an additional rotational transformation is required. This is achieved by constructing a 2×2 rotation matrix to adjust the thumb's orientation vector. Then, the three interphalangeal joints of the thumb are recursively placed according to this adjusted orientation, ensuring the thumb's unique anatomical structure is correctly modeled. The coordinates of this reference point serve as the basis for topological constraints and will guide dynamic adjustments in subsequent steps.

[0056] To adapt to the dynamic changes of different gestures, this invention designs an adaptive offset learning network to learn the structural adjustment of the current gesture relative to the standard topology, and applies it to the reference point coordinates. This adaptive offset learning network receives input features and outputs a 2D offset for each joint. The sampling positions after applying constraints are:

[0057] ;

[0058] in, Sampling location, This represents the tangent function. This design balances structural prior constraints with modeling flexibility, ensuring that the module adheres to anatomical rationality while adapting to specific gesture variations.

[0059] Based on the previously obtained sampling positions The relative positional relationships between joints are calculated in the constructed structured semantic space and converted into attention biases to guide attention computation. Specifically, for sampling positions... For any two joints in the structure, their relative position vectors in the 2D structure space are first calculated. To enhance the sensitivity of the topology-aware Transformer module to small displacement changes, a logarithmic transformation is performed on the relative positions. This transformation amplifies the positional differences between proximal joints while compressing the positional differences between distant joints, allowing the module to focus more on fine-grained local structural relationships. Finally, the transformed relative positions are processed through a position encoding network and mapped to a multi-head attention bias. This enhances the dependency between adjacent joints in space.

[0060] To simultaneously incorporate semantic, geometric, and hierarchical information into topology-aware attention computation, we utilize... and hierarchical features The concatenated features participate in the computation, enabling the attention mechanism to simultaneously perceive local motion details and global structural dependencies, thus achieving a more structure-aware representation capability. Finally, the topology-aware attention computation is as follows:

[0061] ;

[0062] in, Topology-aware attention weights, feature similarity attention weights Responsible for capturing semantic correlations between joints; structured position attention weights Embedding the geometric constraints of the hand-topology into the attention computation gives stronger association weights to spatially adjacent joints; hierarchical semantic attention. This incorporates the parent-child dependency relationship from hierarchical features, ensuring that the attention allocation for each joint can perceive its hierarchical position in the kinematic chain. Here, Q represents the query matrix, and K represents the key matrix. The transpose of the key matrix representing hierarchical features. Indicates the matrix dimension.

[0063] After progressive feature abstraction and structural constraint propagation through n topology-aware Transformer modules, the output is a joint feature containing rich structural information. Then, the joint features Average pooling is performed to obtain compact global hand features. The global features are input into the joint regression head to directly generate accurate 3D hand joint poses. .

[0064] like Figure 4 As shown, the joint-guided mesh reconstruction module can generate a refined hand mesh. The module employs a two-stage strategy: first, a topologically consistent initial hand mesh is obtained through MANO parametric reconstruction; then, a joint-guided refinement module is used to accurately reconstruct geometric details. MANO is a parametric hand model that generates a corresponding 3D hand mesh, including both pose and shape aspects, by inputting the correct parameters.

[0065] Specifically, in the first stage, the joint features output by the topology-aware Transformer module are... Perform average pooling to obtain compact global features. The global features are input into dedicated pose and shape heads: the pose head predicts 48-dimensional joint angle parameters, including 3-dimensional global rotation and 45-dimensional finger joint angles; the shape head predicts 10-dimensional hand shape parameters. After inputting these parameters into the MANO layer, an initial hand mesh with 778 vertices is obtained. The mesh has the correct topology but lacks fine geometric details.

[0066] Due to the parametric constraints of the MANO model, the generated initial hand mesh often lacks detailed information. To address this, a joint-guided refinement module was designed in the second stage, utilizing the rich semantic joint features encoded by the topology-aware Transformer module. This effectively propagates to all 778 mesh vertices, thereby leveraging joint-level fine-grained information to guide vertex-level geometric refinement. Specifically, the joint-guided refinement module first refines the generated initial hand mesh... Mapping to joint features via a feature projection network The shared semantic space allows vertices to participate as query vectors in the subsequent dual-attention mechanism architecture computation. In this mechanism, the first layer, cross-attention, enables semantic propagation from joint features to vertex features, allowing each vertex to adaptively extract the most relevant guidance information from the joint features. The second layer, self-attention, specifically models the spatial correlation between vertices, ensuring the geometric continuity and structural consistency of the mesh during refinement. After this dual-attention processing, the original vertex coordinates are fused with the attention-enhanced semantic features to obtain an enhanced feature representation that integrates joint semantics and spatial constraints. Subsequently, the spatial offset vector of each vertex is predicted through network layers. Simultaneously, a learnable residual weight mechanism is introduced to adaptively scale the predicted offset and then perform a residual connection with the base mesh to obtain a refined mesh vertex. .

[0067] This invention conducted systematic ablation experiments on the FreiHAND dataset to quantitatively analyze the specific contribution of each component to the model performance.

[0068] Table 1 shows the analysis of each module of this invention. Starting from the baseline model (the average position error per joint based on Protodyakonov analysis is 2.71 mm), different designs are gradually introduced. The hand layered encoding module explicitly models the anatomical layers of the hand, enabling the network to better distinguish the semantic functions of different joints, thereby reducing the average position error per joint based on Protodyakonov analysis to 2.50 mm. The topology-aware Transformer module introduces learnable hand reference points and incorporates the skeletal topology as a priori into the attention mechanism, effectively alleviating the blindness of standard self-attention in structural modeling, further reducing the error by 0.17 mm. Combining the two modules reduces the error by 0.45 mm, significantly better than individual components. Furthermore, adding a joint-guided mesh reconstruction module further reduces the average position error per vertex based on Protodyakonov analysis to 5.2 mm, demonstrating that the joint-guided mesh reconstruction module can effectively propagate features containing joint information to vertices, thereby achieving high-precision mesh reconstruction. The baseline model is derived from the Human Pose Estimation Model (DDHPose), which is designed for temporal human pose estimation tasks and can model cross-temporal human motion relationships from consecutive frames. For the characteristics of hand pose estimation tasks, the baseline model, while retaining its overall structural design principles, has been specifically simplified and adapted. Specifically, firstly, the temporal modeling module in the original model has been removed, changing the input format from a sequence of human keypoints to a single-frame two-dimensional hand pose input. Secondly, the original human skeleton topology has been replaced with a hand topology containing 21 joint nodes, and the corresponding joint hierarchy and adjacency relationships have been redefined. Simultaneously, the multi-hypothesis mechanism in the original model has been changed to a single-hypothesis prediction structure to reduce redundant computation and improve prediction certainty. Furthermore, this invention has also trimmed and simplified some sub-modules related to whole-body motion in the original model, retaining only the core branches related to local spatial relationship modeling, making the model structure more lightweight and adaptable to learning fine-grained hand motion features. The modified model serves as the baseline model of this invention.

[0069] Table 1

[0070]

[0071] Table 2 shows the analysis of joint type-guided coding and thumb-independent coding compared to hierarchical coding based on finger structure in this invention. In posture estimation, different fingers and joints differ in motor function and structural constraints. If a uniform coding method is directly adopted, it is often difficult to fully express this hierarchical information. Therefore, the hand hierarchical coding module organizes joints into seven anatomical layers to provide differentiated semantic representations for joints with different functions. The hand hierarchical coding improves the average per-vertex position error by 0.21 mm compared to the baseline model based on Protodyakonov analysis, verifying the effectiveness of hierarchical modeling. Furthermore, hierarchical coding based on finger structure is compared with two other alternatives (joint type-guided coding and thumb-independent coding). The results in Table 2 show that hierarchical coding based on finger structure exhibits the best performance in posture localization, proving that structure-guided design can better capture the anatomical patterns of hand movements.

[0072] Table 2

[0073]

[0074] In joint-guided mesh remodeling, the focus is on how to more effectively utilize the complementarity between joint features and vertex features. Simply relying on vertex features can easily lead to a loss of local detail, while joint guidance alone cannot guarantee overall consistency. To address this issue, this invention designs a dual attention mechanism that simultaneously incorporates joint-to-vertex interactive attention and vertex self-attention. This mechanism can refine local geometry under joint constraints while maintaining global structural consistency by utilizing long-range dependencies between vertices. Comparative experimental results with those using only cross-attention or a hybrid attention and graph convolution scheme, as shown in Table 3, demonstrate that dual attention achieves the best performance in key metrics such as average position error per vertex, average position error per vertex based on Protodyakonov analysis, and area under the curve. Furthermore, this mechanism better maintains joint position accuracy and mesh surface geometric fidelity under complex postures such as finger crossing and extreme bending, especially excelling in the fingertip-joint connection region, thus verifying the rationality and effectiveness of the design.

[0075] Table 3

[0076]

[0077] Another embodiment of the present invention provides a multimodal 3D hand mesh reconstruction system based on topology-aware Transformer, comprising:

[0078] The data preprocessing and feature fusion module is used to receive the real 3D joint coordinates of the hand, generate the corresponding 2D joint coordinates and noisy joint sequences through perspective projection and forward noise processing, respectively, and perform cross-dimensional feature fusion to output the fused features.

[0079] A topology-aware hierarchical coding denoising network module, connected to the data preprocessing and feature fusion module, is used to receive the fused features and output joint features containing structural information; this module further includes:

[0080] The hand layered coding unit is used to divide the hand joints into multiple layers from the wrist to the fingertips according to their anatomical structure, and equips each functional group with learnable embedding parameters.

[0081] Hierarchical feature aggregation units are set between adjacent topology-aware Transformer modules. Based on the topological relationship of the hand skeleton tree, hierarchical features are constructed through bottom-up information propagation.

[0082] At least one topology-aware Transformer module is used to model the spatial topological relationships between joints based on hand reference points. It integrates a baseline topology construction subunit, an adaptive offset learning network subunit, and a structured relative position encoding subunit to compute topology-aware attention weights.

[0083] A 3D reconstruction output module, connected to the topology-aware hierarchical coding denoising network module, is used to average pool the joint features into global features, and further includes:

[0084] The joint regression head unit is used to directly regress the global features to generate accurate 3D hand joint poses;

[0085] The joint-guided mesh reconstruction unit is used to generate refined hand mesh vertices by taking the global features and the joint features as inputs and employing a two-stage reconstruction strategy from coarse to fine.

[0086] Another embodiment of the present invention provides a non-volatile computer storage medium storing computer-executable instructions that execute the multimodal three-dimensional hand mesh reconstruction method.

[0087] Another embodiment of the present invention provides an electronic device, including: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the multimodal three-dimensional hand mesh reconstruction method.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal 3D hand mesh reconstruction method based on topology-aware Transformer, characterized in that, Includes the following steps: Step 1: Given the real 3D joint coordinates of the hand, perform perspective projection and forward noise addition to obtain the corresponding 2D joint coordinates and 3D noisy joint sequence, and then perform cross-dimensional feature fusion to obtain the fused feature; Step 2: The fused features are output as joint features containing structural information through a topology-aware hierarchical coding denoising network. This network comprises a hand layered coding module, a hierarchical feature aggregation mechanism, and a topology-aware Transformer module. First, the hand layered coding module organizes the hand joints from the wrist to the fingertips into seven functional groups based on anatomical function: wrist, metacarpophalangeal joints, thumb, index finger, middle finger, ring finger, and little finger. Each functional group is equipped with learnable embedding parameters. Then, the spatial topological relationships between joints are modeled using hand reference points learned by several topology-aware Transformer modules to obtain joint features. A hierarchical feature aggregation mechanism is set between adjacent topology-aware Transformer modules to construct hierarchical features through bottom-up information propagation. In the hierarchical feature aggregation mechanism, the mapping relationship between each joint and its child joints is recorded according to the topological structure of the hand skeletal tree. For a parent joint with child joints, its own features are concatenated with the features of all direct child joints and then averaged. For leaf joints, their original features are used directly. Among them, the topology-aware attention weight of the topology-aware Transformer module is obtained by summing the feature similarity attention weight, structured position attention weight, and hierarchical semantic attention weight and then activating it with the Softmax function; Step 3: After average pooling of the joint features, the global features are obtained. The global features are input into the joint regression head to directly generate the 3D hand joint pose. At the same time, the global features are input into the joint-guided mesh reconstruction module, and the hand mesh vertices are obtained by adopting a coarse-fine two-stage strategy.

2. The multimodal three-dimensional hand mesh reconstruction method according to claim 1, characterized in that, The topology-aware Transformer module includes baseline topology construction, an adaptive offset learning network, and structured relative position encoding; The baseline topology construction parameterizes the basic topological structure of the hand into four learnable parameters: ； in: This indicates the relative angular distribution of the five fingers on the palm. These represent the relative angles of the five fingers on the palm. This indicates the proportional relationship of bone length from proximal to distal. These represent the lengths of the metacarpophalangeal joint, proximal interphalangeal joint, distal interphalangeal joint, and fingertip, respectively. This indicates a specific angular offset of the thumb; Indicates the range limitation factor for dynamic offset; Topology-aware hierarchical coding denoising network receives 2D joint coordinates and 3D noise joint sequences Spatial patch embedding layers are used as input to achieve cross-dimensional feature fusion; After the fusion features are processed by the hand-layered encoding module, spatial location information is further fused. and time step information This forms the final input features. Input features After being processed in the adaptive offset learning network of the topology-aware Transformer, the 2D offset of each joint is obtained. The sampling location should be determined under constraints. The sampling location and the four learnable parameters are processed together in the structured relative position encoding to obtain the multi-head attention bias. The multi-head attention bias and hierarchical features are processed in the normalization layer of the topology-aware Transformer module, and the attention weights are calculated in the self-attention module to obtain the output features. The output features of the two branches are weighted and fused and then entered into the normalization layer for normalization. Finally, the feature transformation is performed by the multilayer perceptron to obtain the output features of the topology-aware Transformer module.

3. The multimodal three-dimensional hand mesh reconstruction method according to claim 2, characterized in that, The adaptive offset learning network is used to learn the structural adjustment of the current gesture relative to the standard topology and apply it to the reference point coordinates, and finally outputs the 2D offset of each joint.

4. The multimodal three-dimensional hand mesh reconstruction method according to claim 2, characterized in that, Topology-aware attention is calculated as follows: ； in, Topology-aware attention weights, feature similarity attention weights Responsible for capturing semantic correlations between joints; structured position attention weights Embedding the geometric constraints of the hand-topology into the attention computation gives stronger association weights to spatially adjacent joints; hierarchical semantic attention. This incorporates the parent-child dependency relationship from hierarchical features, ensuring that the attention allocation for each joint can perceive its hierarchical position in the kinematic chain; where Q represents the query matrix and K represents the key matrix. The transpose of the key matrix representing hierarchical features. Indicates the matrix dimension.

5. A system for implementing the multimodal three-dimensional hand mesh reconstruction method according to any one of claims 1-4, characterized in that, include: The data preprocessing and feature fusion module is used to receive the real 3D joint coordinates of the hand, generate the corresponding 2D joint coordinates and 3D noisy joint sequences through perspective projection and forward noise addition, respectively, and perform cross-dimensional feature fusion to output the fused features. A topology-aware hierarchical coding denoising network module is connected to the data preprocessing and feature fusion module, and is used to receive the fused features and output joint features containing structural information; This module further includes: The hand layered coding unit is used to divide the hand joints into multiple layers from the wrist to the fingertips according to their anatomical structure, and equips each functional group with learnable embedding parameters. Hierarchical feature aggregation units are set between adjacent topology-aware Transformer modules. Based on the topological relationship of the hand skeleton tree, hierarchical features are constructed through bottom-up information propagation. At least one topology-aware Transformer module is used to model the spatial topological relationships between joints based on hand reference points. It integrates a baseline topology construction subunit, an adaptive offset learning network subunit, and a structured relative position encoding subunit to compute topology-aware attention weights. A 3D reconstruction output module, connected to the topology-aware hierarchical coding denoising network module, is used to average pool the joint features into global features, and further includes: The joint regression head unit is used to directly regress the global features to generate 3D hand joint poses; Joint-guided mesh reconstruction units are used to generate hand mesh vertices based on the global features.

6. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer can execute instructions to perform the multimodal three-dimensional hand mesh reconstruction method according to any one of claims 1-4.

7. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, characterized in that the instructions are executed by the at least one processor to cause the at least one processor to perform the multimodal three-dimensional hand mesh reconstruction method according to any one of claims 1-4.

Images