A three-dimensional human body fitting method based on multi-modal dense marker point prediction

Abstract

This invention discloses a 3D human body fitting method based on multimodal dense marker prediction. The steps include: predefining human body markers on an SMPL-X template mesh using a non-uniform density distribution strategy as the core supervision signal for fitting; receiving 3D human body point cloud data and processing the input point cloud data through a feature extraction module; decoding predicted human body markers from high-dimensional feature vectors using a marker prediction module; and fitting an SMPL-X parametric human body model using an optimization algorithm. If there is an accompanying RGB image input, image features are extracted using a plug-and-play image feature extraction module. If the input point cloud data has scale distortion, scale-independent fitting is performed using a scale prediction module. The fitting accuracy of this invention reaches the millimeter level; it has excellent multimodal adaptability and robustness; it automatically identifies and corrects scale; it balances computational efficiency and accuracy; and it has strong generalization ability.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a three-dimensional human body fitting method based on multimodal dense marker point prediction. Background Technology

[0002] Fitting low-level parametric human models (such as SMPL-X) to 3D Clothed HumanAssets is a fundamental task in computer vision and graphics, and is widely used in human animation, virtual avatar creation, and AR / VR fields.

[0003] Traditional methods are mainly divided into two categories:

[0004] Optimization-based methods: Early methods utilized the Iterative Closest Point (ICP) algorithm and its variants to align the template with the target geometry. Modern methods typically employ a multi-stage pipeline: first, the input scan is rendered as a multi-view RGB image, 2D key points are detected, and then... Figure 3 Angle measurements are then performed to 3D keypoints, and finally, SMPL-X parameters are optimized to align these keypoints. This type of approach relies on multi-view rendering, and errors accumulate and propagate at each stage.

[0005] Learning-based methods utilize large-scale 3D human body datasets and deep neural networks to directly regress statistical parameters of human body models or provide initialization. Since direct parameter regression is difficult, existing methods often rely on intermediate surrogate representations, such as joint features, correspondence maps, component segmentation, or sparse marker points.

[0006] In real-world scenarios, data sources are often multimodal (e.g., RGB-D cameras simultaneously capture color and geometry, or AIGC tools generate textured 3D meshes). However, most existing methods focus only on single-modal inputs (point clouds only or multi-view images only). Furthermore, AIGC-generated assets commonly suffer from scale distortion (e.g., normalization to unit bounding boxes leads to anomalous scales), and existing methods typically assume a known metric scale, making it difficult to handle inputs with such unknown scales.

[0007] The main improvement methods currently include fitting methods based on sparse marker points and optimization methods based on multi-view images:

[0008] Methods based on sparse markers (such as ETCH) aggregate dense correspondences into a compact set of markers fixed on a human body template. While this simplifies training, markers are easily lost when dealing with missing or partial inputs (such as single-view point clouds), leading to fitting failures. Sparse markers are also difficult to cover regions with complex degrees of freedom and fine structures, such as the hands and face, resulting in lower fitting accuracy for these areas.

[0009] Multi-view image optimization methods (such as EasyMocap) rely on multi-view RGB images and camera poses, performing optimization through rendering and 2D detection. These methods cannot handle textureless meshes or partial point clouds, have complex computational processes, and are highly demanding on input data. Furthermore, these methods heavily depend on the accuracy of the 2D detector; if 2D detection fails (e.g., occlusion leading to keypoint loss), subsequent optimization is difficult to correct.

[0010] It is evident that the existing technical solutions have significant shortcomings in the following aspects:

[0011] Poor modal adaptability: Most existing methods are designed for a single modality and cannot flexibly utilize the existing point cloud and image information, resulting in decreased robustness when some data is missing or the modality is incomplete.

[0012] High scale sensitivity: For AIGC-generated 3D assets, due to the lack of real scale information and the scale distortion caused by normalization, existing methods often cannot accurately fit the data and require manual intervention or additional scale calibration.

[0013] Low accuracy in fitting details: The sparse marker-based method does not cover fine areas such as hands, feet, and face, resulting in large fitting errors in these key areas; while the multi-view optimization method suffers from severe error propagation and is difficult to achieve millimeter-level accuracy.

[0014] Limited input processing capability: When the input is only a single view or a partial point cloud, existing methods based on aggregated markers are prone to failure due to missing regions. Summary of the Invention

[0015] To address the aforementioned problems in existing technologies, this invention proposes a unified multimodal framework that solves these problems through dense marker prediction, plug-and-play image adaptation, and scale prediction modules, thereby achieving high-precision, robust, and scale-independent 3D human body fitting.

[0016] To achieve the above objectives, the present invention provides the following technical solution:

[0017] A 3D human body fitting method based on multimodal dense marker prediction includes the following steps:

[0018] Step 1: Predefine M human body markers on the SMPL-X template mesh as the core supervision signal for model fitting;

[0019] Step 2: Receive 3D human point cloud data and process the input point cloud data through the feature extraction module; first, the input unordered data is divided into blocks and transformed into an embedding sequence through linear projection; then the sequence is fed into the encoder for deep feature learning. The encoder captures local geometric structure information and global spatial context through a self-attention mechanism, and finally outputs a high-dimensional feature vector for each embedding sequence.

[0020] Step 3: Use the marker prediction module to decode the predicted human body markers from the high-dimensional feature vector;

[0021] The marker prediction module is based on a decoder containing a multi-layer Transformer structure; it embeds the M predefined human markers in step 1 into a marker embedding vector, which is used as the initial query vector, and uses the learnable marker embedding vector as the query vector, and uses the high-dimensional feature vector output by the feature extraction module in step 2 as the key-value pair; it iteratively updates through a multi-layer cross attention mechanism and a self-attention mechanism, and finally outputs the predicted human marker coordinates through the multi-layer perceptron MLP regression head;

[0022] Step 4: Based on the predicted human body marker coordinates obtained in Step 3, the SMPL-X parametric human body model is fitted using an optimization algorithm with the spatial distance of the corresponding marker points as the loss.

[0023] Furthermore, in step 1, a non-uniform density distribution strategy is adopted to predefine a set of non-uniform dense human body markers on the SMPL-X template grid as the core supervision signal for model fitting. The non-uniform dense human body markers are deployed in a relatively denser number of markers in the fine and complex areas of human body movements to enhance the fitting accuracy of the fine and complex areas of human body movements. The fine and complex areas of human body movements include, but are not limited to, the facial area and the hand area.

[0024] Furthermore, in step 2, if there is an accompanying RGB image input, image features are extracted through a plug-and-play image feature extraction module. The image feature extraction module introduces a lightweight image cross-attention branch in parallel with the point cloud cross-attention branch in each layer of the decoder. This fusion mechanism corrects prediction biases that may arise from relying solely on geometric information by utilizing high-frequency details in the image. If there is no image input, the image cross-attention branch automatically fails, and the fitting method degenerates into a pure geometry-driven mode to make the system more versatile.

[0025] Furthermore, in step 2, if the input point cloud data has scale distortion, a scale-independent fitting is performed through the scale prediction module. The scale prediction module embeds a learnable "scale token" at the beginning of the point cloud encoding sequence. Then, it shares the encoder architecture, embeds scale estimation into the feature encoding process, uses the Transformer's self-attention mechanism to aggregate the global geometric information of the point cloud, and finally outputs the scale factor through the regression head to restore the scale-distorted data to the true physical scale.

[0026] Furthermore, in step 2, before the data enters the feature extraction module, the input point cloud data is normalized to a fixed unit bounding box to eliminate the influence of absolute scale.

[0027] Furthermore, in step 2, the high-dimensional feature vector, while preserving the geometric shape information of the point cloud, also implicitly contains the semantic correspondence of various parts of the human body, providing rich feature support for the accurate prediction of subsequent marker points.

[0028] Furthermore, in step 2, the encoder is based on the Point-BERT architecture.

[0029] Furthermore, in step 3, the decoding process of the decoder includes... Layer iterative update, for the th Layer, hidden state The update steps are as follows:

[0030] Firstly, through the cross-attention mechanism With point cloud feature vectors Interact with the target surface points to establish an implicit correspondence between them:

[0031] ;

[0032] Following this, through the self-attention mechanism Integrate global body structure constraints within the query vector:

[0033] ;

[0034] After multiple transformations, the final feature vector output by the decoder The regression head of the multilayer perceptron (MLP) is used to directly map the data into M three-dimensional spaces. Coordinates of densely marked points in ,Right now:

[0035] .

[0036] Furthermore, the fitting method is a "prediction + lightweight optimization" process. First, the coordinates of the marker points are obtained through a dense marker point predictor that includes a feature extraction module and a marker point prediction module. Then, the SMPL-X model parameters are optimized based on the marker point distance to achieve high-precision fitting of the input human body.

[0037] Furthermore, in step 3, the optimization method is based on the Theseus second-order optimizer, which optimizes the densely marked points obtained from the current SMPL-X parameters. , and the predicted marker points The mean square loss of the three-dimensional coordinates between them is used to optimize the final SMPL-X parameter model. The objective function is:

[0038]

[0039] Furthermore, the optimization process optimizes the SMPL-X parameter as the variable and iterates 50 times.

[0040] Furthermore, the applicability of the method includes, but is not limited to, any of the following application scenarios:

[0041] Automated binding of film and game assets;

[0042] Human posture analysis in medical rehabilitation;

[0043] Real-time virtual avatar driving in AR / VR;

[0044] Post-processing and binding of 3D models generated by AIGC.

[0045] By comparing and testing with the closest existing technologies such as ETCH (Sparse Marker Representation) and EasyMocap (Multi-View Optimized Representation), this invention demonstrates significant technical advantages:

[0046] 1. Fitting accuracy reaches the millimeter level, comprehensively surpassing the state-of-the-art (SOTA) standard.

[0047] Data support: On authoritative CAPE and 4D-DRESS benchmark sets, this invention reduces vertex-to-vertex distance error by 57.1% - 75.5% and average joint position error by 67.1% - 80.9%.

[0048] Detail performance: Especially in complex areas such as the hands (finger joints) and face (facial muscles), the error reduction is even greater. This invention is the first three-dimensional human body fitting method to stably achieve millimeter-level (<2mm) fitting accuracy on a publicly available benchmark.

[0049] 2. Excellent multimodal adaptability and robustness

[0050] Compared to ETCH: ETCH suffers from a fitting failure rate exceeding 40% under single-view or partial point cloud input due to marker loss. This invention, benefiting from dense landmarks and an attention mechanism, maintains high accuracy under the same conditions, with a failure rate below 5%. Furthermore, the method of this invention outperforms existing point cloud-based methods (IPNet, PTF, NICP, ArtEq, ETCH) in both qualitative and quantitative performance. It achieves superior results in the quantitative metrics of vertex distance (distance between vertices in the predicted SMPL-X mesh and the label, in centimeters) and joint distance (distance between joints in the predicted SMPL-X model and the label, in centimeters), particularly for the hand and face regions, where it achieves significant lead. A comparison of the qualitative results of this invention with other point cloud-based methods is provided below. Figure 6 As shown in Table 1, the quantitative results of the method of this invention and other point cloud input-based methods are compared:

[0051] Table 1

[0052]

[0053] Compared to EasyMocap: EasyMocap cannot handle pure point clouds or textureless meshes. This invention can handle pure point clouds (with accuracy even superior to EasyMocap's multi-view results) and further enhance them by fusing images, truly achieving "all-modal compatibility." Compared to other methods based solely on multi-view images (DiffProxy, EasyMocap), the qualitative results of this invention's method and other methods based on multi-view image input are as follows: Figure 7 As shown in Table 2, the quantitative results of the method of this invention and other methods based on multi-view image input are compared:

[0054] Table 2

[0055]

[0056] 3. Unique scale-independent universal capability

[0057] Addressing the pain point: Existing methods almost entirely fail with normalized models generated by AIGC. This invention's built-in scale predictor can automatically identify and correct the scale without any manual intervention. Real-world testing shows that even with inputs scaled down by a factor of 10 or up by a factor of 5, this invention can still output a correctly proportioned SMPL-X model, filling a gap in the industry.

[0058] 4. Balancing computational efficiency and accuracy:

[0059] Although the Transformer is introduced, the inference speed remains considerable thanks to the use of the efficient Point-BERT backbone and parallel adapters. Compared to pure optimization methods that require multiple rounds of rendering and optimization, the "prediction + lightweight optimization" process of this invention is an order of magnitude faster.

[0060] 5. Strong generalization ability:

[0061] By training with a mixture of synthetic and real data and introducing some data augmentation, the general model generated by this invention exhibits strong generalization on unseen datasets and can be used directly without retraining for new scenarios. Attached Figure Description

[0062] Figure 1 It is a map showing the distribution of human body markers;

[0063] Figure 2 This is a schematic diagram of a dense marker predictor;

[0064] Figure 3 This is a schematic diagram of the image feature extraction module;

[0065] Figure 4 This is a schematic diagram of the scale prediction module;

[0066] Figure 5 It is a semantic relationship graph of point cloud features;

[0067] Figure 6 This is a comparison chart of the qualitative results of the method of this invention and other point cloud input-based methods;

[0068] Figure 7 This is a comparison chart of the qualitative results of the method of this invention and other methods based on multi-view image input. Detailed Implementation

[0069] The present invention will now be described in further detail with reference to the accompanying drawings.

[0070] This invention proposes a framework to improve upon existing 3D human body fitting methods, addressing their poor modal adaptability, scale sensitivity, and low detail accuracy. The core idea is to utilize a conditional Transformer decoder to directly transform surface points into dense human body markers, supplemented by image enhancement and scale correction. This method is independent of specific input devices or generation tools; it only requires access to 3D point cloud data.

[0071] 1. Overall Framework Overview

[0072] First, the system receives a point cloud input of a 3D dressed human body (which can be from a real scan, 3DGS point location, or depth map reconstruction). If an accompanying RGB image exists, visual features are fused using a plug-and-play image adapter. If the input point cloud exhibits scale distortion (such as AIGC assets), it is first restored to standard human body dimensions using a scale predictor. Subsequently, a predefined set of dense SMPL-X markers is estimated using a dense marker predictor. Finally, the SMPL-X model parameters are iteratively optimized based on the predicted markers to complete the fitting process.

[0073] 2. Definition and distribution strategy of dense markers

[0074] like Figure 1 As shown, the present invention first predefines a set of features on the SMPL-X template mesh that include... A set of dense human body markers is denoted as _____ points. As the core supervisory signal for model fitting, this invention employs a non-uniform distribution strategy to overcome the shortcomings of traditional uniformly distributed markers in fine-grained regions. 120 markers are deployed in the head region to capture complex facial expressions and details; 180 markers are deployed in the hand region to accurately fit the fine gestures of the fingers; and the remaining 300 markers are distributed across the torso and limbs to cover the main body structure. All markers are obtained using the farthest point sampling algorithm across different parts of the human body. This invention utilizes a non-uniform, densely distributed strategy in high-degree-of-freedom regions of the human body (such as finger joints and facial features). This targeted distribution design ensures higher fitting accuracy in critical and error-prone areas such as the face and hands, laying a data foundation for subsequent millimeter-level reconstruction.

[0075] 3. Dense Marker Prediction Based on Transformer

[0076] Figure 2 This describes the structure of the human body marker prediction module. Given an input point cloud, the marker predictor aims to predict the corresponding dense markers. The input point cloud contains a range of points. First, point cloud features are obtained through a point cloud encoder, and then injected as conditions into the marker point decoder. The marker point decoder is a transformer structure that uses learnable marker point embeddings as indices to fuse with the point cloud features, and finally decodes the final marker points through an MLP. Meanwhile, the marker point predictor can also utilize image features, which will be discussed in detail below.

[0077] like Figure 2 As shown, when processing input data, the marker prediction module uses a point cloud encoder based on the Point-BERT architecture to process the original 3D point cloud. Feature extraction is then performed. First, the input unordered point cloud data is divided into blocks and then transformed into an embedding sequence through linear projection. ,in The number of embeddings is given. The sequence is then fed into an encoder containing multiple Transformer blocks for deep feature learning. The encoder captures local geometric information and global spatial context of the point cloud through a self-attention mechanism, ultimately outputting a high-dimensional feature representation for each embedding. ,in These features not only preserve the geometric shape information of the point cloud, but also implicitly contain the semantic correspondences between different parts of the human body (the semantic relationships of point cloud features are as follows). Figure 5 As shown, attention heatmaps for the head, body, and hands are displayed respectively, demonstrating that the point cloud features implicitly contain semantic correspondences between different parts of the human body, providing rich feature support for the accurate prediction of subsequent labeled points.

[0078] The core prediction module is a decoder based on a Transformer architecture. This decoder is embedded with M predefined, randomly initialized, learnable markers. As the initial query vector, the features output by the point cloud encoder As key-value pairs. The decoding process includes... Layer-by-layer iterative update: For the first layer Layer, hidden state The update formula is as follows: First, through the cross-attention mechanism Interact with point cloud features to establish an implicit correspondence between input surface points and target semantic marker points:

[0079]

[0080] Following this, through the self-attention mechanism Integrating global body structure constraints within the query vector, the calculation formula is:

[0081]

[0082] After multiple transformations, the final feature vector output by the decoder Through a multilayer perceptron (MLP) regression head, it is directly mapped to dense landmark coordinates in three-dimensional space, that is:

[0083]

[0084] For the training part of the core module, this embodiment uses 4D-DRESS, CAPE, and self-collected datasets for training, and the loss function for training is the predicted coordinates of the labeled points. With real marker labels Mean square loss of three-dimensional coordinate points :

[0085] .

[0086] The aforementioned dense marker prediction mechanism based on the Transformer architecture abandons the traditional approach of regressing dense correspondence fields or predicting sparse markers. Instead, it defines a large set of learnable landmark embedding vectors (e.g., over 600) with optimized distribution strategies as the query input for the decoder. These query vectors, through a cross-attention mechanism, dynamically aggregate information directly based on the geometric features of the input point cloud, thereby regressing the coordinates of dense 3D landmarks on the human body surface end-to-end. This invention utilizes the global attention characteristic of the Transformer to robustly infer the location of landmarks in missing regions based on contextual information even when the input point cloud suffers from severe occlusion, truncation, or noise interference. This mechanism fundamentally solves the problems of sparse marker methods failing when data is missing and the insufficient accuracy of direct regression parameter methods.

[0087] 4. Plug-and-play image feature fusion module

[0088] To fully utilize the rich texture and appearance information in RGB images to improve fitting results, this invention designs a plug-and-play image feature fusion module. Figure 3 This is a structural diagram of the plug-and-play image feature fusion module. This module adds a lightweight cross-attention branch to the marker prediction network and connects it in parallel with the existing point cloud cross-attention layer. Image and point cloud features are fused in parallel without changing the structure of the original marker prediction network, thus achieving plug-and-play functionality.

[0089] like Figure 3 As shown, when a front view RGB image exists During input, the system utilizes a pre-trained image encoder DINOv2 network to extract high-dimensional visual features. In each layer of the marker-based decoder, a lightweight image cross-attention branch is introduced in parallel, extending the original cross-attention formula to...

[0090]

[0091] This fusion mechanism allows the model to utilize high-frequency details in the image (such as facial contours, clothing folds, and finger edges) to correct prediction biases that might arise from relying solely on geometric information. If the input does not contain an image, this branch automatically becomes invalid (i.e., ...). (With zero terms), the model degenerates into a pure geometry-driven mode, ensuring the system's versatility.

[0092] This plug-and-play image feature fusion prediction module introduces a parallel image cross-attention branch design into the landmark decoder. This architecture innovatively decouples point cloud geometry processing from image texture fusion, allowing the model to dynamically inject 2D visual features extracted by the pre-trained image encoder into the 3D landmark decoding process without altering the backbone network structure. This approach is a modular design of "backbone processing geometry, branches enhancing texture," supporting flexible enabling or disabling of image input during the inference phase based on data availability. Regardless of whether the input is pure point cloud, pure image, or multimodal data, this architecture can adaptively adjust, especially in geometrically ambiguous regions (such as interlaced fingers or clothing folds), significantly eliminating ambiguity and improving fitting accuracy by utilizing image texture cues, achieving true multimodal complementarity and universal adaptability.

[0093] 5. Scale prediction module for scale distortion

[0094] To address the common scale distortion problem in AI-generated assets (AIGC) or unknown sensor data, this invention integrates a separate scale prediction module to achieve scale-independent fitting. Figure 4 This is a structural diagram of the scale prediction module. The module is similar in structure to the point cloud feature extractor, but it adds a dedicated learnable scale token to the input point cloud embedding sequence to decode the scale of the input human point cloud.

[0095] like Figure 4 As shown, before the data enters the main network, the input point cloud is first processed. Normalize to a fixed unit bounding box (e.g., range) This eliminates the influence of absolute scale. Subsequently, a shared point cloud encoder architecture is used, but a dedicated, learnable "scale token" is injected at the beginning of the input sequence. The input sequence becomes The token is passed through the Transformer layer, aggregating global geometric statistics, and ultimately outputting the predicted scaling factor through a regression head. (As a multiplicative factor, it has no range limit; labels are obtained through back-calculation). The system uses this factor to recover the normalized point cloud to the true human physical scale (recovered after normalization, separated from the main module, and not involved in the gradient backpropagation of the main module's labeled point prediction). The calculation formula is as follows: This solves the problem of proportional errors caused by the normalization operation of the generative model.

[0096] The scale prediction module aims to completely solve the scale distortion problem in AIGC-generated assets and normalized data. This module embeds a learnable, special "scale token" at the beginning of the point cloud encoding sequence. Leveraging the global awareness capability of the Transformer's self-attention mechanism, this token aggregates the global geometric distribution features of the entire point cloud (such as overall span and aspect ratio), and then outputs an accurate scale factor through the regression head. This is a technical approach that embeds scale estimation into the feature encoding process. The predicted scale factor is used to perform front-end rescaling correction on the input point cloud, restoring it to a standard physical scale before it is fed into the subsequent fitting module, forming a complete closed-loop process. This mechanism allows the algorithm to automatically adapt to input data of any scale without relying on external calibration or manual intervention, filling the gap in existing technologies for handling assets generated at unknown scales.

[0097] With the explosive growth of metaverse, digital humans, virtual anchors, and AI-generated 3D content creation, the global demand for high-quality, automated human modeling is increasing exponentially. It is predicted that the digital human market will reach hundreds of billions of dollars in the next five years. This invention can process data from various sources (professional scanning, consumer-grade RGB-D, AI-generated), breaking through data source bottlenecks and possessing extremely high commercial applicability. Examples of application scenarios for this invention are as follows:

[0098] (1) Film and game production

[0099] In traditional workflows, converting actor scan data or concept designs into animated, rigging models requires experienced technical artists to spend hours to days manually drawing topologies and weights, resulting in high costs. This invention can automate this process to the second level, directly outputting high-precision models with standard SMPL-X skeletons. This will significantly reduce the manpower costs of film and television special effects and game development, shortening asset production cycles from weeks / days to minutes, greatly accelerating project iterations, and supporting the rapid generation of large-scale NPC extras assets.

[0100] (2) Medical treatment and rehabilitation

[0101] Traditional human posture and body shape analysis relies on expensive specialized laboratory equipment or subjective visual assessment, making it difficult to popularize. This invention, utilizing low-cost depth cameras or monocular cameras, enables high-precision human posture analysis, scoliosis detection, gait analysis, and body shape monitoring. This provides doctors with objective and quantitative rehabilitation assessment tools, assisting in orthotic customization and surgical planning, and promoting the widespread adoption of telemedicine and home-based rehabilitation monitoring.

[0102] (3) AR / VR interaction

[0103] Current virtual avatars in VR social networking and games often exhibit stiff movements and lack detail, severely diminishing the sense of immersion. This invention supports real-time, high-precision full-body tracking, enabling realistic digital avatars driven by monocular or RGB-D data even without a full-body sensor kit. This will significantly enhance the user experience of metaverse social platforms, virtual concerts, and immersive games.

[0104] (4) AIGC workflow integration

[0105] Current 3D model generation methods generally suffer from scale distortion and the inability to directly bind animations, hindering the industrial application of AIGC content. This invention can serve as a standard post-processing plugin for AIGC 3D generation platforms, automatically correcting the scale of generated models and extracting bones, making it directly compatible with mainstream engines such as Unity and Unreal Engine. This will remove the key bottleneck from "generation" to "application" of AIGC, becoming an essential component in all 3D AIGC toolchains.

[0106] (5) High technical barriers

[0107] The millimeter-level fitting accuracy achieved by this invention in publicly available benchmark tests (with a V2V error reduction of over 70%) far surpasses existing commercial solutions. Its prediction mechanism for AIGC asset scale distortion fills a gap in the current market for a fully automated, high-precision solution to this problem.

[0108] The above embodiments are merely typical implementations of the present invention and are not intended to limit the present invention. Any equivalent substitutions or improvements made within the scope of the claims of the present invention are within the protection scope of the present invention.

Claims

1. A three-dimensional human body fitting method based on multimodal dense marker point prediction, characterized in that, Includes the following steps: Step 1: Predefine M human body markers on the SMPL-X template mesh as the core supervision signal for model fitting; Step 2: Receive 3D human point cloud data and process the input point cloud data through the feature extraction module; first, the input unordered data is divided into blocks and transformed into an embedding sequence through linear projection; then the sequence is fed into the encoder for deep feature learning. The encoder captures local geometric structure information and global spatial context through a self-attention mechanism, and finally outputs a high-dimensional feature vector for each embedding sequence. Step 3: Use the marker prediction module to decode the predicted human body markers from the high-dimensional feature vector; The marker prediction module is based on a decoder containing a multi-layer Transformer structure; corresponding to the M human markers predefined in step 1, M learnable marker embedding vectors are set as the initial query vectors, and the high-dimensional feature vectors output by the feature extraction module in step 2 are used as key-value pairs; through multi-layer cross-attention mechanism and self-attention mechanism, the module iteratively updates and finally outputs the predicted human marker coordinates through the multi-layer perceptron MLP regression head. Step 4: Based on the predicted human body marker coordinates obtained in Step 3, fit the SMPL-X parametric human body model using an optimized algorithm.

2. The method according to claim 1, characterized in that, In step 1, a non-uniform density distribution strategy is adopted. A set of non-uniform dense human body markers is predefined on the SMPL-X template mesh as the core supervision signal for model fitting. The non-uniform dense human body markers are deployed in a relatively denser number of markers in the fine and complex areas of human body movements to enhance the fitting accuracy of the fine and complex areas of human body movements. The fine and complex areas of human body movements include, but are not limited to, the facial area and the hand area.

3. The method according to claim 1, characterized in that, In step 2, if there is an accompanying RGB image input, image features are extracted through a plug-and-play image feature extraction module. The image feature extraction module introduces a lightweight image cross-attention branch in parallel with the point cloud cross-attention branch in each layer of the decoder. This fusion mechanism corrects prediction biases that may arise from relying solely on geometric information by utilizing high-frequency details in the image. If there is no image input, the image cross-attention branch automatically fails, and the fitting method degenerates into a pure geometry-driven mode to make the system more versatile.

4. The method according to claim 1, characterized in that, In step 2, if the input point cloud data has scale distortion, a scale-independent fitting is performed through the scale prediction module. The scale prediction module embeds a learnable "scale token" at the beginning of the point cloud encoding sequence. Then, it shares the encoder architecture, embeds scale estimation into the feature encoding process, uses the Transformer's self-attention mechanism to aggregate the global geometric information of the point cloud, and finally outputs the scale factor through the regression head to restore the scale-distorted data to the true physical scale.

5. The method according to claim 4, characterized in that, In step 2, before the data enters the feature extraction module, the input point cloud data is normalized to a fixed unit bounding box to eliminate the influence of absolute scale.

6. The method according to claim 1, characterized in that, In step 2, the high-dimensional feature vector, while preserving the geometric shape information of the point cloud, also implicitly contains the semantic correspondence of various parts of the human body, providing rich feature support for the accurate prediction of subsequent marker points.

7. The method according to claim 1, characterized in that, In step 2, the encoder is based on the Point-BERT architecture.

8. The method according to claim 1, characterized in that, In step 3, the decoding process of the decoder includes... Layer iterative update, for the th Layer, hidden state The update steps are as follows: Firstly, through the cross-attention mechanism With point cloud feature vectors Interact with the target surface points to establish an implicit correspondence between them: ； Following this, through the self-attention mechanism Integrating global body structure constraints within the query vector yields the next layer. Hidden state of layers : ； After multiple transformations, the final feature vector output by the decoder The regression head of the multilayer perceptron (MLP) is used to directly map the data into M three-dimensional spaces. Coordinates of densely marked points in ,Right now: 。 9. The method according to claim 1, characterized in that, The fitting method is a "prediction + lightweight optimization" process. First, the coordinates of the marker points are obtained through a dense marker point predictor that includes a feature extraction module and a marker point prediction module. Then, the SMPL-X model parameters are optimized based on the marker point distance to achieve high-precision fitting of the input human body.

10. The method according to any one of claims 1 to 9, characterized in that, The scope of application includes, but is not limited to, any of the following application scenarios: Automated binding of film and game assets; Human posture analysis in medical rehabilitation; Real-time virtual avatar driving in AR / VR; Post-processing and binding of 3D models generated by AIGC.

Images