Physically Constrained Three-Dimensional Complete Human Reconstruction Method

Abstract

This application relates to the field of computer vision technology, and particularly to a method for reconstructing a complete 3D human body based on physical constraints. The method includes the following steps: performing multi-subject recognition on the input image, and obtaining initial coarse SMPL-X parameters based on the recognition results; employing a diffusion method that integrates spatial information and physical constraints to refine the initial coarse SMPL-X parameters through distributed reshaping, and performing 3D human body reconstruction to obtain optimized SMPL-X parameters; using a face regression network to reconstruct the face region and obtain facial parameters; using a Transformer-based regression model to predict hand parameters, and combining adversarial loss to guide hand reconstruction in occluded scenarios, obtaining hand parameters; integrating the optimized SMPL-X parameters, facial parameters, and hand parameters to obtain refined and realistic facial expressions. This application reconstructs a complete human body model from a single RGB image, and through physical pose memory and physical constraints, achieves the generation of feasible and reasonable human poses in real-world environments.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method for reconstructing a complete three-dimensional human body based on physical constraints. Background Technology

[0002] In recent years, 3D human body modeling based on single images has become a highly promising research direction, with broad application prospects in fields such as augmented reality and human-computer interaction. Traditional methods typically rely on complex equipment such as 3D scanners, multi-view cameras, or inertial measurement units (IMUs), which, while achieving high-precision results, are limited in accessibility for everyday applications. While significant progress has been made in 3D human body reconstruction based on monocular images in recent years, most studies have focused on human posture or hand joint modeling, often neglecting facial expression modeling and physical plausibility. Facial cues play a crucial role in expressing emotions and intentions, while ignoring physical constraints can lead to unrealistic interaction effects, such as the problem of human body parts intertwining in close-contact scenarios. Summary of the Invention

[0003] This application provides a physical constraint-based method for reconstructing a complete three-dimensional human body, aiming to reconstruct a complete human body model from a single RGB image. Through physical pose memory and physical constraints, it enables the generation of feasible and reasonable human body poses in a real environment.

[0004] To address the aforementioned technical problems, this application provides a method for reconstructing a complete 3D human body based on physical constraints, comprising the following steps: First, multi-subject recognition is performed on the input image, and based on the recognition results, initial coarse SMPL-X parameters are obtained; then, a diffusion method fusing spatial information and physical constraints is used to refine the initial coarse SMPL-X parameters through distribution reshaping, and 3D human body reconstruction is performed to obtain optimized SMPL-X parameters; next, a face regression network based on expression calibration is used to reconstruct the face region to obtain facial parameters; then, a Transformer-based regression model is used to predict hand parameters, and adversarial loss is combined to guide hand reconstruction in occluded scenarios to obtain hand parameters; finally, the optimized SMPL-X parameters, facial parameters, and hand parameters are integrated to obtain refined and realistic facial expressions.

[0005] In some exemplary embodiments, multi-subject recognition is performed on the input image, and based on the recognition results, initial coarse SMPL-X parameters are obtained, including: for the input image, determining whether it is a single-person or two-person configuration using the multi-subject recognition module; and using the CLIP-Guided Fine-Tuning for Full-Body Human Mesh Recovery (CLIFF) method as an initial estimator to predict the initial coarse SMPL-X parameters. ;in, Indicates the global orientation. Describe human body shape. Indicates the posture of the torso and limbs. These are parameters for a low-perspective camera.

[0006] In some exemplary embodiments, a diffusion method that integrates spatial information and physical constraints is employed to refine the initial coarse parameters of SMPL-X through distributed reshaping. This includes: modeling human reconstruction as an iterative denoising process using a diffusion method that integrates spatial information and physical constraints to generate human pose results that satisfy contact constraints and maintain consistent relationships; integrating multiple synergistic physical perception cues during the denoising process to achieve fine adjustment of overall human parameters; wherein, the physical perception cues include: tokenized kinematic memory based on vector quantization, which encodes the coarse parameters of SMPL-X into a discrete codebook representing reasonable poses; a geometric gap regularization term used to constrain non-intersecting gaps in self-collision or multi-human interactions, which penalizes the overlap between itself or multiple human meshes to ensure the minimum non-intersecting gap constraint; and a reprojection consistency constraint, which uses two-dimensional projection error to keep the SMPL-X model aligned with the detected image features. In some exemplary embodiments, in a single-person scenario, tagged kinematic memory is used to capture individual motion patterns, and GCR is used to prevent human body meshes from intersecting; in a close-range two-person interaction scenario, by introducing paired flow couplers, combined with mutual influence layers and relational tagged memory, the spatial distance and contact relationship between human bodies are modeled, and GCR is used to optimize the gap constraints between the two human body meshes.

[0007] In some exemplary embodiments, tokenized kinematic memory compresses continuous parameters into a discrete token space and performs nearest neighbor token lookup and decoding reconstruction to gradually align noisy states to realistic human poses. In close interaction scenarios between two people, tokenized kinematic memory is used to decompose parameters into self-tags and relational tags to enhance the ability to express interpersonal dynamic relationships.

[0008] In some exemplary embodiments, a face regression network based on expression calibration is used to reconstruct the face region, including: selecting multiple facial key points as references and fitting the facial key points to the original RGB image; expanding the bounding box to cover the entire face region; cropping and normalizing the expanded face region; and feeding the processed parameters as input to the face reconstruction module. The processed parameters are represented as follows: ,in, This indicates a cropped image containing only the head region. The overall orientation of the face. Indicates the rotation angle of the facial joints. PCA parameters representing facial expressions, For face camera parameters, and These are albedo and illuminance, respectively.

[0009] In some exemplary embodiments, the face regression network consists of two parts; the first part comprises an encoder and a decoder; the encoder uses ResNet50 to extract 2048-dimensional features from the input image; the decoder consists of several fully connected layers for regressing head parameters, including expression, pose, shape, and camera parameters; the regressed parameters are input into the FLAME model to generate corresponding 3D face mesh vertex coordinates and 3D keypoints; the second part is used for further calibration of the expression; and the face image is obtained through differentiable rendering based on the 3D face mesh generated by the first part. Specifically, the reconstructed 3D face mesh and texture are rendered together using differentiable methods to generate a new facial image. By minimizing the difference between the rendered image and the input image The pixel-level differences between the images make the rendered result approximate the original image.

[0010] In some exemplary embodiments, the Transformer-based regression model uses VisionTransformer (ViT) as the backbone network, with a Transformer head following ViT to regress hand and camera parameters. The Transformer-based regression model predicts hand parameters and incorporates adversarial loss to guide hand reconstruction in occluded scenarios. The process for obtaining hand parameters includes: dividing the input RGB image into several image patches and feeding them as input labels into the ViT network; ViT performing feature modeling on each image patch and outputting a series of labeled representations; the Transformer head, acting as a Transformer decoder, processing these labels and capturing the relationships between different image patches through a cross-attention mechanism to generate the final regression result; and the network output including the pose parameters of the input hand. and shape parameters .

[0011] In some exemplary embodiments, during the hand reconstruction process, an adversarial loss term is introduced on the basis of the two-dimensional and three-dimensional supervision loss; the model is simultaneously supervised by two-dimensional and three-dimensional supervision, and three-dimensional supervision is applied on the joint space X* to ensure the consistency of the prediction results in the real three-dimensional space; at the same time, a two-dimensional projection loss is introduced between the projection of the three-dimensional joint x and the corresponding two-dimensional key point annotation x*, thereby enhancing the consistency of the model in the image space.

[0012] In some exemplary embodiments, integrating the optimized SMPL-X parameters, the facial parameters, and the hand parameters includes: integrating the optimized SMPL-X parameters, the facial parameters, and the hand parameters into a unified SMPL-X human representation; the unified SMPL-X human representation includes an SMPL-X model and a FLAME model; generating two-dimensional human joints, two-dimensional hand joints, and two-dimensional facial feature points respectively; using the optimized SMPL-X parameters, the facial parameters, and the hand parameters as initialization, and using the generated two-dimensional human joints, two-dimensional hand joints, and two-dimensional facial feature points as supervision signals, jointly optimizing the overall human model; and fusing the displacement map generated by the face regression model with the optimized overall human model to obtain refined and realistic facial expressions.

[0013] The technical solution provided in this application has at least the following advantages: This application provides a method for reconstructing a complete 3D human body based on physical constraints, comprising the following steps: First, multi-subject recognition is performed on the input image, and based on the recognition results, initial coarse SMPL-X parameters are obtained; then, a diffusion method that integrates spatial information and physical constraints is used to refine the initial coarse SMPL-X parameters through distribution reshaping, and 3D human body reconstruction is performed to obtain optimized SMPL-X parameters; next, a face regression network based on expression calibration is used to reconstruct the face region to obtain facial parameters; then, a Transformer-based regression model is used to predict hand parameters, and adversarial loss is combined to guide hand reconstruction in occluded scenarios to obtain hand parameters; finally, the optimized SMPL-X parameters, facial parameters, and hand parameters are integrated to obtain a refined and realistic facial expression.

[0014] This application provides a physical constraint-based 3D complete human body reconstruction method. For the reconstruction of the main human body, this application employs a diffusion method that integrates spatial information and physical constraints, refining the initial coarse SMPL-X parameters through distributed reshaping. This method introduces vector-quantized tokenized kinematic memory, a geometric gap regularization term to constrain non-interlacing gaps in self-collision or multi-human interactions, and reprojection consistency constraints. This method can generate human body reconstruction results that conform to contact constraints and maintain consistent relationships in both single-person self-occlusion scenarios and close-range two-person interaction scenarios with social occlusion. For hand restoration, this application uses a Transformer-based regression network to estimate hand model parameters and incorporates adversarial losses to address occlusion caused by interactions, achieving robust 3D joint alignment under 2D / 3D supervision. Facial reconstruction uses a network based on expression calibration and incorporates gender recognition information to guide the final SMPL-X mapping, thereby ensuring the rationality of the human anatomical structure. Attached Figure Description

[0015] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments, and unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0016] Figure 1 This is a flowchart illustrating a method for reconstructing a three-dimensional complete human body based on physical constraints, as provided in an embodiment of this application.

[0017] Figure 2 This is a schematic diagram of the overall framework of a physical constraint-based three-dimensional complete human body reconstruction method provided in an embodiment of this application.

[0018] Figure 3 This is a schematic diagram of the overall integration framework provided for an embodiment of this application. Detailed Implementation

[0019] As the background technology indicates, human parameter regression methods that rely solely on images face significant challenges in scenarios involving physical interactions, as occlusion issues severely impact the accuracy of human body reconstruction. Furthermore, most existing methods typically employ independent modules to regress different human body parts separately, lacking a unified processing flow that organically integrates these components.

[0020] Parametric models are widely used in computer animation for 3D human reconstruction and 3D pose estimation. Among these, the most commonly used model is based on Linear Blend Skinning (LBS), a vertex-based skinning method used to characterize the relationship between mesh vertices and the underlying skeletal structure. The basic idea of ​​LBS is to transform vertices in a single mesh through a linear combination of multiple transformations. However, a major limitation of this type of model is that it often produces unrealistic deformations at joints.

[0021] To facilitate the generation of realistic animated human bodies, a related technique proposes the Skinned Multi-PersonLinear (SMPL) model. This model learns to characterize the human body shape and its shape deformation as it changes pose. SMPL is a vertex-based skinning method that utilizes corrective blend shapes. It is based on a template mesh containing 6890 vertices and 23 joints, combined with a set of blend weights, blend shape functions, joint position prediction functions, and pose-dependent blend shape functions. The model is primarily driven by these parameters and functions, but it does not model facial and hand details.

[0022] Another related technology further incorporates articulated hands and an expressive face into SMPL, generating a unified 3D human model. Specifically, SMPL first merges with the FLAME (FacesLearned with an Articulated Model and Expressions) head model and the MANO (handModel with Articulated and Non-rigid deOrmations) hand model. Then, it learns the relationship between model parameters and shape, pose, and expression by aligning with 3D scan data. The SMPL-X model contains 10,475 vertices and 54 joints. Both SMPL and SMPL-X are compatible with graphics software and rendering engines that use standard skinning methods. This technology uses SMPL-X as the parametric human model.

[0023] 3D human body estimation mainly includes human pose, gestures, and facial expressions. Although numerous methods have been proposed for human body reconstruction or motion capture, most neglect gesture or facial expression modeling. Existing human motion capture techniques typically focus only on the human body as a whole, or model gestures or facial expressions separately, while all three are equally important in the overall representation of a human. Current 3D human body reconstruction methods can be broadly divided into optimization-based methods and regression-based methods. The former obtains 3D human body estimation results by fitting a 3D human body model to 2D keypoints in an image; the latter directly regresses the parameters of the parameterized human body model, or uses non-parametric methods (such as directly regressing vertex coordinates).

[0024] In recent years, deep neural networks (DNNs) have been widely used for parameter regression in 3D human body models. Another related technique proposes a method to fit SMPL-X models to 2D features of the human body, hands, and face. This technique introduces a mixture-of-experts rotational prior for joint angle regression. Exemplar Fine-Tuning (EFT) estimates 3D joints by reparameterizing them into network functions while utilizing 2D joint and RGB image information. Expressive Pose and Shape rEgression (ExPose) directly regresses human body, hand, and face parameters in SMPL-X format using a body-driven attention mechanism.

[0025] Due to the lack of complete human annotated data, some methods choose to predict different human body parts separately and then integrate them to achieve a holistic reconstruction of the body, hands, and face. FrankMocap uses two independent network modules to predict human pose and gestures conforming to the SMPL-X model, and then uses an ensemble module to fuse the hand and body results to obtain a complete 3D human model. However, this method does not model facial expressions.

[0026] 3D face reconstruction is a long-standing research topic in computer vision. Currently, monocular image-based face reconstruction methods typically employ parametric 3D face models, among which the 3D Morphable Model (3DMM) is widely used in tasks such as face recognition, face image analysis, expression transfer, and face animation. The face is represented as a triangular mesh with 53,490 vertices, and newer versions introduce facial expression modeling. Furthermore, some techniques propose fusing the LSFM face model with the LYHM full-head model to achieve more complete head modeling.

[0027] Deep learning methods have also been widely applied to 3D face reconstruction. For example, one related technique uses deep neural networks to predict UV location maps from a single image for 3D face reconstruction, but such methods typically only reconstruct tightly cropped face regions. Another related technique merges FaceWarehouse with BFM2009 to construct a new parametric face model; yet another technique uses a coarse encoder to regress 3DMM parameters and introduces displacement maps in UV space, thus forming a self-supervised learning framework. The FLAME (Faces Learned with an Articulated Model and Expressions) head model covers the face, the complete head, and the neck region, containing a total of 5023 vertices and 4 joints (neck, jaw, and eyeballs). This model extends the modeling ideas of SMPL to the head, introducing expression blending shapes on top of shape and pose modeling.

[0028] Traditional methods typically rely on complex equipment such as 3D scanners, multi-view cameras, or inertial measurement units (IMUs). While these methods can achieve high accuracy, their accessibility in everyday applications is limited. In recent years, significant progress has been made in 3D human reconstruction based on monocular images, but most studies focus on human posture or hand joint modeling, often neglecting facial expression modeling and physical plausibility. Facial cues play a crucial role in expressing emotions and intentions, and ignoring physical constraints can lead to unrealistic interaction effects, such as the problem of human body parts intertwining in close-contact scenarios.

[0029] To address the aforementioned technical problems, this application provides a method for 3D complete human body reconstruction based on physical constraints, comprising the following steps: First, multi-subject recognition is performed on the input image, and based on the recognition results, initial coarse SMPL-X parameters are obtained; then, a diffusion method fusing spatial information and physical constraints is used to refine the initial coarse SMPL-X parameters through distributed reshaping, and 3D human body reconstruction is performed to obtain optimized SMPL-X parameters; next, a face regression network based on expression calibration is used to reconstruct the face region to obtain facial parameters; then, a Transformer-based regression model is used to predict hand parameters, and adversarial loss is combined to guide hand reconstruction in occluded scenarios to obtain hand parameters; finally, the optimized SMPL-X parameters, facial parameters, and hand parameters are integrated to obtain refined and realistic facial expressions. This application proposes a physical constraint-based whole human body mesh reconstruction framework. The framework consists of three modules specifically designed for reconstructing the body, hands, and face, and incorporates an integration module that merges the results of each part into a complete and consistent human model. It aims to reconstruct a complete human model from a single RGB image and generate feasible and reasonable human poses in real-world environments through physical pose memory and physical constraints.

[0030] The embodiments of this application will now be described in detail with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the embodiments of this application to facilitate a better understanding of the application. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.

[0031] refer to Figure 1 This application provides a method for reconstructing a three-dimensional complete human body based on physical constraints, including the following steps: Step S1: Perform multi-subject recognition on the input image, and obtain the initial coarse parameters of SMPL-X based on the recognition results.

[0032] Step S2: Using a diffusion method that integrates spatial information and physical constraints, the initial coarse SMPL-X parameters are refined through distributed reshaping, and three-dimensional human body reconstruction is performed to obtain optimized SMPL-X parameters.

[0033] Step S3: Use a face regression network based on expression calibration to reconstruct the face region and obtain facial parameters.

[0034] Step S4: Use a Transformer-based regression model to predict hand parameters, and combine it with adversarial loss to guide hand reconstruction in occluded scenarios to obtain hand parameters.

[0035] Step S5: Integrate the optimized SMPL-X parameters, facial parameters, and hand parameters to obtain refined and realistic facial expressions.

[0036] The framework of this application is based on the Skinned Multi-Person Linear Model eXpressive (SMPL-X) parametric model, which uniformly represents human body shape, pose, hand joints, and facial expressions in a single mesh. The overall process adopts an "analysis-synthesis" paradigm: first, the human body, hand, and facial regions are cropped using detectors, then parameter regression is performed separately through dedicated modules, and finally, a unified SMPL-X human body mesh is synthesized.

[0037] The goal of this application is to achieve a holistic 3D human body model with physical perception capabilities from a single monocular image, encompassing the shape and posture of the human body, hand gestures, and facial expressions. The overall framework of the proposed method is as follows: Figure 2 As shown, the specific details are as follows.

[0038] First, perform three-dimensional human body reconstruction.

[0039] The physically-aware SMPL-X framework in this application aims to meet the need for reliable human body mesh recovery in complex scenes. This application proposes a physically-constrained human body representation method with spatial awareness capabilities, supporting both single-person and close-range two-person interaction scenarios based on SMPL-X. Given an input image, a multi-subject recognition module first determines whether it is a single-person or two-person configuration, and then a vision-guided estimator predicts a coarse set of SMPL-X parameter distributions. ;in, Indicates the global orientation. Describe human body shape. Indicates the posture of the torso and limbs. These are parameters for a low-perspective camera.

[0040] To establish an initial baseline for SMPL-X parameter estimation, this application employs the CLIP-Guided Fine-Tuning for Full-Body Human Mesh Recovery (CLIFF) method as the initial estimator. While this method provides reasonable initial estimates, its performance is limited in interactive scenarios with occlusion and depth uncertainty. Therefore, this application introduces a diffusion method that integrates spatial and physical information. This method progressively refines the initial estimate through distribution reshaping, modeling human reconstruction as an iterative denoising process to generate human pose results that satisfy contact constraints and maintain consistent relationships. During the inference phase, backdiffusion performs only a finite number of steps, gradually removing noise. The accurate SMPL-X parameters are restored, and a contact-aware correction mechanism is introduced in the end-to-end optimization to handle physical constraints.

[0041] The diffusion refinement process integrates three synergistic physical perception cues during the denoising process to achieve fine adjustment of overall human body parameters: (1) Tokenized kinematic memory based on vector quantization, which encodes SMPL-X parameters into discrete codebooks representing reasonable postures; (2) Geometric gap regularization, which penalizes the overlap between itself or multiple human body meshes to ensure non-intersecting minimum gap constraints; and (3) Reprojection consistency constraints, which keep the model aligned with the detected image features through two-dimensional projection errors.

[0042] In single-player scenarios, tokenized kinematic memory is used to capture individual motion patterns, while GCR (Geometric Response Code) prevents mesh overlap between the human body and other parts of the body. In close-range two-player interaction scenarios, paired flow couplers are introduced, combined with mutual influence layers and relational tokenized memory, to model the spatial distance and contact relationships between the human bodies. In this case, GCR is used to optimize the gap constraints between the two human body meshes. This spatial modeling foundation guides a Transformer-based diffusion refinement module to update human body parameters in a socially reasonable and physically feasible manner. When the scene degenerates to single-player mode or the distance between characters is large, the second branch is closed, and the system automatically switches to single-player mode.

[0043] Tokenized kinematic memory compresses continuous parameters into a discrete token space and performs nearest neighbor token lookup and decoding reconstruction, gradually aligning noisy states to realistic human poses. In close two-person interaction scenarios, this mechanism further decomposes parameters into ego tokens and relational tokens to enhance the ability to express dynamic interpersonal relationships.

[0044] The overall optimization objective function consists of calibration loss, label commitment loss, diffusion loss, and relationship coupling loss.

[0045] in, and These are the weighting coefficients for each loss term. The calibration loss, used to monitor SMPL-X parameters and intermediate geometric quantities, is defined as follows:

[0046]

[0047] Where T represents the number of time steps, These represent the pose and shape parameters of the b-th individual, respectively. (Indicates true annotation), J( ) and V( These are functions mapped to 3D joints and vertices, respectively. These are the corresponding weighting coefficients.

[0048] The labeled commitment loss adopts a structured commitment mechanism, which is defined as follows:

[0049] in, This represents the encoder output of the input x (normalized SMPL-X pose and shape parameters), where e is the nearest neighbor reference vector. The gradient stopping operator is used to decouple the encoder from the tag library updates during backpropagation. To label commitment weights.

[0050] The diffusion loss employs a score-matching-based diffusion objective function, imposing constraints on both branches simultaneously under a pairwise flow configuration: ,in, This represents the time step t, the true SMPL-X parameter x0, and the noise. Expectations For the score function estimator, This represents the perturbation data at step t. These are the diffusion scheduling parameters.

[0051] To address the physical interleaving problem, this application penalizes it using a minimum distance constraint:

[0052]

[0053] Where V represents the set of vertices of a single SMPL-X mesh, These represent the vertex sets of two SMPL-X meshes. Let t be the position of the corresponding vertex at time step t. The safety distance threshold is used to prevent oneself from weaving in or to ensure a reasonable contact or distance between two people.

[0054] In the experiment, this application set , After reconstruction, the generated SMPL-X human model is fitted to the original RGB image, and the correspondence between the model and the aligned joints is established.

[0055] Next, we will perform 3D face reconstruction and 3D hand reconstruction.

[0056] To achieve accurate face reconstruction, this application proposes a face reconstruction module. First, the face region is identified and detected during the body reconstruction stage. Seventeen facial key points are selected as references and fitted onto the original RGB image. The bounding boxes are then expanded to cover the entire face region. Next, the expanded face region is cropped and normalized, and then fed as input into the face reconstruction module. Its formal expression is as follows: ,in, This indicates a cropped image containing only the head region. The overall orientation of the face. Indicates the rotation angle of the facial joints. PCA parameters representing facial expressions, For face camera parameters, and These are albedo and illuminance, respectively.

[0057] This application focuses on face regression networks. It consists of two main parts. The first part consists of an encoder and a decoder: the encoder uses ResNet50 to extract 2048-dimensional features from the input image; the decoder consists of several fully connected layers used to regress head parameters (including expression, pose, shape, and camera parameters). The regressed parameters are input into the FLAME model to generate the corresponding 3D face mesh vertex coordinates and 3D keypoints.

[0058] In the supervised face restoration process, this application introduces a keypoint loss, using L1 distance to measure the difference between 2D face keypoints detected by OpenPose and 3D keypoints generated by the FLAME model and projected onto a 2D plane. Specifically, face parameters are input into the FLAME model to generate a 3D face geometry, and its keypoints are projected into 2D space under the influence of camera parameters, allowing for direct comparison with 2D keypoints in the original image. The corresponding loss function is defined as follows: , The second part is used for further calibration of facial expressions, based on facial images obtained by differentiable rendering of the 3D face mesh generated in the first stage. Specifically, the reconstructed 3D face mesh and texture are rendered together using differentiable rendering to generate a new facial image. By minimizing the difference between the rendered image and the input image. The pixel-level differences between the two make the rendered result approximate the original image; its pixel loss is defined as... , where M represents the segmentation mask, used to retain only the skin and nose regions in the face image, thereby avoiding interference from irrelevant regions in the optimization process.

[0059] In terms of 3D hand reconstruction, this application employs a Transformer-based regression model to predict hand parameters and introduces adversarial loss to guide hand reconstruction in occluded scenes. Specifically, this application uses VisionTransformer (ViT) as the backbone network and connects a Transformer head after it to regress hand and camera parameters.

[0060] First, the input RGB image is divided into several patches, which are then fed into the ViT network as input tokens. ViT models the features of each patch and outputs a series of token representations. Subsequently, the Transformer head, implemented as a Transformer decoder, processes these tokens and captures the relationships between different patches through a cross-attention mechanism, thereby generating the final regression result. The network output includes the pose parameters of the input hand. and shape parameters .

[0061] To address the occlusion issue arising during hand interactions, this application introduces an adversarial loss term in addition to the existing 2D and 3D supervised losses. The model receives both 2D and 3D supervision, with 3D supervision applied to the joint space X* to ensure consistency of prediction results in the real 3D space. Furthermore, to more effectively utilize 2D labeled data for training, this application introduces a 2D projection loss between the projection of the 3D joint x* and the corresponding 2D keypoint annotation x*, thereby enhancing the model's consistency in image space.

[0062] To address the occlusion problem caused by hand physical interaction (which often leads to inaccurate parameter regression), this application trains a discriminator. This is used to perform discriminative regression on the hand's shape parameters, posture parameters, and various joint angles, respectively. In fact, in addition to the aforementioned key loss functions, MANO loss is also introduced during training to further constrain the rationality of hand deformation.

[0063] The corresponding loss function is defined as follows:

[0064]

[0065]

[0066] The overall loss function is defined as The weighting coefficients are set as follows:

[0067] Finally, this application uses multiple two-dimensional and three-dimensional hand annotation datasets to train the model.

[0068] Next, we will perform module integration. The overall integration framework is as follows: Figure 3 As shown.

[0069] After predicting the body, hands, and face separately, this application integrates the outputs of the three reconstruction modules into a unified SMPL-X human representation. Specifically, each module first predicts the parameters of the human body, hands, and face, and then inputs these parameters into the SMPL-X and FLAME models. It's important to note that the hand regression module outputs MANO parameters, so parameter transformation is required before inputting them into the SMPL-X model. Subsequently, based on the respective camera parameters, two-dimensional human joints, two-dimensional hand joints, and two-dimensional facial feature points are generated. Using the parameters predicted by each module as initialization, and the two-dimensional joints and facial key points as supervision signals, the overall human model is jointly optimized. Finally, the displacement map generated by the face module is fused with the optimized overall human model to obtain more refined and realistic facial expressions.

[0070] Compared with existing technologies, the physical constraint-based three-dimensional complete human body reconstruction method provided in this application has the following advantages: This application introduces physical constraints into the body and hand reconstruction modules, enabling them to maintain robust performance under complex physical interaction conditions. Furthermore, experimental results demonstrate that the proposed method achieves state-of-the-art performance in both qualitative and quantitative evaluations, outperforming existing methods in both pose accuracy and physical plausibility. Extensive experiments on multiple datasets, including NoW, 3DPW, Hi4D, FreiHAND, HO3D, and EHF, show that the proposed method achieves state-of-the-art performance, not only better characterizing facial expressions but also maintaining good robustness under physical interaction occlusion.

[0071] This application verifies the effectiveness of its method through extensive experiments, and the results are shown in the table below. Quantitative comparisons of different methods on the NoW face dataset are shown in Table 1, with reported metrics including median, mean, and standardized vertex-to-vertex error.

[0072] Table 1 shows the quantitative comparison results of different methods on the NoW face dataset.

[0073] Table 2 shows the quantitative comparison results of different methods on the 3DPW dataset. PA-MPJPE and MPJPE represent the 3D joint error in millimeters (mm).

[0074] Table 2 shows the quantitative comparison results of different methods on the 3DPW dataset.

[0075] The results on the HI4D (BODY) dataset are shown in Table 3, with the error unit being millimeters (mm).

[0076] Table 3 shows the results on the HI4D (BODY) dataset.

[0077] The results on the FREIHAND (HAND) dataset are shown in Table 4, with the error unit being millimeters (mm).

[0078] Table 4 shows the results on the FreiHAND (HAND) dataset.

[0079] The results for the HO3D (HAND) dataset are shown in Table 5, with the error unit being millimeters (mm).

[0080] Table 5 shows the results on the HO3D (HAND) dataset.

[0081] Table 6 shows the comparison results of vertex errors of different methods on the EHF dataset, where PA-V2V represents the vertex error in millimeters (mm).

[0082] Table 6 shows the comparison of vertex errors of different methods on the EHF dataset.

[0083] Based on the above technical solutions, this application provides a method for three-dimensional complete human body reconstruction based on physical constraints, including the following steps: First, multi-subject recognition is performed on the input image, and based on the recognition results, initial coarse SMPL-X parameters are obtained; then, a diffusion method that integrates spatial information and physical constraints is used to refine the initial coarse SMPL-X parameters through distribution reshaping, and three-dimensional human body reconstruction is performed to obtain optimized SMPL-X parameters; next, a face regression network based on expression calibration is used to reconstruct the face region to obtain facial parameters; then, a Transformer-based regression model is used to predict hand parameters, and adversarial loss is combined to guide hand reconstruction in occluded scenarios to obtain hand parameters; finally, the optimized SMPL-X parameters, facial parameters, and hand parameters are integrated to obtain refined and realistic facial expressions.

[0084] This application provides a physical constraint-based 3D complete human body reconstruction method. For the reconstruction of the main human body, this application employs a diffusion method that integrates spatial information and physical constraints, refining the initial coarse SMPL-X parameters through distributed reshaping. This method introduces vector-quantized tokenized kinematic memory, a geometric gap regularization term to constrain non-interlacing gaps in self-collision or multi-human interactions, and reprojection consistency constraints. This method can generate human body reconstruction results that conform to contact constraints and maintain consistent relationships in both single-person self-occlusion scenarios and close-range two-person interaction scenarios with social occlusion. For hand restoration, this application uses a Transformer-based regression network to estimate hand model parameters and incorporates adversarial losses to address occlusion caused by interactions, achieving robust 3D joint alignment under 2D / 3D supervision. Facial reconstruction uses a network based on expression calibration and incorporates gender recognition information to guide the final SMPL-X mapping, thereby ensuring the rationality of the human anatomical structure.

[0085] Those skilled in the art will understand that the above-described embodiments are specific examples of implementing this application, and in practical applications, various changes in form and detail may be made without departing from the spirit and scope of this application. Any person skilled in the art can make their own modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application should be determined by the scope defined in the claims.

Claims

1. A method for reconstructing a complete three-dimensional human body based on physical constraints, characterized in that, Includes the following steps: Multi-subject recognition is performed on the input image, and based on the recognition results, the initial coarse parameters of SMPL-X are obtained; A diffusion method that integrates spatial information and physical constraints is adopted to refine the initial coarse SMPL-X parameters through distributed reshaping, and then perform 3D human body reconstruction to obtain optimized SMPL-X parameters. A face regression network based on expression calibration is used to reconstruct the face region and obtain facial parameters; A Transformer-based regression model is used to predict hand parameters, and adversarial loss is combined to guide hand reconstruction in occluded scenarios to obtain hand parameters. By integrating the optimized SMPL-X parameters, the facial parameters, and the hand parameters, a refined and realistic facial expression is obtained.

2. The method for reconstructing a three-dimensional complete human body based on physical constraints according to claim 1, characterized in that, Multi-subject recognition is performed on the input image, and based on the recognition results, initial coarse parameters of SMPL-X are obtained, including: For the input image, the multi-subject recognition module determines whether it is a single person or a two-person configuration; The CLIP-Guided Fine-Tuning for Full-Body Human Mesh Recovery (CLIFF) method was used as the initial estimator to predict the initial coarse parameters of SMPL-X. ; in, Indicates the global orientation. Describe human body shape. Indicates the posture of the torso and limbs. These are parameters for a low-perspective camera.

3. The method for reconstructing a three-dimensional complete human body based on physical constraints according to claim 1, characterized in that, A diffusion method integrating spatial information and physical constraints is employed to refine the initial coarse parameters of SMPL-X through distribution reshaping, including: A diffusion method that integrates spatial information and physical constraints is adopted to model human body reconstruction as an iterative denoising process, generating human body posture results that satisfy contact constraints and have consistent relationships. Multiple synergistic physical perception cues are integrated during the noise reduction process to achieve fine-tuning of overall human body parameters; these physical perception cues include: Based on vector quantization, the tokenized kinematic memory encodes the coarse parameters of SMPL-X into a discrete codebook representing reasonable poses; Geometric gap regularization terms are used to constrain non-intersecting gaps in self-collision or multi-human interaction. By penalizing the overlap between the meshes of itself or multiple human beings, the minimum spacing constraint for non-intersecting is guaranteed. The reprojection consistency constraint uses two-dimensional projection error to keep the SMPL-X model aligned with the detected image features.

4. The method for reconstructing a complete three-dimensional human body based on physical constraints according to claim 3, characterized in that, In single-person scenarios, tokenized kinematic memory is used to capture individual movement patterns, and GCR is used to prevent the human body's own mesh from intersecting. In close-range two-person interaction scenarios, by introducing paired flow couplers and combining mutual influence layers and relational marker memory, the spatial distance and contact relationship between human bodies are modeled. At this time, GCR is used to optimize the gap constraints between the two human body meshes.

5. The method for reconstructing a three-dimensional complete human body based on physical constraints according to claim 3, characterized in that, Tokenized kinematic memory compresses continuous parameters into a discrete token space and performs nearest neighbor token lookup and decoding reconstruction, gradually aligning noisy states to realistic human poses. In close-knit two-person interaction scenarios, labeled kinematic memory is used to decompose parameters into self-labels and relational labels to enhance the ability to express interpersonal dynamics.

6. The method for reconstructing a three-dimensional complete human body based on physical constraints according to claim 1, characterized in that, A face regression network based on expression calibration is used to reconstruct the face region, including: Multiple facial key points are selected as references, and the facial key points are fitted to the original RGB image. The bounding box is then expanded to cover the entire facial area. The expanded face region is cropped and normalized, and the processed parameters are fed into the face reconstruction module as input. The processed parameters are represented as follows: ,in, This indicates a cropped image containing only the head region. The overall orientation of the face. Indicates the rotation angle of the facial joints. PCA parameters representing facial expressions For face camera parameters, and These are albedo and illuminance, respectively.

7. The method for reconstructing a three-dimensional complete human body based on physical constraints according to claim 6, characterized in that, The face regression network consists of two parts; The first part consists of an encoder and a decoder; the encoder uses ResNet50 to extract 2048-dimensional features from the input image; The decoder consists of several fully connected layers used to regress head parameters, including expression, pose, shape, and camera parameters. The regressed parameters are input into the FLAME model to generate the corresponding 3D face mesh vertex coordinates and 3D key points. The second part is used for further calibration of facial expressions; the facial image is obtained through differentiable rendering based on the 3D face mesh generated in the first part. Specifically, the reconstructed 3D face mesh and texture are rendered together using differentiable methods to generate a new facial image. ; By minimizing the difference between the rendered image and the input image The pixel-level differences between the images make the rendered result approximate the original image.

8. The method for reconstructing a three-dimensional complete human body based on physical constraints according to claim 1, characterized in that, The Transformer-based regression model uses the Vision Transformer (ViT) as the backbone network and connects a Transformer head after the ViT to regress hand parameters and camera parameters. A Transformer-based regression model is used to predict hand parameters, and adversarial loss is combined to guide hand reconstruction in occluded scenarios. The resulting hand parameters include: The input RGB image is divided into several image blocks, and these blocks are used as input labels and fed into the ViT network. After performing feature modeling on each image patch, ViT outputs a series of labeled representations; The Transformer head, implemented as a Transformer decoder, processes these markers and captures the relationships between different image patches through a cross-attention mechanism, thereby generating the final regression result; the network output includes the pose parameters of the input hand. and shape parameters .

9. The method for reconstructing a three-dimensional complete human body based on physical constraints according to claim 8, characterized in that, In the hand reconstruction process, an adversarial loss term is introduced on the basis of 2D and 3D supervised loss. The model is simultaneously supervised by 2D and 3D, and 3D supervision is applied on the joint space X* to ensure the consistency of the prediction results in the real 3D space. At the same time, a 2D projection loss is introduced between the projection of the 3D joint x and the corresponding 2D key point annotation x*, thereby enhancing the consistency of the model in the image space.

10. The method for reconstructing a three-dimensional complete human body based on physical constraints according to claim 1, characterized in that, Integrating the optimized SMPL-X parameters, the facial parameters, and the hand parameters includes: The optimized SMPL-X parameters, facial parameters, and hand parameters are integrated into a unified SMPL-X human body representation; the unified SMPL-X human body representation includes the SMPL-X model and the FLAME model. Two-dimensional human joints, two-dimensional hand joints, and two-dimensional facial feature points are generated respectively. Using the optimized SMPL-X parameters, facial parameters, and hand parameters as initialization, and the generated two-dimensional human joints, two-dimensional hand joints, and two-dimensional facial feature points as supervision signals, the overall human model is jointly optimized. By fusing the displacement map generated by the face regression model with the optimized overall human body model, a detailed and realistic facial expression can be obtained.

Images