Method and system for reconstructing an animatable three-dimensional human body model based on a single occluded image

Abstract

This invention discloses a method and system for reconstructing an animated 3D human body model based on a single occluded image. First, it receives a single RGB image of a human body containing occlusion, and obtains the human body region mask, occluded region mask, and visible region mask. Then, it estimates the SMPL model parameters of the human body. Next, based on the SMPL model parameters and the visible region mask, it performs visibility-guided geometric completion to obtain a complete human body normal map. It also estimates pixel-level uncertainty maps for each generated viewpoint. Then, it obtains an initial 3D Gaussian set through a feedforward human body reconstruction model, and subsequently reconstructs a 3D Gaussian human body model. Finally, it uses skin weights to perform a rigid transformation on the Gaussian units through a linear hybrid skin transformation, and combines this with a pose-dependent appearance mapping network to predict the attribute offsets of the Gaussian units, achieving skeletal animation-driven operation. This method can simultaneously achieve complete occlusion region completion, high-quality 3D geometry and appearance reconstruction, and skeletal animation-driven operation.

Description

Technical Field

[0001] This invention relates to the field of computer vision and computer graphics, specifically to a method and system for reconstructing an animated 3D human body model based on a single occluded image. Background Technology

[0002] With the rapid development of virtual reality, augmented reality, film and television special effects, and the digital human industry, reconstructing a drivable 3D human body model from a single image has become a key requirement in the fields of computer vision and graphics. However, in real-world scenes, human body images often suffer from severe information loss due to object occlusion or limb occlusion, posing a serious challenge to the robustness, completeness, and drivability of reconstruction methods.

[0003] In existing technologies, methods based on parametric human models (such as SMPL / SMPL-X) naturally support skeletal animation, but they struggle to represent complex non-rigid details such as clothing and are prone to generating unreasonable geometric inferences in occluded areas. Methods based on implicit functions (such as the PIFu series) can recover complete geometry including clothing details, but their rendering efficiency is low, and geometric fragmentation easily occurs in occluded scenes. Methods based on Neural Radiation Fields (NeRF) typically rely on multi-view input, limiting single-image reconstruction performance. In recent years, 3D Gaussian Splashing (3DGS) has become the mainstream paradigm due to its high rendering efficiency and explicit geometric representation, but existing single-image reconstruction methods... Figure 3 DGS (Distributed Gaussian Geometry) methods perform poorly under occluded input and mostly only generate static models, not supporting animation-driven models. Diffusion model methods perform well in unoccluded scenes, but directly generating multi-view images under occluded conditions can easily lead to geometric inconsistencies and texture discrepancies.

[0004] For occlusion scenarios, some specialized solutions already exist. For example, CN111739161B requires RGB-D input and relies on a pre-built occlusion model library, has high hardware requirements, and only outputs parametric models; OAHuman can recover relatively complete 3D geometry from a single occluded image, but only outputs a static model; CHROME uses a diffusion model to generate de-occluded multi-view images before reconstruction, but lacks consistency across multiple views; U-4DGS and AHOY can output animated models, but require video sequence input and are not suitable for single images. Other technologies require additional multimodal input, limiting their application scenarios.

[0005] In summary, the existing technology has not yet provided a unified method that can simultaneously achieve complete occlusion region completion, high-quality 3D geometry and appearance reconstruction, and skeletal animation driving from a single occluded RGB image. This constitutes a technical challenge that urgently needs to be overcome in this field. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a method and system for reconstructing an animated 3D human body model based on a single occluded image. While existing methods directly utilize generative models to complete the occluded region from multiple perspectives, they lack complete geometric prior constraints, easily leading to geometric contradictions and texture inconsistencies between different perspectives. Furthermore, in the animation-driven stage, rigid skeletal deformation is coupled with non-rigid appearance changes, making it difficult to preserve facial identity information. Currently, there is no unified solution that can simultaneously achieve complete occluded region completion, high-quality 3D geometry and appearance reconstruction, and skeletal animation driving from a single occluded RGB image.

[0007] To achieve the above objectives, the technical solution specifically adopted by the present invention is as follows:

[0008] This invention provides a method for reconstructing an animated 3D human body model based on a single occluded image, comprising the following steps:

[0009] Step S1: Receive a single RGB image of a human body containing occlusion, perform human instance segmentation and occlusion detection, and obtain the human body region mask, occlusion region mask and visible region mask; use an occlusion-robust human parameter regression network to estimate the SMPL model parameters of the human body based on feature extraction under the constraint of the visible region mask. The SMPL model parameters include body shape parameters, pose parameters and camera parameters.

[0010] Step S2: Based on the SMPL model parameters and the visible region mask, perform visibility-guided geometric completion to obtain a complete human body normal map; using the completed frontal image and the multi-view pose control signal generated according to the SMPL model parameters as conditions, generate K de-occluded human body images from different viewpoints through a multi-view diffusion model, and estimate pixel-level uncertainty maps for each generated viewpoint.

[0011] Step S3: Input the occluded original image, K multi-view generated images, and the template mesh corresponding to the SMPL model into the pre-trained feedforward human body reconstruction model to obtain an initial 3D Gaussian set; guided by the uncertainty graph, perform uncertainty-weighted parameter optimization on the initial 3D Gaussian set to obtain the reconstructed 3D Gaussian human body model.

[0012] Step S4: Predict the skin weights of the corresponding SMPL skeletal joints for each Gaussian skeletal unit in the 3D Gaussian human model; in response to the target pose parameters, use the skin weights to perform a rigid transformation on the Gaussian skeletal units through a linear hybrid skin transformation, and combine a pose-dependent appearance mapping network to predict the attribute offsets of the Gaussian skeletal units to achieve skeletal animation-driven operation.

[0013] Preferably, the occlusion-robust human parameter regression network in step S1 adopts a Transformer architecture and introduces a visibility-guided attention mechanism in its self-attention calculation. This mechanism applies the visibility weight map generated from the visible region mask to the attention score, and applies a negative bias to the attention score corresponding to the occluded region, suppressing the interference of occlusion noise on parameter estimation. Step S1 also includes obtaining a vertex-level visibility map of the human body through forward kinematics rendering based on the estimated SMPL model parameters, providing geometric priors for subsequent geometric completion.

[0014] Preferably, in step S2, the visibility-guided geometric completion utilizes an encoder-decoder architecture geometric completion network. It takes the visible region image, the visible region mask, and the normal prior map rendered by SMPL as input to predict the complete frontal and back normal maps of the human body. During training, feature-level supervised regularization is introduced, using a pre-trained visual base model to extract semantic feature differences between the predicted and reference normal maps at multiple network depths for constraint. A dual-view normal refinement step is also included, where a cross-view attention module is introduced at the bottleneck layer to enable bidirectional information exchange between frontal and back normal features, outputting refined normal map pairs. The multi-view pose control signal consists of depth maps and normal maps rendered at various target viewpoints using SMPL model parameters. These are injected into a multi-view diffusion model based on a denoising diffusion architecture via ControlNet to generate de-occluded human images with K ≥ 4 different azimuth angles. The estimation of the pixel-level uncertainty map uses the heteroscedastic Gaussian noise assumption, reflecting both the inherent uncertainty of missing occlusion information and the cognitive uncertainty of the randomness of the diffusion model's sampling.

[0015] Preferably, in step S3, the feedforward human reconstruction model achieves occlusion robustness through a self-supervised occlusion adaptation framework. This framework employs a teacher-student distillation architecture, where the teacher network receives a complete, unoccluded image to generate a reference reconstruction result, and the student network receives an artificially synthesized occluded image. The student network output is constrained to approximate the teacher network output through rendering-level L1 loss and feature-level L2 loss. Uncertainty-weighted parameter optimization uses heteroscedasticity negative log-likelihood photometric loss, normalizing the rendering error of each pixel according to the uncertainty value and jointly optimizing it with the uncertainty logarithm; simultaneously, the gradient at the Gaussian center position is adaptively scaled based on uncertainty. It also includes confidence-aware regularization: introducing depth smoothing regularization loss, weighting the rendering depth map gradient according to the uncertainty adaptive weights; and increasing the splitting and cloning thresholds for Gaussian cells projected to high-uncertainty regions in Gaussian adaptive density control. Furthermore, it includes a self-calibration refinement step, using a single-step diffusion refinement model for texture enhancement. During training, the input is replaced with the original rendered image with a certain probability as an identity-preserving sample, and a hybrid loss function is used to constrain the output to not deviate from the original appearance.

[0016] Preferably, in step S4, the pose-dependent appearance mapping network is StyleUNet, which takes the SMPLUV parameterized mapping of the target pose as input and predicts the color and position shifts of Gaussian elements. A decoupling mechanism between Map-Pose and LBS-Pose is introduced, where the LBS-Pose path handles spatial transformations caused by rigid skeletal deformation, and the Map-Pose path handles non-rigid appearance changes. The outputs of the two paths are added and fused in the feature space. A facial identity preservation strategy is also included: Gaussian elements in the head region are separated from the full-body linear hybrid skin skeleton, and a FLAME parameterized head model is introduced for independent modeling. Facial identity consistency is constrained by identity similarity loss in the face recognition feature space. A geometrically faithful texture rendering constraint is also included, calculating the difference between the rendered normal map and the gradient direction normalization result of the depth map under the target pose as the geometric consistency loss.

[0017] This invention also provides an animable 3D human body model reconstruction system based on a single occluded image, comprising: an occlusion perception and parsing module for performing instance segmentation, occlusion detection, and occlusion-robust SMPL parameter estimation; a multi-view de-occlusion generation module for performing visibility-guided geometric completion, multi-view diffusion generation, and uncertainty estimation; an uncertainty-guided reconstruction module for performing feedforward Gaussian initialization, uncertainty-weighted optimization, and self-calibration refinement; and a skeletal animation driving module for performing skin weight prediction, linear blending skin deformation, pose-dependent appearance mapping, face-independent modeling, and geometric consistency rendering constraints.

[0018] The present invention also provides an electronic device, including a memory and a processor, wherein the processor performs the steps of the above method when executing a computer program in the memory.

[0019] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0020] This invention has the following characteristics and beneficial effects:

[0021] For the first time, end-to-end reconstruction from a single occluded image to an animated 3D human body model has been achieved, breaking through the technical bottleneck of existing methods that "can handle occluders but cannot drive, and can drive but not applicable to single images". Only a single RGB input is required to simultaneously complete occlusion completion and animation-ready 3D reconstruction, greatly expanding the applicable scenarios of single-image human body reconstruction.

[0022] A dual-source uncertainty joint modeling and full-process guidance mechanism is proposed. Based on the heteroscedasticity observation model, it simultaneously captures the inherent uncertainty of occlusion and the cognitive uncertainty generated by diffusion. The uncertainty information is integrated into the full optimization process of photometric loss, gradient modulation, depth regularization and density control, which significantly improves the robustness and realism of 3D reconstruction in occluded scenes.

[0023] We construct a joint strategy for geometrically decoupled occlusion completion and pose-controlled diffusion. First, we restore the complete geometric prior through feature-level supervised regularization and dual-view normal refinement. Then, we use this as a condition to constrain the multi-view diffusion generation, which effectively solves the problems of geometric contradictions and texture inconsistencies that exist when directly generating multi-view images from occluded images.

[0024] A feedforward reconstruction framework for self-supervised occlusion adaptation is developed. Through a teacher-student distillation architecture, large-scale pre-trained human reconstruction models acquire occlusion robustness without the need for additional 3D annotation. Combined with a self-calibration learning strategy, it effectively prevents identity information drift while enhancing details.

[0025] We propose a Map-Pose / LBS-Pose decoupled animable representation and face-independent modeling, which decouples the rigid components of skeletal deformation from the non-rigid components of appearance changes. Combined with the FLAME face-independent driving strategy, we achieve high-fidelity animation effects with natural clothing dynamics and preserved facial identity under pose-driven conditions, effectively suppressing texture tearing and geometric penetration. Attached Figure Description

[0026] Figure 1 This is an overall flowchart of an animated 3D human body model reconstruction method based on a single occluded image, according to an embodiment of the present invention.

[0027] Figure 2 This is a schematic diagram of the occlusion perception human body analysis and pose estimation module in an embodiment of the present invention.

[0028] Figure 3 This is a schematic diagram of the occlusion completion and multi-view consistency image generation module in an embodiment of the present invention.

[0029] Figure 4 This is a schematic diagram of the structure of the uncertainty-guided three-dimensional Gaussian splash human body reconstruction module in an embodiment of the present invention.

[0030] Figure 5 This is a schematic diagram of the structure of the skeleton binding and animation driving module in an embodiment of the present invention.

[0031] Figure 6 This is a visual schematic diagram of the uncertainty graph in an embodiment of the present invention.

[0032] Figure 7 This is a block diagram of the module structure of the system of the present invention.

[0033] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0034] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0035] Example 1

[0036] This embodiment provides a method for reconstructing an animated 3D human body model based on a single occluded image, such as... Figure 1 As shown, the method includes the following steps:

[0037] Step S1: Occlusion perception, human body analysis, and pose estimation;

[0038] This stage receives a single RGB image of a human body, including any occlusions. As input, perform human instance segmentation, occlusion detection, and occlusion-robust human parameter estimation.

[0039] 1.1 Human Instance Segmentation and Occlusion Detection

[0040] The input image is segmented into human instances using a Transformer-based instance segmentation model (preferably the Segment Anything Model series in this embodiment), outputting semantic labels and masks for each instance. By analyzing the depth ranking and spatial overlap relationships between instances, foreground objects overlapping with the human body region are marked as occluders. Specifically, the occlusion relationships are determined through the following steps:

[0041] (1) Obtain the human body region mask and all foreground object masks ;

[0042] (2) Calculate the intersection area of ​​each object mask and the human body mask. ;

[0043] (3) Use a monocular depth estimation model (such as DPT or Depth Anything series) to predict the depth of the input image and obtain a depth map. ;

[0044] (4) For objects whose intersection region is not empty, compare their average depth in the intersection region with the average depth of the human body. If the depth of the object is less than the depth of the human body (i.e. the object is closer to the camera), then the object is determined to be an obstruction.

[0045] (5) Combine the projections of all occluders onto the human body area to obtain the occlusion area mask.

[0046] and visible area mask .

[0047] 1.2 Occlusion-Robust SMPL Parameter Estimation

[0048] Based on pixel observations of the visible region, occlusion-robust human parameter regression networks are used to estimate the parameters of the SMPL model. The regression network adopts a ViT-based encoder-regressor architecture, with the encoder part using the ViT-Large model (24 Transformer blocks, 1024 hidden dimensions, and 16 attention heads) to encode the input image into a feature sequence.

[0049] In the self-attention calculation of the encoder, a visibility-guided attention mechanism is introduced. Specifically, the visible region is masked. Downsampled to the same spatial resolution as the ViT feature map to generate a visibility weight map. In each self-attention layer, the attention score is modulated using visibility weighting. This is calculated by applying the natural logarithm of the visibility weight map as a bias term to the standardized result of the dot product of the query and key divided by the square root of the dimension. The attention weights are then calculated using the softmax function and summed with the resulting values. This mechanism applies a strong negative bias to the attention score in occluded regions, causing the model to prioritize visual cues in visible areas during parameter regression.

[0050] The regressor part employs an iterative regression strategy, containing a 3-layer fully connected network (hidden dimensions 1024-512-256), and outputs SMPL body size parameters after 3 iterations of regression. Attitude parameters (Including 3D global rotation of the root joint and 3D local rotation of each of the 23 joints) and weak perspective camera parameters ,in Scaling factor These are the translation parameters.

[0051] 1.3 Vertex-level Visibility Graph Generation

[0052] Based on the estimated SMPL parameters, the human body mesh is obtained through forward kinematics calculations using the SMPL model. ,in Let F be the vertex coordinates and F be the patch index. The human body mesh is rendered to the image plane using a differentiable rasterizer. The visibility state of each vertex from the current viewpoint is determined by z-buffer depth testing, generating a vertex-level visibility map. This provides geometric priors for subsequent geometric completion.

[0053] Step S2: Occlusion completion and multi-view consistent image generation;

[0054] like Figure 6As shown, this stage performs visibility-guided geometric completion, multi-view diffusion generation, and pixel-level uncertainty estimation based on the SMPL model parameters and visible region mask obtained in stage one.

[0055] 2.1 Visibility-guided geometric completion

[0056] Predicting complete frontal normal maps of the human body using a geometric completion network with an encoder-decoder architecture. and back normal diagram The geometric completion network adopts the U-Net architecture. The encoder part is a ResNet-34 (4 residual block groups with channel numbers of 64, 128, 256, and 512 respectively), and the decoder part gradually restores the spatial resolution through bilinear upsampling and convolutional layers. The input consists of an image of the visible region (with zeros filled in for occluded regions) and a visibility mask. The normal prior map rendered by SMPL is stitched together along the channel dimension and then fed into the network to output front and back normal maps with a resolution of 512×512.

[0057] Feature-level supervised regularization is introduced during the training of the geometric completion network. Multi-layer semantic features of the predicted and reference normal maps are extracted using a pre-trained visual base model (DINOv2-ViT-Large model in this embodiment). A feature-level consistency loss is constructed, calculated as follows: for multiple network layers specified in the pre-trained visual model, the feature representations of each layer for the predicted and reference normal maps are extracted. The weighted sum of the squared Euclidean distances between the features of each layer is calculated, and the weight coefficients assign higher weights to deeper features to emphasize high-level semantic consistency. Specifically, the selected Transformer layer indices are {6, 12, 18, 24}, and the corresponding weight coefficients are set to... The loss function can be expressed as:

[0058]

[0059] in This represents the feature map output by the l-th layer Transformer block of DINOv2. To predict the normal map, This is a reference normal diagram for training.

[0060] 2.2 Dual-view normal refinement

[0061] The front and back normal maps obtained from coarse completion are fused using a dual-view normal refinement network with cross-attention. This network employs a U-Net architecture (4 encoder layers, 4 decoder layers, 64 basic channels), introducing cross-view cross-attention modules at the bottleneck positions of the 3rd and 4th layers of the encoder. The cross-attention modules work by fusing the front normal features... As a query, back face normal feature Cross-attention is performed using keys and values, and the resulting residuals are concatenated. Obtain the updated frontal normal features At the same time, the back normal features As a query, positive normal feature Cross-attention is performed using keys and values, and the resulting residuals are concatenated. The updated back face normal features were obtained. This process can be expressed as:

[0062]

[0063]

[0064] By geometrically constraining the normal predictions from two perspectives, a refined, high-quality normal map is output. .

[0065] 2.3 Multi-view diffusion generation for attitude control

[0066] like Figure 2 As shown, using the completed frontal unoccluded image as a condition, and combining the multi-view pose control signal generated by the SMPL parameters, a pre-trained pose control multi-view diffusion model generates K unoccluded human images from different perspectives. In this embodiment, K=4, corresponding to azimuth angles of 0° (front), 90° (right side), 180° (back), and 270° (left side).

[0067] The multi-view attitude control signal is generated by rendering depth maps and refined normal maps from each target viewpoint using SMPL model parameters. The two signals are concatenated along the channel dimension to form the control signal. This control signal is injected into the denoising process of the multi-view diffusion model based on the denoising diffusion architecture via the ControlNet architecture: ControlNet is a copy of the U-Net encoder of the diffusion model (with the main network parameters frozen, only the ControlNet branch is trained), and its output is added to the corresponding layer of the main network through a zero-convolutional layer. The diffusion model is based on the Stable Diffusion architecture (U-Net, 4 downsampling layers + 4 upsampling layers, basic channel count 320), uses DDPM noise scheduling, and has 50 denoising steps.

[0068] 2.4 Pixel-level uncertainty estimation

[0069] like Figure 3 As shown, for each generated viewpoint image Pixel-level uncertainty estimation is performed using the heteroscedastic Gaussian noise assumption. The observation model is defined as follows: ,in is the observed value of pixel p at the viewpoint k (i.e., the pixel value of the generated image). For the corresponding true value, To conform to a mean of zero and a variance of The noise is distributed in a Gaussian pattern. This uncertainty reflects both the inherent uncertainty caused by information loss due to occlusion and the cognitive uncertainty caused by the randomness of sampling in the diffusion model. Total uncertainty. Combining two independent components using variance additivity:

[0070]

[0071] (a) Uncertainty of the occlusion source Acquisition

[0072] The uncertainty of occlusion sources reflects the inherent cognitive uncertainty caused by information deficiency. First, the occlusion area mask is... Gaussian blurring is performed to expand the range of uncertainty affecting the occlusion boundary (blur kernel size 21×21, standard deviation 3) to obtain a softened occlusion map. Design a lightweight occlusion uncertainty mapping network. Using a softened occlusion map as input, pixel-level occlusion uncertainty is predicted, and a softplus activation is applied to the output, which is then scaled to the range [0.1, 1.0].

[0073]

[0074] in This is a normalization constant to ensure that the occlusion uncertainty is within a reasonable range. The occlusion uncertainty in the visible region is close to the lower bound of 0.1, and the uncertainty in the completely occluded region is close to the upper bound of 1.0.

[0075] (b) Uncertainty about the diffusion source Acquisition

[0076] The uncertainty of the diffusion source reflects the inherent randomness and sampling differences in the generation process of the multi-view diffusion model. Multiple runs are performed under the same conditions (de-occlusion image, pose control signal) (in this embodiment). (Multi-view diffusion sampling) was performed to obtain... A separate set of generated results. For the same viewpoint k One generated result The variance is calculated at the pixel level as an estimate of diffusion uncertainty.

[0077]

[0078] in This represents the numerical stability term. To reduce computational costs, a lightweight diffusion uncertainty prediction network can be used. Alternative to multiple sampling: This network receives the latent space features of the denoising step in the 30th step of the diffusion process and predicts the diffusion uncertainty through a 3-layer U-Net network. This network is obtained through supervised learning on a multi-view dataset by L2 loss with the variance of the real multiple sampling.

[0079] Step 3: Uncertainty-guided 3D Gaussian splash human body reconstruction, such as... Figure 4 and Figure 5 As shown;

[0080] 3.1 Feedforward 3D Gaussian Initialization

[0081] Obscuring the original image K multi-view generated images The SMPL template mesh is fed into a feedforward human reconstruction model based on large-scale pre-training, and then directly regresses the initial 3D Gaussian set. In this embodiment, the initial number of Gaussian elements N is approximately 50,000. The backbone network of the feedforward reconstruction model employs a ViT-Large-based image encoder and a Transformer-based 3D decoder. The image encoder encodes each input viewpoint image into a 256×256 feature map, and the 3D decoder fuses multi-viewpoint features into the 3D coordinates of the SMPL template vertices through a cross-attention mechanism, predicting local Gaussian meta-attributes (position offset, scaling, rotation, opacity, and spherical harmonic coefficients) for each vertex position.

[0082] 3.2 Self-supervised Occlusion Adaptation Framework

[0083] To ensure the feedforward reconstruction model is robust to occlusion, a self-supervised occlusion adaptation framework is used for training, which employs a teacher-student distillation architecture. The teacher network receives a complete, unoccluded image as input and generates a high-quality reference reconstruction result; the student network receives an artificially synthesized occluded image as input and learns to approximate the teacher network's output under occlusion conditions.

[0084] Artificial occlusion synthesis strategies include: (a) rectangular occlusion: randomly placing 1 to 3 rectangular occlusion blocks on the human body region, with an area ratio of 10% to 60% of the human body region; (b) irregular occlusion: using random walk to generate irregularly shaped occlusion regions; (c) real occlusion object templates: extracting segmentation masks of common objects (furniture, vehicles, etc.) from the COCO dataset and superimposing them on the human body image.

[0085] The distillation loss is a weighted combination of rendering-level loss and feature-level loss. The rendering-level loss is the L1 distance between the rendered image reconstructed by the student network and the rendered image reconstructed by the teacher network, and the feature-level loss is the L2 distance between the intermediate layer features of the student network and the teacher network decoder. The distillation loss can be expressed as:

[0086]

[0087] in This is a differentiable Gaussian rasterization rendering operation. This represents the feature representation of the intermediate layer of the decoder. The weights are used for feature distillation. The student network is injected into the adaptation layer via LoRA, with the LoRA rank set to... Only about 2% of the parameters were adjusted.

[0088] 3.3 Uncertainty-Weighted Parameter Optimization

[0089] In the initial Gaussian set Based on this, Gaussian parameters are optimized through multi-view photometric consistency. An uncertainty map is incorporated into the optimization objective, and for the rendering loss of each viewpoint k, heteroscedastic negative log-likelihood photometric loss is employed. This loss is calculated as follows: for each viewpoint, the absolute difference between the rendered image and the target image at each pixel is calculated, this difference is divided by the uncertainty value corresponding to that pixel, and the logarithm of the uncertainty value is added, then summed over all pixels. This loss can be expressed as:

[0090]

[0091] in This is the rendering result of the current Gaussian set at viewpoint k. For the target image, This is to account for joint uncertainty. The physical meaning of this loss function is: for pixels with high uncertainty, the photometric residual is attenuated by the uncertainty denominator, and the corresponding gradient contribution is reduced; simultaneously, This is a degenerate solution to prevent uncertainty from tending towards infinity.

[0092] During the optimization process, the gradient acting on the center of the Gaussian pixel is adaptively scaled based on uncertainty. The scaling factor is inversely proportional to the average confidence level of the pixel uncertainty in the projected region, in order to reduce the interference of unreliable observations on the reconstruction process. The adaptive gradient scaling is calculated as follows:

[0093]

[0094] in For Gorsky Yuan The set of visible perspectives For Gaussian center in perspective The projected pixel coordinates on the surface, where the confidence level is defined as .

[0095] 3.4 Confidence-aware regularization

[0096] A confidence-aware regularization term is introduced to constrain the spatial distribution of the three-dimensional Gaussian.

[0097] (1) Uncertainty-weighted depth smoothing regularization: The gradient magnitude of the rendered depth map is weighted according to an adaptive weight based on uncertainty. The weight is larger in high-confidence regions (low uncertainty) to force depth smoothing, and smaller in high-uncertainty regions to relax the constraints. The expression is as follows:

[0098]

[0099] Adaptive weights ,in The temperature parameter represents the average uncertainty of pixel p across different viewpoints. .

[0100] (2) Uncertainty-guided Gaussian density control: In the three-dimensional Gaussian adaptive density control process, for Gaussian elements projected into regions with uncertainty higher than the threshold, the threshold in their splitting and cloning criteria is increased. Specifically, for those projected into regions with high uncertainty... In this embodiment The Gaussian unit (GSU) has its splitting threshold increased to twice the standard threshold to avoid excessively increasing the number of GSU in unreliable observation regions.

[0101] 3.5 Self-calibration refinement

[0102] Detail enhancement is performed on the coarsely reconstructed Gaussian ensemble. A single-step diffusion refinement model is employed, using the coarse rendering result as conditional input for super-resolution processing and texture refinement. The refinement model is based on the Stable Diffusion architecture, injecting conditional information via an IP-Adapter, and outputting the refined result after performing only one denoising step.

[0103] To prevent excessive illusions caused by the refinement model from shifting identity information, a self-calibration learning strategy is introduced. During refinement training, the input is replaced with the unenhanced original rendered image as an identity-preserving sample with a first probability γ=0.3. A hybrid loss function is employed to ensure that the model enhances details while constraining the output to not deviate from the original appearance. The hybrid loss function, composed of a denoising loss and a loss from the identity-preserving branch's noisy target, weighted by the first probability, can be expressed as:

[0104]

[0105] in For the noise predicted by the model, For the noise target of the denoising branch, Noise targets that maintain identity branches.

[0106] Step 4: Skeleton binding and animation-driven animation;

[0107] 4.1 Gaussian Skin Weight Prediction

[0108] Predict the linear hybrid skin weights of the corresponding SMPL skeletal joints for each 3D Gaussian element i. The number of joints in the SMPL model The skin weights are predicted using a lightweight MLP network (3 fully connected layers, hidden dimensions 128-64-24, ReLU activation function), with the Gaussian center position as the input. The coordinates in the SMPL canonical space and the local geometric features (32-dimensional) extracted from the encoder are used to output a softmax-normalized weight vector, ensuring that the sum of the weights is 1.

[0109] 4.2 Linear Hybrid Skin Transformation

[0110] In response to the target attitude parameters, a rigid transformation of the Gaussian elements is performed using the skin weights through a linear hybrid skin transformation. Given new target attitude parameters... A rigid transformation is performed on each Gaussian element using LBS transformation. The transformation of the Gaussian center position is as follows: the coordinates of the Gaussian center in gauge space are transformed by a weighted combination of the joint transformation matrices according to the skin weights, where the rigid body transformation matrix of each joint is... The Gaussian covariance matrix is ​​determined by the target attitude and body shape parameters. The transformation of the Gaussian covariance matrix involves rotating the normal space covariance matrix using a weighted rotation component. This rotation component is the rotation matrix obtained by interpolating the rotation matrices of each joint according to the skin weights. This transformation can be expressed as:

[0111]

[0112]

[0113] 4.3 Pose-dependent appearance mapping and decoupling of Map-Pose / LBS-Pose

[0114] like Figure 5 As shown, a pose-dependent appearance mapping network is designed to model non-rigid appearance changes caused by pose variations. This network is StyleUNet, employing a U-Net architecture (4 encoder layers with 64-128-256-512 channels, 4 decoder layers), with AdaIN-based style modulation introduced in each decoder layer. The network takes the SMPL UV parameterized mapping corresponding to the target pose as input (resolution 256×256, including UV space position). Figure 3 Channel, Normal Figure 3 Channel and semantic segmentation Figure 1 (7 channels in total), predicting the color shift of each Gaussian cell. and position offset .

[0115] The network introduces a decoupling mechanism between Map-Pose and LBS-Pose, dividing the Gaussian attribute mapping into two paths: the LBS-Pose path handles spatial transformations (position and orientation changes) caused by rigid bone deformation based on the skin weights and joint transformation matrices; the Map-Pose path handles non-rigid appearance changes (texture and shape pose-dependent changes) based on the UV parameterized mapping. The outputs of the two paths are added and fused in the feature space to obtain the Gaussian meta-attributes used to drive the animation.

[0116]

[0117]

[0118] 4.4 Facial Identity Preservation Strategies

[0119] Considering the high sensitivity of the facial region to identity perception, the Gaussian units of the head region are separated from the whole-body linear hybrid skinned skeleton and introduced into the FLAME parametric head model for independent modeling. The FLAME model includes shape parameters (300-dimensional), expression parameters (100-dimensional), and neck pose parameters (6-dimensional). The deformation of the head Gaussian units is driven by the expression parameters of the FLAME and the rotation of the neck joints.

[0120] Facial identity loss constrains the consistency of facial identities during animation-driven processes by calculating the identity similarity loss between the rendered facial image and the reference facial image in the face recognition feature space. This embodiment uses an ArcFace-ResNet100 pre-trained network F(·) to extract a 512-dimensional identity feature vector. The identity similarity loss is defined as:

[0121]

[0122] 4.5 Geometrically Faithful Texture Rendering Constraints

[0123] A geometric consistency loss is applied to the rendered normal map and depth map under the target pose. The difference between the normalized gradient directions of the rendered normal map and depth map is calculated and used as an optimization constraint. This constraint can be expressed as:

[0124]

[0125] This loss ensures that the direction of the Gaussian surface normal after animation deformation is consistent with the actual depth gradient direction, preventing visual defects such as texture tearing and geometric penetration during animation.

[0126] It should be further noted that in this embodiment, the total training loss is a weighted combination of the losses from each stage, expressed as follows:

[0127]

[0128] The losses in step 1 include keypoint reprojection loss, 3D joint loss, and parameter regularization loss.

[0129] ;

[0130] in , . For predicting key points and labeling key points distance, To predict 3D joint coordinates and ground truth distance.

[0131] The losses in step 2 include normal prediction loss, feature-level regularization loss, and diffusion denoising loss:

[0132] ;

[0133] in , .

[0134] The losses in step 3 include uncertainty-weighted photometric loss, SSIM structural similarity loss, depth regularization loss, and distillation loss:

[0135] ;

[0136] in , , .

[0137] The losses in step 4 include rendering loss under animation pose, facial identity loss, and geometric consistency loss:

[0138] ;

[0139] in ,

[0140] In this embodiment, the training dataset is constructed as follows: (1) High-quality human body scanning models are obtained from publicly available 3D human body scanning databases, covering different genders, body types and clothing styles; (2) 36 perspectives are uniformly sampled on the sphere for each scanning model, and a physically based renderer is used to render under diverse HDR ambient lighting to generate multi-view RGB images, normal maps, depth maps and segmentation masks; (3) Synthetic occlusions are superimposed on the rendered human body images. The sources of occlusions include common object segmentation instances and random geometric shapes in the COCO dataset, and the occlusion area ratio is uniformly sampled between 10% and 70%; (4) Background images are randomly sampled from a large-scale scene image database, and the background diversity is enhanced by color jitter, blur and noise injection.

[0141] Data augmentation strategies include: horizontal flipping (probability 0.5), random cropping and scaling (scale factor 0.8-1.2), color dithering (brightness / contrast / saturation ±0.2 each), and random erasure (probability 0.1).

[0142] A phased training strategy was adopted, using AdamW or Adam optimizer. The learning rate decayed according to the cosine annealing strategy (the warm-up rounds were 5% of the total training rounds), and training was completed on 4×A100 (80GB) GPUs.

[0143] Example 2

[0144] This embodiment provides an animated 3D human body model reconstruction system based on a single occluded image, such as... Figure 7 As shown, it includes the following modules:

[0145] The occlusion-aware parsing module is used to perform instance segmentation, occlusion detection, and occlusion-robust SMPL parameter estimation on a single input occluded human image, and outputs human region mask, occluded region mask, visible region mask, and SMPL model parameters.

[0146] The multi-view occlusion generation module performs visibility-guided geometric completion, pose-controlled multi-view diffusion generation, and pixel-level uncertainty estimation based on the output of the occlusion perception and parsing module, outputting K occluded human images from different views and their corresponding uncertainty maps. The geometric completion incorporates feature-level supervised regularization and dual-view normal thinning, while the uncertainty estimation simultaneously quantifies both the inherent uncertainty of occlusion and the cognitive uncertainty of diffusion generation.

[0147] An uncertainty-guided reconstruction module is used to input the occluded original image, the multi-view de-occluded human image, and the SMPL template mesh into a feedforward reconstruction network to obtain an initial 3D Gaussian set. Guided by the uncertainty map, the module performs uncertainty-weighted parameter optimization and self-calibration refinement on the initial 3D Gaussian set, outputting a reconstructed 3D Gaussian human model. This module incorporates the uncertainty map into the optimization objective, constructs a maximum a posteriori estimation framework, optimizes the Gaussian parameters using a photometric loss function based on heteroscedasticity negative log-likelihood, and suppresses the interference of unreliable observations on the optimization process through uncertainty adaptive gradient modulation. The feedforward reconstruction network obtains occlusion robustness through teacher-student distillation.

[0148] The skeletal animation-driven module predicts skin weights for Gaussian primitives in the 3D Gaussian human model and achieves target pose-driven animation through a linear hybrid skin transformation and pose-dependent appearance mapping network, while simultaneously performing independent facial modeling and geometrically consistent rendering constraints. This module introduces a Map-Pose and LBS-Pose decoupling mechanism, modeling rigid skeletal deformation and non-rigid appearance changes along separate paths, and employs a FLAME head model to independently drive the facial region, thus preserving facial identity information.

[0149] Example 3

[0150] This embodiment provides an electronic device, such as... Figure 8 As shown, the system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor may be a CPU, a GPU, or a combination of both, and in a preferred embodiment includes at least one GPU (preferably an NVIDIA A100 or equivalent or higher). The memory includes high-speed RAM (at least 80GB) and non-volatile memory (at least 1TB solid-state drive). The processor, memory, and communication interface communicate with each other via a communication bus. When the processor executes the computer program, it implements all the steps of the method described in Embodiment 1.

[0151] This embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements all the steps of the method described in Embodiment 1. The computer-readable storage medium may include, but is not limited to: electrical connections, portable disks, hard disks, RAM, ROM, EPROM or flash memory, optical fibers, CD-ROMs, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0152] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for reconstructing an animatable 3D human body model based on a single occluded image, characterized in that, include: Step S1: Receive a single RGB image of a human body containing occlusion, perform human instance segmentation and occlusion detection, and obtain the human body region mask, occlusion region mask and visible region mask; use an occlusion robust human parameter regression network to estimate the SMPL model parameters of the human body based on feature extraction under the constraint of the visible region mask, wherein the SMPL model parameters include body shape parameters, pose parameters and camera parameters. Step S2: Based on the SMPL model parameters and the visible area mask, perform visibility-guided geometric completion to obtain a complete human body normal map; using the completed frontal image and the multi-view pose control signal generated according to the SMPL model parameters as conditions, generate K occluded human body images from different viewpoints through a multi-view diffusion model, and estimate pixel-level uncertainty maps for each generated viewpoint. Step S3: Input the occluded original image, K multi-view generated images, and the template mesh corresponding to the SMPL model into the pre-trained feedforward human reconstruction model to obtain an initial three-dimensional Gaussian set; guided by the uncertainty map, perform uncertainty-weighted parameter optimization on the initial three-dimensional Gaussian set to obtain the reconstructed three-dimensional Gaussian human model. Step S4: Predict the skin weights of the corresponding SMPL skeletal joints for each Gaussian primitive in the 3D Gaussian human body model; in response to the target pose parameters, use the skin weights to perform a rigid transformation on the Gaussian primitive through a linear hybrid skin transformation, and combine a pose-dependent appearance mapping network to predict the attribute offset of the Gaussian primitive to achieve skeletal animation drive.

2. The method of claim 1, wherein, In step S1, the occlusion-robust human parameter regression network adopts the Transformer architecture and introduces a visibility-guided attention mechanism in its self-attention calculation. This mechanism applies the visibility weight map generated by the visible region mask to the attention score and applies a negative bias to the attention score corresponding to the occluded region to suppress the interference of occlusion noise on parameter estimation.

3. The method of claim 1, wherein, Step S1 further includes: obtaining a vertex-level visibility map of the human body by forward kinematic rendering of the SMPL model based on the estimated SMPL model parameters, providing geometric priors for the geometric completion.

4. The method of claim 1, wherein, In step S2, visibility-guided geometric completion includes: Using a geometric completion network with an encoder-decoder architecture, the complete frontal and back normal maps of the human body are predicted by taking the visible region image, the visible region mask, and the normal prior map rendered by SMPL as input. When training the geometric completion network, feature-level supervised regularization is introduced. The pre-trained visual base model is used to extract the semantic features of the predicted normal map and the reference normal map in multiple deep layers of the network. The weighted sum of the feature differences of each layer is calculated as the feature-level consistency loss to constrain the semantic consistency between the completed region and the visible region.

5. The method of claim 4, wherein, Step S2 further includes a dual-view normal refinement step: the front normal map and the back normal map are processed through a dual-view normal refinement network. This network introduces a cross-view cross-attention module at the bottleneck layer, using the front normal features as the query and the back normal features as the key for cross-attention calculation. At the same time, the back normal features are used as the query and the front normal features as the key for cross-attention calculation. Through bidirectional information exchange, the normal predictions of the two views are geometrically constrained, and the refined normal map pair is output.

6. The method according to claim 1, characterized in that, In step S2, the multi-view pose control signal is: the depth map and normal map rendered by the SMPL model parameters under each target view; the multi-view diffusion model is based on the denoising diffusion architecture, and the multi-view pose control signal is injected into the denoising process through ControlNet to generate occluded human images with K≥4 different azimuth angles.

7. The method according to claim 1, characterized in that, In step S2, the estimation of the pixel-level uncertainty map adopts the heteroscedastic Gaussian noise assumption. The observation model is defined as the observed value of any pixel in each generated viewpoint image equals the true value plus a Gaussian noise with a mean of zero and a variance equal to the uncertainty corresponding to that pixel. The uncertainty reflects both the inherent uncertainty caused by information loss due to occlusion and the cognitive uncertainty caused by the randomness of sampling in the diffusion model.

8. The method according to claim 1, characterized in that, In step S3, the feedforward human reconstruction model obtains occlusion robustness through training with a self-supervised occlusion adaptation framework. This framework adopts a teacher-student distillation architecture, in which the teacher network receives a complete, unoccluded image input and generates a reference reconstruction result; the student network receives an artificially synthesized occluded image input and is constrained by a weighted combination of rendering-level loss and feature-level loss, forcing the student network output to approximate the teacher network output; the rendering-level loss is the L1 distance between the rendered image reconstructed by the student network and the rendered image reconstructed by the teacher network, and the feature-level loss is the L2 distance between the intermediate layer features of the student network and the teacher network.

9. The method according to claim 1, characterized in that, In step S3, the uncertainty-weighted parameter optimization employs heteroscedastic negative log-likelihood photometric loss. For each viewpoint, the absolute difference between the rendered image and the target image at each pixel is calculated. This difference is divided by the uncertainty value corresponding to that pixel and then summed with the logarithm of the uncertainty value, serving as part of the optimization objective. Furthermore, during the optimization process, the gradient acting on the center position of the Gaussian pixel is adaptively scaled. The scaling factor is inversely proportional to the average confidence level of the pixel uncertainty in the projection region, in order to reduce the interference of unreliable observations on the reconstruction process.

10. The method according to claim 9, characterized in that, Step S3 also includes confidence-aware regularization: introducing depth smoothing regularization loss, weighting the gradient magnitude of the rendered depth map according to an adaptive weight based on uncertainty, with a larger weight in high-confidence regions to force depth smoothing, and a smaller weight in high-uncertainty regions to relax constraints; and in the process of 3D Gaussian adaptive density control, for Gaussian elements projected to regions with uncertainty higher than the threshold, increasing the threshold in their splitting and cloning criteria to avoid excessively increasing the number of Gaussian elements in unreliable observation regions.

11. The method according to claim 1, characterized in that, Step S3 further includes a self-calibration refinement step: the coarse reconstruction result is enhanced in texture through a single-step diffusion refinement model. When training the refinement model, the input is replaced with the unenhanced original rendered image as an identity preservation sample with a first probability, and a hybrid loss function is used to ensure that the model enhances details while constraining the output not to deviate from the original appearance. The hybrid loss function is composed of a combination of denoising loss and loss of the identity preservation branch noise target with the first probability as weight.

12. The method according to claim 1, characterized in that, In step S4, the pose-dependent appearance mapping network is StyleUNet, which takes the SMPL UV parameterized mapping corresponding to the target pose as input and predicts the color shift and position shift of each Gaussian unit. The network introduces a Map-Pose and LBS-Pose decoupling mechanism, in which the LBS-Pose path processes the spatial transformation caused by rigid bone deformation according to the skin weights and joint transformation matrix, and the Map-Pose path processes non-rigid appearance changes according to the UV parameterized mapping. The outputs of the two paths are added and fused in the feature space to obtain the Gaussian unit attributes used to drive the animation.

13. The method according to claim 1, characterized in that, Step S4 also includes a facial identity preservation strategy: the Gaussian units of the head region are separated from the whole-body linear hybrid skin skeleton, and the FLAME parameterized head model is introduced for independent modeling. The deformation of the head Gaussian units is driven by the expression parameters of FLAME and the rotation of the neck joint. The consistency of facial identity is constrained when the animation is driven by calculating the identity similarity loss between the rendered facial image and the reference facial image in the face recognition feature space.

14. The method according to claim 1, characterized in that, Step S4 also includes a geometrically faithful texture rendering constraint: for the normal map and depth map rendered under the target pose, the difference between the gradient direction normalization results of the normal map and the depth map is calculated and used as the geometric consistency loss to suppress texture tearing and geometric penetration.

15. A system for reconstructing an animated 3D human body model based on a single occluded image, implementing the method as described in any one of claims 1 to 14, characterized in that, include: The occlusion-aware parsing module is used to perform instance segmentation, occlusion detection, and occlusion-robust SMPL parameter estimation on a single input occluded human image, and outputs human region mask, occlusion region mask, and SMPL model parameters. The multi-view de-occlusion generation module is used to perform visibility-guided geometric completion, pose-controlled multi-view diffusion generation, and pixel-level uncertainty estimation based on the output of the occlusion perception and analysis module, and output a multi-view de-occluded human image and the corresponding uncertainty map. The uncertainty-guided reconstruction module is used to input the occluded original image, the multi-view de-occluded human body image and the SMPL template mesh into the feedforward reconstruction network to obtain an initial three-dimensional Gaussian set. Guided by the uncertainty map, the module performs uncertainty-weighted parameter optimization and self-calibration refinement on the initial three-dimensional Gaussian set and outputs the reconstructed three-dimensional Gaussian human body model. The skeletal animation driving module is used to predict skin weights for Gaussian primitives in the 3D Gaussian human body model, and realize target pose-driven animation through linear hybrid skin transformation and pose-dependent appearance mapping network, while performing face-independent modeling and geometric consistency rendering constraints.

16. The system according to claim 15, characterized in that, The uncertainty-guided reconstruction module incorporates the uncertainty map into the optimization objective, constructs a maximum a posteriori estimation framework, optimizes the Gaussian parameters using a photometric loss function based on heteroscedasticity negative log-likelihood, and suppresses the interference of unreliable observations on the optimization process through uncertainty adaptive gradient modulation.

17. The system according to claim 15, characterized in that, The skeletal animation driving module introduces a Map-Pose and LBS-Pose decoupling mechanism, models rigid skeletal deformation and non-rigid appearance changes separately, and uses the FLAME head model to independently drive the facial region in order to maintain facial identity information.

Images