A method and apparatus for generating and reconstructing closed-loop 3D human body reconstruction guided by occlusion perception confidence.
By explicitly measuring the unit-by-unit observation reliability and prior constraints of human structure in the 3D human body reconstruction method, and adopting a closed-loop method of alternating generation and reconstruction, the problems of incomplete human body reconstruction and contamination under occlusion conditions are solved, and efficient and anatomically reasonable 3D human body reconstruction is achieved.
Patent Information
- Application Number
- CN202611416376.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-14
- Publication Date
- 2026-11-17
AI Technical Summary
Existing human reconstruction methods under occlusion conditions cannot effectively distinguish the reliability of observations, resulting in incomplete reconstruction results in occluded areas or prior contamination of the observation area. Furthermore, the iterative process is unstable and cannot meet the requirements of anatomical rationality and efficient convergence.
By explicitly measuring the reliability of unit-by-unit observations in the 3D representation, utilizing prior constraints of human structure, and employing a closed-loop method of alternating generation and reconstruction, the method adaptively schedules real observations and generated inference information, achieving spatially adaptive loss function modulation and iteration termination determination, thus forming a confidence-guided generation and reconstruction closed loop.
It achieves complete completion of occluded areas and zero-contamination protection of the observation area, improves reconstruction accuracy, stabilizes and optimizes the iteration process, generates content with reasonable structure, and reduces the average number of iterations by about 30%.
Smart Images

Figure CN123118916A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision and computer graphics technology, specifically to a method and apparatus for generating and reconstructing closed-loop 3D human body reconstruction guided by occlusion perception confidence. Background Technology
[0002] 3D human reconstruction aims to restore the complete 3D geometry and appearance of a target human body from image or video observations. It is widely used in virtual reality, augmented reality, film and television special effects production, virtual try-on, digital human live streaming, remote presentation, and sports and medical motion analysis. Traditional high-quality human reconstruction relies on multi-camera synchronous acquisition arrays (densely arranged around dozens to hundreds of cameras), which are costly, complex to calibrate, and difficult to deploy, making it difficult to reach consumer-grade applications.
[0003] In recent years, differentiable rendering representations, such as Neural Radiance Field (NeRF) and 3D Gaussian Splatting (3DGS), have made human reconstruction possible under sparse viewpoints and even monocular conditions. However, under everyday shooting conditions, the target human body is often obscured by furniture, other people, handheld objects, limbs, or loose clothing, or only partially visible due to framing limitations, resulting in an inherently incomplete observation. In such cases, reconstruction becomes severely inaccurate: areas not observed by any viewpoint lack supervisory signals, leading to voids, floating artifacts, geometric collapse, or color leakage in the reconstruction results, significantly impacting downstream applications.
[0004] For human reconstruction under incomplete observation, there are three main approaches in existing technologies.
[0005] The first category is the parametric human model prior approach. This type of method uses skinned parametric human models such as SMPL and SMPL-X as strong priors, regressing human pose and roughness parameters under occlusion conditions. Representative methods improve occlusion robustness through body part attention mechanisms. Its drawback is that parametric models only express smooth human geometry at the bare model level, failing to reconstruct appearance details such as clothing and hairstyle. The output results differ significantly from realistic photorealistic human figures, making it unsuitable for applications requiring high fidelity in appearance.
[0006] The second type is the "reconstruction followed by repair" unidirectional pipeline approach. This type of method first performs conventional reconstruction in the visible area, and then calls a 2D image inpainting network or diffusion generation model to perform unidirectional repair on the incomplete rendering result. Its drawbacks are: (a) the repair occurs in the 2D image domain, and the results of independent repair from multiple perspectives lack 3D consistency, resulting in content jumps when switching perspectives; (b) the repair results are not fed back into the 3D representation optimization process, and the 3D model itself is still incomplete; (c) the repair model cannot distinguish between the real observed area and the missing area, and the generated content may overwrite reliable observations, causing "observation pollution"—that is, the real details captured by the user are replaced by the imagined content of the generated model.
[0007] The third category is the collaborative approach of generating priors to guide reconstruction. This type of method introduces a pre-trained generative model into the reconstruction process using either score distillation sampling (SDS) loss or a fixed-period triggered generative inpainting followed by fusion back into 3D optimization, forming a preliminary generative-reconstruction collaboration. This approach represents the current research frontier; however, according to the applicant's analysis, existing methods suffer from the following common technical shortcomings: (1) The lack of explicit spatial modeling of observation reliability leads to "insufficient supplementation where necessary and contamination of areas that should not be modified". The generation guidance acts on the entire 3D model with a spatially uniform intensity, making it impossible to distinguish between "which units have been reliably observed and should be protected" and "which units are completely missing observations and require strong generation inference". If the generation guidance intensity is set too low, the occluded areas will be incompletely supplemented; if it is set too high, the real details of the visible areas will be skewed by the generation prior. This globally uniform guidance strategy cannot, in essence, simultaneously satisfy the two conflicting needs of fidelity preservation and supplementation.
[0008] (2) The alternation between generation and reconstruction lacks an adaptive scheduling and termination mechanism, resulting in unreasonable allocation of computing power and uncontrollable convergence. Existing methods trigger generation with a fixed manually set period and end iterations with a fixed total number of rounds, making it impossible to perceive the completion progress: this leads to wasted computing power for mildly occluded samples and poor convergence for severely occluded samples, and the fixed strategy cannot differentiate the progress for different spatial regions of the model. In addition, the completion results lack a stability guarantee mechanism in multiple iterations, making it prone to repeated oscillations and unable to determine when the optimal termination point is reached.
[0009] (3) The closed-loop method for general scenarios does not utilize prior knowledge of human body structure, and the completion results lack anatomical rationality. When the closed-loop method for general scenarios is directly applied to the human body, the completion of occluded areas lacks constraints of human anatomical structure, and is prone to generating content that violates human topology and symmetry (such as missing limbs, broken clothing structure), which cannot meet the requirements of digital human applications for pose dynamism and structural integrity.
[0010] (4) Existing methods do not establish a unit-by-unit progress recording and iteration locking mechanism in three-dimensional space. The generated completion results are repeatedly overwritten during iteration, lacking a ratchet-like advancement mechanism of "locking and protecting the completed region after verification", resulting in slow and unstable closed-loop convergence, and the termination criterion lacks physical interpretability.
[0011] In summary, there is an urgent need for a 3D human reconstruction method that can explicitly measure the reliability of unit-by-unit observations in 3D representation, adaptively schedule both real observations and generative inferences in space, utilize prior constraints of human body structure for completion, and adaptively determine the termination of iteration based on the completion progress. This method would simultaneously achieve zero-contamination protection of the observation area and complete, anatomically reasonable completion of missing areas under occlusion conditions, while ensuring that the iterative process is efficient and controllable. Summary of the Invention
[0012] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a confidence-guided method and apparatus for generating and reconstructing a closed-loop 3D human body based on occlusion perception. This method can explicitly measure the reliability of unit-by-unit observations in the 3D representation, thereby adaptively scheduling two information sources—real observations and generative inferences—in space, utilizing prior constraints of human body structure for completion, and adaptively determining the termination of iteration based on the completion progress. This 3D human body reconstruction method can simultaneously achieve zero-contamination protection of the observation area and complete, anatomically reasonable completion of missing areas under occlusion conditions, and the iterative process is efficient and controllable.
[0013] To achieve the above objectives, the technical solution specifically adopted by the present invention is as follows: A confidence-guided method for occlusion perception in generating and reconstructing closed-loop 3D human bodies includes the following steps: Step S1: Obtain an image or video sequence containing the occluded target human body as observation data, and construct an initial three-dimensional representation of the target human body based on the observation data. The initial three-dimensional representation is composed of multiple basic units. Step S2: Calculate the observation confidence for each basic unit to obtain a unit-by-unit confidence field corresponding to the initial three-dimensional representation; the observation confidence is obtained by normalizing the local features of the basic unit's visibility measure relative to the observation data, the photometric consistency measure across viewpoints, and the geometric consistency measure with the parameterized human prior model through a learnable gating network. Step S3: The fusion scheduler performs spatial adaptive optimization of the 3D representation based on the unit-by-unit confidence field. The spatial adaptive method is as follows: First, a first threshold and a second threshold are preset. For basic units with observation confidence higher than the first threshold, the reconstruction loss constructed from the observation data is used for optimization. For basic units with observation confidence lower than the second threshold, the pre-trained generation model is called to infer pseudo-observations based on the rendering results of the current 3D representation and the prior human conditions. The generation guidance loss constructed from the pseudo-observations is used as the main component for completion to obtain the optimized 3D representation. Step S4: Re-render the optimized 3D representation, update the unit confidence field based on the consistency between the rendering results and the observation data and pseudo-observations, and return to step S3 to form a closed loop of alternating generation and reconstruction. After completing the iteration, output a complete 3D human body model.
[0014] Preferably, in step S2, the visibility measurement is determined by projecting the basic unit to each observation viewpoint and statistically analyzing the proportion of views that are not occluded and fall into the effective human body area of the image; the occlusion judgment is achieved by combining the rendering depth buffer and the human body segmentation mask. The geometric consistency metric is determined by calculating the symbolic distance from the basic unit to the surface of the skinned parametric human model and mapping it to a consistency score using a distance decay function.
[0015] Preferably, the local features of the basic unit include at least one of opacity, scale, and neighborhood density.
[0016] Preferably, the generation model is a multi-view consistent image diffusion model, and the human body prior conditions include at least one of a normal map, skeleton map, or semantic segmentation map rendered by a parameterized human body model.
[0017] Preferably, in step 3, the generation guidance loss weight of the i-th basic unit decreases monotonically as the observation confidence of the unit increases, while the reconstruction loss weight increases monotonically as the observation confidence of the unit increases.
[0018] Preferably, the generation guidance loss weight is determined as follows: a baseline weight and a modulation index are set, and the baseline weight is multiplied by the modulation index raised to the power of the difference between the observation confidence of the unit; the reconstruction loss weight is directly taken as the observation confidence of the unit.
[0019] Preferably, the initial noise intensity during the generation of the diffusion model is adaptively set according to the observation confidence of the target region: the lower the observation confidence, the higher the initial noise intensity.
[0020] As a preferred option, when updating the confidence field in step S4, for basic units whose rendering results are stable and consistent across iterations after completion, their confidence is increased based on their stability score, and the upper limit of the increased confidence is lower than the upper limit of the confidence that can be reached in the directly observed area.
[0021] Preferably, the incremental statistic is the average absolute value of the unit-by-unit difference in the confidence field, and the coverage rate is the proportion of basic units with a confidence level higher than the first threshold to all basic units.
[0022] Preferably, step 4 further includes calculating a termination criterion after each iteration. The termination criterion includes the incremental statistics and confidence coverage of the confidence field in two adjacent iterations. When the incremental statistics are lower than the incremental threshold and the coverage is higher than the coverage threshold, the iteration is terminated and a complete three-dimensional human body model is output.
[0023] Preferably, the termination criterion also includes a maximum iteration round protection condition.
[0024] Preferably, the observation data is a dynamic video sequence, the basic unit is bound to the skin deformation field of the parameterized human body model, and the confidence field is maintained in the gauge space and mapped to each observation frame through deformation.
[0025] This invention also discloses an occlusion-aware confidence-guided generation and reconstruction device for closed-loop 3D human body reconstruction, comprising: The observation acquisition and initialization module is used to acquire at least one image or video sequence of the target human body that includes occlusion or incomplete observation as observation data, and to construct an initial three-dimensional representation of the target human body based on the observation data. The initial three-dimensional representation is composed of multiple basic units, which are three-dimensional Gaussian units or voxel units. The confidence field estimation module is used to calculate the observation confidence for each of the basic units to obtain a unit-by-unit confidence field corresponding to the initial three-dimensional representation; wherein, the observation confidence is determined by fusing at least the visibility measure of the basic unit relative to the observation data, the photometric consistency measure across viewpoints or frames, and the geometric consistency measure with the parameterized human prior model; The fusion scheduling module is used to perform spatial adaptive optimization of the 3D representation based on the unit-by-unit confidence field through a fusion scheduler: for basic units with observation confidence higher than a first threshold, optimization is mainly performed using the reconstruction loss constructed from the observation data, and the modification of the generative model to that unit is suppressed; for basic units with observation confidence lower than a second threshold, a pre-trained generative model is invoked to infer pseudo-observations based on the rendering results of the current 3D representation and human prior conditions, and the generation guidance loss constructed from the pseudo-observations is mainly used for completion; wherein the reconstruction loss weight and the generation guidance loss weight are both continuously and adaptively modulated spatially with the observation confidence, so as to achieve a smooth transition between the two types of regions; The closed-loop update module is used to re-render the 3D representation optimized by the fusion scheduling module, update the unit confidence field based on the consistency between the rendering result and the observation data and pseudo-observations, and trigger the fusion scheduling module to execute again, forming a closed loop of alternating generation and reconstruction. The termination determination and output module is used to calculate the termination criterion after each iteration. The termination criterion includes at least the incremental statistics and confidence coverage of the confidence field in two adjacent iterations. When the incremental statistics are lower than the incremental threshold and the coverage is higher than the coverage threshold, the iteration is terminated and a complete three-dimensional human body model is output.
[0026] Preferably, in the confidence field estimation module, the visibility metric is determined by projecting basic units onto each observation viewpoint and statistically analyzing the proportion of viewpoints that are not occluded and fall into the effective human body region of the image; the occlusion judgment is jointly implemented using the rendering depth buffer and the human body segmentation mask.
[0027] Preferably, in the confidence field estimation module, the geometric consistency metric is determined by calculating the signed distance from the basic unit to the surface of the skinned parametric human model and mapping it to a consistency score through a distance decay function.
[0028] Preferably, in the confidence field estimation module, the fusion determination is implemented using a learnable gating network. The gating network takes the visibility metric, the photometric consistency metric, the geometric consistency metric, and the local features of the basic unit as inputs and outputs a normalized observation confidence.
[0029] Preferably, in the closed-loop update module, when updating the confidence field, for basic units whose rendering results are stable and consistent across iterations after completion, their confidence is increased according to the stability score, and the upper limit of the increased confidence is lower than the upper limit of the confidence that can be reached in the directly observed area.
[0030] This invention has the following characteristics and beneficial effects: Low-confidence regions receive sufficient generative inference to achieve complete completion, while high-confidence regions are explicitly protected from being overwritten by generative priors. Experiments show that the reconstruction accuracy of the visible region does not decrease but rather increases after introducing generative priors, with PSNR improving by approximately 0.8 dB. The fundamental reason for this is that the confidence field serves as the basis for fusion scheduling, enabling continuous adaptive modulation of the generative guidance intensity in space. High-confidence regions have extremely low or frozen weights, while low-confidence regions have sufficient weights, resulting in a smooth transition between the two without hard boundaries.
[0031] The confidence field, as a closed-loop dynamic state variable, records the completion progress. The dual-threshold termination criterion directly links the iteration termination time to the completion completion level. Mildly occluded samples are stopped early, while severely occluded samples are iterated sufficiently. Compared to a fixed-round-count strategy, this reduces the average number of iterations by approximately 30% without compromising reconstruction quality. The fundamental reason is that the confidence increment statistic reflects the rate of change in completion progress; when the increment is below the threshold, it indicates that completion has converged. The coverage rate reflects the overall reconstruction completeness; the joint determination of these two metrics avoids misjudgment based on a single indicator.
[0032] In the confidence field construction, a geometric consistency metric with the parameterized human prior model is introduced. The signed distance from the basic unit to the SMPL / SMPL-X surface is calculated and mapped to a consistency score through distance decay. This assigns low confidence to floating artifact units far from the human prior surface, and their content is more reconstructed by the generative prior. In conjunction with the generative model, the normal map / skeleton map rendered by the parameterized human model is used as a condition for completion. The surface distance constraint at the 3D level and the normal / skeleton condition at the 2D level ensure the anatomical rationality of the generated content, avoiding structural errors such as limb loss and topological breakage when directly transferring general scene methods.
[0033] The pseudo-reliable region asymptotic promotion mechanism ensures that the confidence of the completed content is increased as soon as it is verified across iterations, automatically entering a protected state in subsequent iterations. This prevents the completed region from being repeatedly rewritten by the subsequent generation process. The closed loop thus progresses monotonically in a ratchet-like manner, ensuring stable convergence and giving the confidence increment statistic a clear physical meaning and reliable numerical trend as a termination criterion.
[0034] The basic unit can be a three-dimensional Gaussian unit or a voxel unit. The observation data can be a single image, a small number of photos, or a monocular video sequence. The generated model can be an image diffusion model or a video diffusion model. In dynamic scenes, the basic unit is bound to the skin deformation field and confidence field of the parameterized human body model. These fields are maintained in the gauge space and mapped to each observation frame through deformation. All kinds of settings can be replaced, providing a wide range of implementation flexibility. Attached Figure Description
[0035] Figure 1 This is a schematic diagram of the overall process of the method provided in the embodiments of the present invention; Figure 2This is a schematic diagram of the unit-by-unit confidence field calculation process provided in an embodiment of the present invention; Figure 3 This is a flowchart of the spatial adaptive optimization method for three-dimensional representation in an embodiment of the present invention; Figure 4 A schematic diagram of the fusion scheduler structure and weight modulation curve provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the generation and reconstruction closed-loop iteration timing and termination criteria provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the device module structure provided in an embodiment of the present invention. Detailed Implementation
[0036] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.
[0037] Example 1 This embodiment provides a confidence-guided method for generating and reconstructing closed-loop 3D human body reconstruction based on occlusion perception, such as... Figure 1 As shown, it includes the following steps: Step S1: Obtain an image or video sequence containing the occluded target human body as observation data, and construct an initial three-dimensional representation of the target human body based on the observation data. The initial three-dimensional representation is composed of multiple basic units.
[0038] The observation data consists of dynamic video sequences. The basic unit is bound to the skin deformation field of the parameterized human body model. The confidence field is maintained in the gauge space and mapped to each observation frame through deformation.
[0039] In this embodiment, a monocular video of a partially occluded human body is input, specifically 30–300 frames. The occlusion source can be furniture, other people, or one's own limbs. Existing methods are used to estimate camera parameters, human instance segmentation masks, and SMPL-X pose / body shape parameters for each frame. N 3D Gaussian units are initialized near the SMPL-X specification space surface. In this embodiment, N = 100,000–300,000. Each unit records its position, covariance, opacity, and spherical harmonic color coefficients, and is bound to the parameterized model via linear hybrid skinning (LBS), mapping to the observation space with pose deformation in each frame. Basic optimization is first performed on the photometric loss of only the visible region (within the segmentation mask and not covered by occlusion objects) for several rounds. In this embodiment, the optimization is performed for 2000 steps to obtain the initial 3D representation. At this point, the occluded region appears as holes or artifacts, and the basic unit is a Gaussian unit.
[0040] Step S2: Calculate the observation confidence for each basic unit to obtain a unit-by-unit confidence field corresponding to the initial three-dimensional representation. The observation confidence is obtained by normalizing the local features of the basic unit's visibility measure relative to the observation data, the photometric consistency measure across viewpoints, and the geometric consistency measure with the parameterized human prior model through a learnable gating network.
[0041] Specifically, such as Figure 3 As shown, the visibility measure relative to the observed data Project the center of the unit onto the image plane according to the skin deformation of each frame, use the current rendering depth buffer to determine whether the unit is occluded by a closer surface, and combine the human body segmentation mask to determine whether the projection point falls into the effective human body area. The proportion of frames that pass the above double verification is taken out of the total number of frames.
[0042] Cross-view photometric consistency measurement The color samples of the unit's projection neighborhood in each visible frame are collected, aligned to the normal space by inverse skinning, and then the variance is calculated and mapped by negative exponential mapping. [jx1] is the consistency score; this item is used to eliminate unreliable observations caused by motion blur, segmentation error, occlusion of boundaries, etc.
[0043] Geometric consistency measure with parameterized human prior models : Symbol distance from the center of the computational cell to the surface of SMPL-X , mapped to [jx2]; The area for loose clothing can be appropriately enlarged. Tolerating deviations.
[0044] Furthermore, the three metrics are concatenated with local features and then input into a two-layer MLP gated network to output normalized confidence scores. Local features specifically refer to opacity, scale, and neighborhood cell density. In this embodiment, the gating network is pre-trained on synthetic data with occlusion ground truth, rendered using human scanning models such as THuman, and the occlusions are procedurally synthesized; parameters are frozen during inference. As a simplified alternative implementation, it can be directly taken... The weighted geometric mean is obtained without any training. This yields the unit-by-unit confidence field. .
[0045] Step S3: The fusion scheduler performs spatial adaptive optimization of the 3D representation based on the unit-by-unit confidence field. The spatial adaptive method is as follows: First, a first threshold and a second threshold are preset. For basic units with observation confidence higher than the first threshold, the reconstruction loss constructed from the observation data is used for optimization. For basic units with observation confidence lower than the second threshold, the pre-trained generation model is called to infer pseudo-observations based on the rendering results of the current 3D representation and the prior human conditions. The generation guidance loss constructed from the pseudo-observations is used as the main component for completion to obtain the optimized 3D representation.
[0046] In this embodiment, the generation guidance loss weight of the i-th basic unit monotonically decreases as the observation confidence of that unit increases, while the reconstruction loss weight monotonically increases as the observation confidence of that unit increases. The generation guidance loss weight is determined as follows: a baseline weight and a modulation index are set, and the baseline weight is multiplied by the modulation index raised to the power of the difference between the observation confidence of that unit and the baseline weight; the reconstruction loss weight is directly taken as the observation confidence of that unit.
[0047] Specifically, such as Figure 4 As shown, let the first threshold be... Second threshold The fusion scheduler calculates the reconstruction loss weight for each unit. With the generation of guiding weights (In this embodiment, Based on this continuous modulation, two discrete protection / triggering strategies are superimposed: for The unit freezes its geometric parameters (position, covariance) and only allows fine-tuning of the appearance, achieving hard protection of observation.
[0048] Furthermore, the generative model is a multi-view consistent image diffusion model, and the human body prior conditions include at least one of the normal map, skeleton map, or semantic segmentation map rendered by the parameterized human body model. Specifically, for The image region covered by the unit triggers completion generation, that is, it calls the multi-view consistent image diffusion model based on the SMPL-X normal map and skeleton map. In this embodiment, a video diffusion model finely tuned on human body scan data is used. Starting from the rendering result of the current 3D representation, partial noise denoising is performed to generate pseudo-observations. The initial noise intensity for denoising is set to ,in The average confidence level of the triggering region. Pseudo-observations are back-optimized for the corresponding unit using a generation-guided loss, which includes photometric and perceptual terms. Since all weights follow... The continuous changes are smooth, with no hard boundary seam artifacts between the observation protection zone, the transition zone and the generation and completion zone.
[0049] Understandably, the initial noise intensity during the generation of the diffusion model is adaptively set according to the observation confidence of the target region: the lower the observation confidence, the higher the initial noise intensity.
[0050] Step S4: Re-render the optimized 3D representation, update the unit confidence field based on the consistency between the rendering results and the observation data and pseudo-observations, and return to step S3 to form a closed loop of alternating generation and reconstruction. After completing the iteration, output a complete 3D human body model.
[0051] Understandably, when updating the confidence field, for basic units whose rendering results are stable and consistent across iterations after completion, their confidence is increased based on their stability score, and the upper limit of the increased confidence is lower than the upper limit of the confidence that can be reached in the directly observed area.
[0052] Specifically, in this embodiment, after each round of optimization, the rendering is repeated, and the structural similarity of the current round and the previous round of rendering in the projection neighborhood of each unit is compared: for the generated completion units that are stable and consistent for K consecutive rounds (SSIM greater than 0.9), in this embodiment, K=2, their confidence is increased according to the stability score, and truncated to the upper limit. (Below 1.0, which is achievable by directly observed units), enabling a gradual promotion to the pseudo-reliable region. The promoted unit automatically enters a protected state in the next round of scheduling, thus the closed loop progresses monotonically in a ratchet-like manner.
[0053] Furthermore, such as Figure 5 As shown, a termination criterion is calculated after each iteration. The termination criterion includes the incremental statistics and confidence coverage of the confidence field in two adjacent iterations. The iteration terminates when the incremental statistics are lower than the incremental threshold and the coverage is higher than the coverage threshold, and a complete 3D human body model is output. In addition, the termination criterion also includes a maximum iteration count protection condition.
[0054] Specifically, in this embodiment, the confidence increment is calculated in each round. With coverage .when and or reaching the maximum number of rounds The process terminates at a certain time, outputting a complete 3D human body model bound to the SMPL-X skeleton and capable of being redriven.
[0055] The incremental statistic is the average absolute value of the unit-by-unit difference in the confidence field, and the coverage rate is the proportion of basic units with a confidence level higher than the first threshold to all basic units.
[0056] Based on the above technical solution, in this embodiment, the observation data consists of 2–5 photographs from different perspectives, some of which are mutually occluded or truncated. The basic unit is replaced with a voxelized SDF feature mesh (typical resolution). The visibility metric is calculated based on the percentage of visible viewpoints. Geometric consistency is replaced with the SMPL surface distance field. The generation model is replaced with a multi-view consistent image diffusion model to generate surrounding pseudo-viewpoints in one go. The fusion scheduling, closed-loop update and termination determination steps are the same as in Example 1.
[0057] The results were validated on synthetic occlusion datasets (THuman rendering + procedural random occlusions, occlusion rate 20%–60%) and real occlusion videos. Compared to the global generation-guided baseline without a confidence mechanism, the perceptual quality metrics (LPIPS, FID) of occlusion region completion were significantly improved, and the visible region PSNR increased by approximately 0.8 dB, demonstrating that the observation protection mechanism effectively eliminated generation contamination. Compared to the fixed 8-round iteration strategy, the average number of iterations was reduced to 5.4 rounds, the reconstruction quality remained the same, and the number of rounds showed a monotonically adaptive change with the occlusion rate (terminating after 3 rounds for lightly occluded samples and using all 8 rounds for heavily occluded samples), validating the adaptability of the termination criterion.
[0058] Example 2 This invention also discloses an occlusion-aware confidence-guided generation and reconstruction device for closed-loop 3D human body reconstruction, such as... Figure 6 As shown, it includes: The observation acquisition and initialization module 401 is used to acquire at least one image or video sequence of the target human body that includes occlusion or incomplete observation as observation data, and to construct an initial three-dimensional representation of the target human body based on the observation data. The initial three-dimensional representation is composed of multiple basic units, which are three-dimensional Gaussian units or voxel units. The confidence field estimation module 402 is used to calculate the observation confidence for each of the basic units, thereby obtaining a unit-by-unit confidence field corresponding to the initial three-dimensional representation; wherein, as shown in the figure... Figure 2 As shown, the observation confidence is determined by fusing at least the visibility metric of the basic unit relative to the observation data, the photometric consistency metric across viewpoints or frames, and the geometric consistency metric with the parameterized human prior model. In the confidence field estimation module, the visibility metric is determined by projecting the basic unit to each observation viewpoint and statistically analyzing the proportion of viewpoints where it is not occluded and falls within the effective human body region of the image. The occlusion judgment is jointly implemented using a rendering depth buffer and a human body segmentation mask. The geometric consistency metric is determined by calculating the signed distance from the basic unit to the surface of the skinned parameterized human body model and mapping it to a consistency score using a distance decay function. The fusion determination is implemented using a learnable gating network, which takes the visibility metric, the photometric consistency metric, the geometric consistency metric, and the local features of the basic unit as inputs and outputs a normalized observation confidence.
[0059] The fusion scheduling module 403 is used to perform spatial adaptive optimization of the 3D representation based on the unit-by-unit confidence field through the fusion scheduler: for basic units with observation confidence higher than a first threshold, optimization is mainly performed using the reconstruction loss constructed from the observation data, and the modification of the unit by the generative model is suppressed; for basic units with observation confidence lower than a second threshold, the pre-trained generative model is invoked to infer pseudo-observations based on the rendering results of the current 3D representation and the prior human conditions, and the generation guidance loss constructed from the pseudo-observations is mainly used for completion; wherein the reconstruction loss weight and the generation guidance loss weight are both continuously and adaptively modulated in space with the observation confidence, so as to achieve a smooth transition between the two types of regions; The closed-loop update module 404 is used to re-render the 3D representation optimized by the fusion scheduling module, update the unit confidence field based on the consistency between the rendering result and the observation data and pseudo-observations, and trigger the fusion scheduling module to execute again, forming a closed loop of alternating generation and reconstruction. When updating the confidence field, for basic units whose rendering results are stable and consistent across iterations after generation and completion, their confidence is increased according to the stability score, and the upper limit of the increased confidence is lower than the upper limit of the confidence that can be reached in the directly observed area.
[0060] The termination determination and output module 405 is used to calculate the termination criterion after each iteration. The termination criterion includes at least the incremental statistics and confidence coverage of the confidence field in two adjacent iterations. When the incremental statistics are lower than the incremental threshold and the coverage is higher than the coverage threshold, the iteration is terminated and a complete three-dimensional human body model is output.
[0061] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.
Claims
1. A method for generating and reconstructing closed-loop 3D human body reconstruction guided by occlusion perception confidence, characterized in that, Includes the following steps: Step S1: Obtain an image or video sequence containing the occluded target human body as observation data, and construct an initial three-dimensional representation of the target human body based on the observation data. The initial three-dimensional representation is composed of multiple basic units. Step S2: Calculate the observation confidence for each basic unit to obtain a unit-by-unit confidence field corresponding to the initial three-dimensional representation; the observation confidence is obtained by normalizing the local features of the basic unit's visibility measure relative to the observation data, the photometric consistency measure across viewpoints, and the geometric consistency measure with the parameterized human prior model through a learnable gating network. Step S3: The fusion scheduler performs spatial adaptive optimization of the 3D representation based on the unit-by-unit confidence field. The spatial adaptive optimization method is as follows: First, a first threshold and a second threshold are preset. For basic units with observation confidence higher than the first threshold, the reconstruction loss constructed from the observation data is used for optimization. For basic units with observation confidence lower than the second threshold, the pre-trained generation model is called to infer pseudo-observations based on the rendering results of the current 3D representation and the prior human conditions. The generation guidance loss constructed from the pseudo-observations is used as the main component for completion to obtain the optimized 3D representation. Step S4: Re-render the optimized 3D representation, update the unit confidence field based on the consistency between the rendering results and the observation data and pseudo-observations, and return to step S3 to form a closed loop of alternating generation and reconstruction. After completing the iteration, output a complete 3D human body model.
2. The method according to claim 1, characterized in that, In step S2, the visibility measurement is determined by projecting basic units onto each observation viewpoint and statistically analyzing the proportion of views that are not occluded and fall into the effective human body area of the image; the occlusion determination is achieved by combining the rendering depth buffer and the human body segmentation mask. The geometric consistency metric is determined by calculating the symbolic distance from the basic unit to the surface of the skinned parametric human model and mapping it to a consistency score using a distance decay function.
3. The method according to claim 1, characterized in that, The local features of the basic unit include at least one of opacity, scale, and neighborhood density.
4. The method according to claim 1, characterized in that, The generative model is a multi-view consistent image diffusion model, and the human body prior conditions include at least one of the normal map, skeleton map, or semantic segmentation map rendered by the parameterized human body model.
5. The method according to claim 1, characterized in that, In step 3, the generation guidance loss weight of the i-th basic unit decreases monotonically as the observation confidence of the unit increases, while the reconstruction loss weight increases monotonically as the observation confidence of the unit increases.
6. The method according to claim 5, characterized in that, The weights for generating the guided loss are determined as follows: a baseline weight and a modulation index are set, and the baseline weight is multiplied by the modulation index raised to the power of the difference between the observation confidence of the unit; the reconstruction loss weight is directly taken as the observation confidence of the unit.
7. The method according to claim 4, characterized in that, The initial noise intensity during the generation of the image diffusion model is adaptively set according to the observation confidence of the target region: the lower the observation confidence, the higher the initial noise intensity.
8. The method according to claim 1, characterized in that, When updating the confidence field in step S4, for basic units whose rendering results are stable and consistent across iterations after completion, their confidence is increased according to the stability score, and the upper limit of the increased confidence is lower than the upper limit of the confidence that can be reached in the directly observed area.
9. The method according to claim 1, characterized in that, Step 4 also includes calculating a termination criterion after each iteration. The termination criterion includes the incremental statistics and confidence coverage of the confidence field in two adjacent iterations. When the incremental statistics are lower than the incremental threshold and the coverage is higher than the coverage threshold, the iteration is terminated and a complete three-dimensional human body model is output.
10. The method according to claim 9, characterized in that, The incremental statistic is the average absolute value of the unit-by-unit difference in the confidence field, and the coverage rate is the proportion of basic units with a confidence level higher than the first threshold to all basic units.
11. The method according to claim 10, characterized in that, The termination criterion also includes a maximum iteration round protection condition.
12. The method according to claim 1, characterized in that, The observation data is a dynamic video sequence, the basic unit is bound to the skin deformation field of the parameterized human body model, and the confidence field is maintained in the gauge space and mapped to each observation frame through deformation.
13. A confidence-guided device for occlusion perception to generate and reconstruct a closed-loop three-dimensional human body, characterized in that, include: The observation acquisition and initialization module is used to acquire at least one image or video sequence of the target human body that includes occlusion or incomplete observation as observation data, and to construct an initial three-dimensional representation of the target human body based on the observation data. The initial three-dimensional representation is composed of multiple basic units. The confidence field estimation module is used to calculate the observation confidence for each of the basic units to obtain a unit-by-unit confidence field corresponding to the initial three-dimensional representation. The observation confidence is obtained by normalizing the local features of the basic unit's visibility measure relative to the observation data, the photometric consistency measure across viewpoints, and the geometric consistency measure with the parameterized human prior model through a learnable gating network. The fusion scheduling module is used to perform spatial adaptive optimization of the three-dimensional representation based on the unit-by-unit confidence field through the fusion scheduler; The closed-loop update module is used to re-render the 3D representation optimized by the fusion scheduling module, update the unit confidence field based on the consistency between the rendering result and the observation data and pseudo-observations, and trigger the fusion scheduling module to execute again, forming a closed loop of alternating generation and reconstruction. The termination judgment and output module is used to calculate the termination criterion after each iteration and output a complete 3D human body model.
14. The apparatus according to claim 13, characterized in that, The confidence field estimation module is implemented using a learnable gating network. The gating network takes visibility measure, photometric consistency measure, geometric consistency measure and local features of basic units as inputs and outputs normalized observation confidence.
15. The apparatus according to claim 14, characterized in that, In the confidence field estimation module, the visibility metric is determined by projecting basic units onto each observation viewpoint and statistically analyzing the proportion of views that are not occluded and fall into the effective human body region of the image; the occlusion determination is achieved by combining the rendering depth buffer and the human body segmentation mask.
16. The apparatus according to claim 15, characterized in that, In the confidence field estimation module, the geometric consistency metric is determined by calculating the signed distance from the basic unit to the surface of the skinned parametric human model and mapping it to a consistency score through a distance decay function.
17. The apparatus according to claim 13, characterized in that, In the closed-loop update module, when updating the confidence field, for basic units whose rendering results are stable and consistent across iterations after completion, their confidence is increased according to the stability score, and the upper limit of the increased confidence is lower than the upper limit of the confidence that can be reached in the directly observed area.
