A 3D portrait reconstruction method and system based on cascade diffusion prior and UV space physical constraint

Abstract

The present application relates to the technical field of computer vision, deep learning and graphics rendering, in particular to a 3D portrait reconstruction method and system based on cascade diffusion prior and UV space physical constraint, which adopts a parameter efficient adaptation paradigm of freezing a unified DiT backbone and a cascade LoRA adapter, and sequentially completes UV space texture completion, illumination homogenization and intrinsic material decomposition; introduces a cross-intrinsic attention mechanism to realize multi-material pixel-level spatial alignment, and executes differential physical rendering supervision to constrain material physical consistency in the UV domain. The present application realizes 100,000-level field image generalization under less than 100 3D scanning samples, and the generated 4K PBR material assets have no baking light and shadow residues, and the physical behavior is consistent under re-illumination, which is suitable for high-fidelity 3D portrait asset production in the fields of digital human creation, film special effects and virtual reality.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, deep learning and graphics rendering technology, specifically to a 3D avatar reconstruction method and system based on cascaded diffusion priors and UV spatial physical constraints. Background Technology

[0002] Monocular 3D face reconstruction technology is a core supporting technology in fields such as digital human creation, virtual reality, and film and television special effects. Its core objective is to recover high-precision three-dimensional geometric structures and physically textured materials from a single two-dimensional face image. Current mainstream technologies still suffer from technical bottlenecks such as high-frequency detail loss, texture-lighting coupling, occlusion artifacts, and insufficient generalization ability. Representative existing technologies are as follows: A single-image 3D head reconstruction method and system based on a guided diffusion model, disclosed in CN120107481B, has the following core scheme: preprocessing the single image to be reconstructed to obtain a head image; reconstructing a 3D head model and optimizing and restoring surface smooth details; weakly projecting the 3D model onto the head image under the prediction camera matrix; using image pixels as 3D point texture information to obtain a 3D head model with frontal texture; performing cylindrical UV unwrapping on the model to obtain a 2D UV texture map; predicting and completing the invisible region texture in the UV texture map using a trained guided diffusion network model; and finally back-pasting the complete texture back onto the 3D head model. While this scheme achieves 3D head texture completion in a single image, it has significant technical limitations: it only completes texture filling of occluded areas without performing illumination removal and intrinsic material decomposition operations, and the generated texture still carries the baked ambient lighting of the original image, which will cause physical conflicts when the 3D model is placed in a new lighting environment; it does not introduce physical rendering constraints in UV space, and texture artifacts and spatial inconsistencies are prone to occur in the occluded completion area; the diffusion model does not adopt an efficient parameter adaptation architecture, and is prone to pre-training prior forgetting when trained on small-scale 3D scan data, limiting its generalization ability to field images.

[0003] The method and system for 3D face reconstruction based on RGB images, disclosed in CN115393513A, has the following core scheme: acquiring an RGB image of a face, locating and calculating facial key points, cropping the image, introducing a Basel parametric face model, projecting the 3D face onto a 2D plane, performing projection normalization feature encoding, updating neural network parameters through a preset loss function, obtaining output face parameters, completing UV texture map conversion, and mapping the texture to the 3D face shape to obtain the final 3D model. This scheme achieves basic 3D reconstruction based on a traditional parametric face model, but it has significant technical limitations: its method of fitting parametric face model coefficients using a backbone neural network is limited by the linear basis dimension of the parametric model, and cannot recover nonlinear high-frequency details such as skin pores, fine wrinkles, and pigmentation; it does not introduce a large-scale pre-trained diffusion model for generation priors, resulting in insufficient texture realism and naturalness; and it does not achieve decoupling and separation of PBR materials, only generating a single texture map, which cannot support physically reusable rendering and has poor asset reusability.

[0004] In summary, existing 3D face reconstruction technologies have not yet resolved the core contradictions of data scarcity and generalization degradation, nor have they achieved decoupling of texture and lighting and physical material constraints, making it difficult to simultaneously meet the industrial-grade requirements of high-fidelity details, re-illuminated properties, and strong generalization capabilities. Summary of the Invention

[0005] The purpose of this invention is to provide a 3D head reconstruction method and system based on cascaded diffusion prior and UV spatial physical constraints, so as to solve the problems of high-frequency detail loss, texture-lighting coupling, occlusion artifacts, and poor generalization ability under small sample size in the existing monocular 3D face reconstruction technology mentioned in the background.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A 3D head reconstruction system based on cascaded diffusion priors and UV spatial physical constraints includes: The geometric reconstruction module is used to perform multi-scale fusion of local and global semantic features on the input monocular face image, regress the identity, expression, lighting parameters and camera pose of the parameterized face model, project the input image pixels into the UV space, and generate incomplete initial textures with occluded areas. The cascaded diffusion texture generation module is based on a pre-trained Diffusion Transformer (DiT) backbone network with full freeze and is configured with a cascaded switchable low-rank adaptation (LoRA) adapter. It sequentially performs UV space texture completion, illumination homogenization and intrinsic material decomposition to generate multiple sets of PBR material maps. The cross-intrinsic attention module, integrated into the DiT network structure in the intrinsic material decomposition stage, is used to realize cross-modal feature interaction and pixel-level spatial alignment constraints between multiple material branches. The UV space differential rendering supervision module constructs a differentiable micro-face shader based on the geometric prior calculated by the fixed template face, performs complete 3D light transport calculation in the 2DUV domain, generates physical constraint loss and backpropagates to optimize LoRA adapter parameters. The super-resolution enhancement module is used to perform resolution upsampling on the generated PBR material maps and combine them with the reconstructed 3D geometric model to output high-fidelity, relit 3D avatar assets.

[0007] Preferably, the texture completion and illumination homogenization stages of the cascaded diffusion texture generation module both employ a flow-matched latent diffusion framework, and the noise latent variables are constructed as follows: ; The model predicts the target velocity field by minimizing the flow matching loss, the loss function of which is: ; In the formula, The noise level is defined as follows, with a value range of [value range missing]. ; Latent encoding for the target texture; To follow a standard normal distribution Random noise; It has a mean of 0 and a covariance matrix that is the identity matrix. The standard normal distribution; For mathematical expectation operations; L2 norm operations; For the prediction function of the DiT network; Input textures for conditions.

[0008] Preferably, in the cross-intrinsic attention module, the query vector for each material branch is calculated independently, and the key vector and value vector are obtained by concatenating the features of all material branches in the batch dimension. The attention output expression is: ; In the formula, For the first The query vector for each material branch; , , These are the key vectors for the reflectivity branch, normal branch, and compact reflectivity branch, respectively. This is the total key vector obtained by concatenating across modalities; , , These are the value vectors for the reflectivity branch, normal branch, and compact reflectivity branch, respectively. This is the total value vector obtained by concatenating across modalities; The dimension of the feature vector; It is a normalized exponential function; For attention operation functions; For the first The updated output features of each material branch.

[0009] Preferably, the UV space differential rendering supervision module adopts the Cook-Torrance micro-element BRDF model, and the specular reflection term adopts the simplified GGX distribution function. The specular BRDF expression is as follows: ; The rendering loss is a weighted combination of pixel-level MSE loss and perceptual loss, expressed as: ; In the formula, The normal vector of the surface micro-element; To observe the direction vector; Let be the incident light direction vector; This is the half-range vector between the observation direction and the incident light direction; It is the surface normal vector; For roughness parameters, satisfying , This represents the predicted material roughness value. Let GGX be the normal distribution function; This is the Fresnel reflection term; For Smith's visibility; For flow matching diffusion loss; Preset weights for pixel-level mean square error loss; Preset weights for perceived loss; Render textures for the target; The rendered texture predicted by the model; To learn the LPIPS perceptual loss function for perceptual image patch similarity.

[0010] Preferably, all LoRA adapters in the cascaded diffusion texture generation module have a rank of 32, which are injected into the attention layer and feedforward projection layer of DiT. The number of parameters of a single adapter accounts for 0.7% of the DiT backbone network. During the training phase, the DiT backbone network is frozen throughout, and only the LoRA adapter parameters are optimized.

[0011] On the other hand, the present invention also provides a 3D head reconstruction method based on the above-mentioned system, which is based on cascaded diffusion prior and UV spatial physical constraints, and includes the following steps: S1 extracts local geometric features and global semantic prior features from the input monocular face image and performs multi-scale fusion. It regresses the identity, expression, lighting and camera pose parameters of the parameterized face model, projects the input image into the UV space, and generates an incomplete initial texture with occlusion. S2 is based on a pre-trained DiT backbone network with full freeze. By switching cascaded LoRA adapters, it sequentially performs UV space texture completion and illumination homogenization on the incomplete initial texture to obtain a homogenized intermediate texture with normalized illumination. S3 is based on a cross-intrinsic attention mechanism. It performs intrinsic material decomposition on homogenized intermediate textures through a multi-branch parallel LoRA adapter to generate multiple sets of initial PBR material maps. S4 performs differentiable micro-face rendering calculations in UV space based on the geometric priors pre-computed by the fixed template face, generates physical constraint loss and backpropagates to optimize LoRA adapter parameters. S5 performs super-resolution upsampling on the optimized PBR material maps and combines them with the reconstructed 3D geometric model to output high-fidelity, relit 3D avatar assets.

[0012] Preferably, step S1 specifically includes the following steps: S1.1 uses the ConvNeXt V2 backbone network to extract local geometric features of the input monocular face image, and at the same time uses a frozen DINOv3 encoder to extract global semantic prior features of the input image. S1.2 performs multi-scale spatial fusion of local geometric features and global semantic prior features to regress the identity coefficient, expression coefficient, spherical harmonic illumination coefficient, and perspective camera pose of the Hifi3D++ parametric facial model. S1.3 Based on the geometric model obtained from regression and the camera pose, the visible skin region pixels of the input image are projected onto the predefined UV space through a differentiable renderer to generate an incomplete initial texture with occlusion gaps.

[0013] Preferably, step S2 specifically includes the following steps: S2.1 loads the texture completion LoRA adapter, takes an incomplete initial texture as conditional input, predicts the pixel distribution of the occluded area based on the flow matching latent diffusion framework, and generates a visually complete completed texture with ambient lighting. S2.2 Switches to the LoRA adapter for homogenizing lighting, using texture completion as the conditional input, eliminates specular highlights and ambient shadows in the texture, maps the texture to a unified neutral lighting space, and generates a homogenized intermediate texture.

[0014] Preferably, step S3 specifically includes the following steps: S3.1 simultaneously loads the reflectivity LoRA adapter, the normal LoRA adapter, and the compact reflectivity LoRA adapter, using homogenized intermediate texture as common input, and extracts the features of the corresponding materials respectively. S3.2 In each Transformer block of the DiT network, a cross-intrinsic attention mechanism is enabled. Each material branch generates a query vector independently. The features of all material branches are concatenated to generate a key vector and a value vector, thus completing cross-modal feature interaction. S3.3 The VAE decoder decodes the features of each branch, generating an initial reflectivity map, a normal map, and a compact reflectivity map that includes roughness, specular reflection, and displacement maps.

[0015] Preferably, step S4 specifically includes the following steps: S4.1 Load the pre-calculated UV space world coordinate position map, geometric normal map and geometric tangent map of the fixed template face as geometric priors; S4.2 updates the pixel world position along the geometric normal of the predicted displacement map, performs Gram-Schmidt reorthogonalization on the tangent space normal, and constructs a corrected local shading framework; S4.3 performs differentiable GGX shading calculations based on the Cook-Torrance micro-facet model, generates rendering textures, calculates the blending loss between the rendering texture and the target texture, and optimizes the parameters of each LoRA adapter through backpropagation.

[0016] Compared with the prior art, the beneficial effects of the present invention are: (1) This invention adopts a parameter-efficient adaptation paradigm that combines a frozen unified backbone with cascaded low-rank adaptive adapters, effectively solving the technical contradictions in the prior art where fully parameter-tuned diffusion models are prone to forgetting pre-trained generation priors and generative adversarial networks are prone to pattern collapse and over-smoothing. This invention carries a large-scale open-domain generation prior by sharing a single frozen diffusion Transformer backbone, and switches lightweight low-rank adaptive adapters between different sub-tasks such as texture completion, illumination homogenization, and material decomposition. With a training data scale of less than a hundred real 3D scan samples, it retains the generalization ability for hundreds of thousands of outdoor face images; at the same time, the number of parameters of a single adapter accounts for only 0.7% of the backbone network, which is two orders of magnitude lower than the number of parameters of independently trained multiple complete diffusion models. The open-domain priors are shared between sub-tasks without interference, breaking through the limitation of the scarcity of high-quality physically rendered material data on the training of large-scale neural networks from the architectural level.

[0017] (2) This invention addresses the fundamental shortcomings of existing 2D screen-space rasterization supervision, which is prone to introducing occlusion artifacts and projection errors, as well as the instability of training and inconsistent material physical properties caused by the excessively large solution space of single-step inverse rendering, by using UV space differential physical rendering supervision and explicit lighting homogenization intermediate representation. This invention directly performs complete 3D light transmission calculation in the UV domain based on a fixed template geometric prior, avoiding the problem of hair, glasses and other occluders being incorrectly baked into the skin material. The fixed geometric prior eliminates the dependence on specific geometry for each sample, and the physical supervision process is stable and reproducible. At the same time, by explicitly inserting the intermediate stage of lighting homogenization, the originally unsuitable single-texture to multi-material inverse rendering problem is decoupled into two well-state sub-problems: lighting removal and intrinsic decomposition. This significantly constrains the solution space, accelerates the training convergence speed, and the generated materials have consistent physical behavior after changing the lighting environment, without any residual light and shadow from the original environment.

[0018] (3) This invention effectively solves the technical problems of high-frequency detail spatial misalignment and modality-specific overfitting in existing multi-material independent generation schemes by using a cross-intrinsic attention mechanism and a two-stage progressive training strategy. This invention achieves feature interaction and constraint between different material branches by cross-modal splicing of key-value vectors of multiple material branches in the batch dimension, ensuring that high-frequency details such as wrinkles, pores, and pigmentation are precisely aligned at the pixel level in multiple material maps such as reflectivity, normals, and roughness, effectively suppressing the problems of fragmented artifacts in normal maps, geometric undulations, and texture color misalignment; combined with the rendering sampling strategy of mixed perspective and lighting and the multi-source heterogeneous data collaborative training scheme, the model has a stable reconstruction effect for extreme poses, complex occlusions, and people with different skin colors, and the generated high-resolution physical material assets can be directly adapted to various industrial-grade physical rendering engines. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are explained in detail together with the embodiments of the invention, but do not constitute a limitation thereof.

[0020] Figure 1 This is a diagram showing the system module architecture of the present invention; Figure 2 This is a flowchart providing an overview of the method of the present invention; Figure 3 This is a complete processing flow diagram of the present invention, from inputting a single outdoor face image to outputting a 4K high-fidelity 3D asset; Figure 4 This is a schematic diagram of the synthetic data generation pipeline of the present invention; Figure 5 This is a diagram illustrating the cross-intrinsic attention mechanism and training architecture of the present invention. Detailed Implementation

[0021] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0022] like Figures 1-5 As shown, the end-to-end 3D avatar reconstruction framework provided by this invention relies on a frozen pre-trained diffusion Transformer backbone network to complete texture completion, illumination homogenization and intrinsic material decomposition through cascaded switching of low-rank adaptive adapters. Combined with UV space differential physical rendering constraints, it realizes the reconstruction from monocular face images to high-fidelity re-illuminated 3D avatar assets.

[0023] Geometric Reconstruction and UV Mapping Preprocessing: First, a monocular field face image is received as input. The ConvNeXtV2 backbone network is used to extract local geometric features from the input image, while a frozen DINOv3 encoder is used to extract global semantic prior features. After multi-scale spatial fusion of the local and global features, the corresponding parameters of the Hifi3D++ parametric face model are regressed. The parameter vector is defined as... ,in For identity coefficient, the dimension is ; The expression coefficient has the following dimensions: ; The texture coefficient has a dimension of . ; The spherical harmonic illumination coefficient has the following dimensions: ; The perspective camera pose is used. The 3D shape and basic texture are modeled using linear principal component analysis, expressed as: ; ; in For average shape, For average texture, Let be the identity basis matrix. For the expression base matrix, This is the texture basis matrix.

[0024] Based on the geometric model obtained from regression and the camera pose The visible skin region pixels of the original image are projected onto a predefined UV space using a differentiable renderer, generating an incomplete initial texture with missing regions. .

[0025] Cascaded texture restoration and illumination homogenization: A pre-trained diffusion Transformer with full-time freezing is used as a shared backbone network. All low-rank adaptive adapters have a rank of 32, and the number of parameters of a single adapter accounts for 0.7% of the total number of parameters of the backbone network. During the training phase, only the adapter parameters are optimized, and the weights of the backbone network remain fixed.

[0026] First, the texture completion process is performed, loading the low-rank texture completion adaptor with an incomplete initial texture. As conditional inputs, a flow-matched latent diffusion framework is used for generation. The noise latent variables are constructed as follows: ; in The noise level is defined as follows, with a value range of [value range missing]. ; Latent encoding for the target texture; To follow a standard normal distribution The model predicts the target velocity field by minimizing the flow matching loss, where the loss function is... ,in For the prediction function of the diffusion Transformer network, Input textures as conditions. For mathematical expectation calculation, This refers to L2 norm operations.

[0027] After texture completion, a visually complete texture with ambient lighting is generated. Switch the loading of the low-rank adaptation adapter for lighting homogenization to complete the texture. As a conditional input, specular highlights and ambient shadows in the texture are eliminated, and the texture is mapped to a unified neutral lighting space to generate a homogenized intermediate texture. .

[0028] Material estimation based on cross-eigenattention: Simultaneously loading a low-rank reflectivity adaptor, a low-rank normal adaptor, and a compact low-rank reflectivity adaptor to homogenize intermediate textures. As a common input, features corresponding to the material are extracted. In each Transformer block of the Diffusion Transformer, a cross-intrinsic attention mechanism is enabled to achieve cross-modal feature interaction.

[0029] The output expression for cross-intrinsic attention is: ;in For the first The query vector for each material branch; The total key vector obtained by cross-modal splicing is composed of key vectors from reflectivity branch, normal branch, and compact reflectivity branch. The total value vector obtained by cross-modal concatenation is composed of the concatenation of value vectors from three branches. The dimension of the feature vector; It is a normalized exponential function.

[0030] The variational autoencoder decoder decodes the features of each branch to generate an initial reflectivity map, a normal map, and a compact reflectivity map that includes roughness, specular reflection, and displacement maps.

[0031] Supervised optimization of UV space differential rendering: A pre-computed UV space world coordinate map, geometric normal map, and geometric tangent map of a fixed template face are loaded as geometric priors. The predicted displacement map is updated along the geometric normals to update the pixel world position, with the expression: ; in For normalized geometric normals, This is the displacement scaling factor, with a value of 0.01.

[0032] A Gram-Schmidt reorthogonalization is performed on the tangent space normals to construct a modified local shading framework. The Cook-Torrance micro-element bidirectional reflection distribution function model is used for shading calculations. The expression for the specular bidirectional reflection distribution function is: ; in The normal vector of the surface micro-element; To observe the direction vector; Let be the incident light direction vector; This is the half-range vector between the observation direction and the incident light direction; It is the surface normal vector; For roughness parameters, satisfying ; Let GGX be the normal distribution function; For Fresnel reflection; For Smith's visibility term.

[0033] The rendering loss is a weighted combination of pixel-level mean squared error loss and perceptual loss, expressed as: ; in For flow matching diffusion loss; The pixel-level mean square error loss weight is set to 0.5. The perceived loss weight is set to 0.1.

[0034] Super-resolution enhancement and asset output: The generated 1K resolution material maps are upsampled fourfold using a finely tuned Real-ESRGAN network to obtain 4K resolution reflectance maps, normal maps, roughness maps, specular reflection maps, and displacement maps. The upsampled material maps are then assembled with the reconstructed 3D geometry model to generate a high-fidelity, relit 3D avatar asset that can be directly used in a physically based rendering engine.

[0035] Example 1: Standard Unobstructed Frontal Face 3D Reconstruction This embodiment uses an unobstructed frontal Caucasian face image with a resolution of 512×512 from the FFHQ dataset as input.

[0036] First, a geometric reconstruction step is performed. The ConvNeXtV2 backbone network extracts local features, and the frozen DINOv3 encoder extracts global semantic features. After multi-scale fusion, the Hifi3D++ model parameters are regressed, with the identity coefficient dimension being 532, the expression coefficient dimension being 45, the spherical harmonic illumination coefficient dimension being 9, and the camera pose corresponding to a pitch angle of 0.02 rad, a yaw angle of 0.01 rad, and a roll angle of 0 rad. The visible skin region pixels are projected onto a 1024×1024 resolution UV space to generate an incomplete initial texture, in which the self-occluded region accounts for 8.7%.

[0037] Entering the cascaded diffusion texture generation stage, the shared backbone network uses a LongCat-Image-Edit pre-trained diffusion Transformer with weights frozen throughout. First, a low-rank adaptive adapter for texture completion is loaded, with a sampling step count of 30 and a guiding scale of 2.0. Under the flow matching framework, the noise level linearly decreases from 1 to 0. The flow matching loss formula for this stage is simply substituted into the calculation: taking the noise level of the current step... Target texture latent encoding Random noise The target velocity field is The model predicts a velocity field of 0.547. Substituting this into the loss formula, the single-step flow matching loss is: ; Complete the occlusion area completion, generating a complete texture with ambient lighting. Switch the loading of the low-rank adaptation adapter for lighting homogenization, eliminate the influence of the original ambient light under the same sampling parameters, and obtain a homogenized intermediate texture under neutral lighting. The overall brightness standard deviation of the texture decreases from 0.21 to 0.08, completing the lighting normalization.

[0038] Entering the material estimation stage, low-rank adaptive adapters for three material branches are loaded simultaneously, and a cross-intrinsic attention mechanism is enabled. Features from each branch are stitched together in the batch dimension to complete cross-modal interaction. After decoding, a 1K resolution reflectance map, normal map, and compact reflectance map are generated, where the roughness map has a value range of 0.3 to 0.8, the specular reflectance map has a value range of 0.2 to 0.6, and the displacement map has a value range of -0.005 to 0.005.

[0039] UV space differential rendering supervision is performed. Fixed template geometry prior calculations are completed, the displacement scaling factor is set to 0.01, tangent space re-orthogonalization is performed, and a simplified GGX micro-facet model is used for shading. A hybrid sampling strategy is employed, sampling the 360-degree azimuth view with a probability of 0.5, and random perturbation of directional light. The total loss formula for this stage is simply substituted into the calculation: taking the current iteration flow matching diffusion loss... The L2 norm squared of the predicted rendered texture and the target texture is 0.0352, and the LPIPS perceptual loss is 0.078. Substituting these values ​​into the loss formula, the total loss is: Backpropagation of the loss optimizes the adapter parameters.

[0040] Finally, super-resolution enhancement was performed, upsampling all material textures from 1K to 4K, with a single step taking 2 seconds. In the REALY benchmark test, the final output 3D avatar asset had a normalized mean square error of 1.487 mm for the four facial regions, with no baked shadows remaining under heavy lighting, and skin pores and wrinkle details were clearly discernible.

[0041] Example 2: Large-pose 3D face reconstruction with occlusion This embodiment uses a 480×640 resolution image of a person wearing glasses turned to the side, collected in the field, as input. The yaw angle of the face is 42 degrees, and the left side of the face is obscured by the glasses frame and hair, with the obscured area accounting for 21.3% of the total face area.

[0042] In the geometric reconstruction stage, the multi-scale spatial fusion encoder, combined with the strong semantic prior of DINOv3, accurately regresses identity and expression parameters under large poses. The mouth region employs an 88-point hybrid keypoint loss constraint, reducing expression estimation error by 18.2% compared to the standard 68-point scheme. The corresponding 3D shape formula for this stage is simply substituted into the calculation: taking the average shape coordinates corresponding to the tip of the nose. Identity basis matrix corresponding terms The current sample identity coefficient corresponds to the item The corresponding terms of the expression base matrix Item corresponding to the expression coefficient Substituting into the shape formula, the final coordinates of the tip of the nose are: ; Vertex geometry reconstruction is complete. After projection into UV space, continuous gaps are formed in the areas occluded by the eyeglass frame and hair, with the gaps in the incomplete initial texture accounting for 27.6%.

[0043] In the texture completion stage, the low-rank adaptive adapter automatically infers the skin texture of the occluded area based on facial symmetry and semantic priors. The corresponding area of ​​the glasses frame generates a continuous and consistent cheek skin texture without obvious stitching artifacts. The completed texture has highlights and shadows caused by the strong ambient light on the left. In the illumination homogenization stage, the influence of non-uniform illumination is eliminated, and the brightness difference between the left and right cheeks in the homogenized texture is reduced from 0.32 to 0.04.

[0044] During the material estimation stage, low-rank adaptive adapters for three material branches are loaded simultaneously, and a cross-intrinsic attention mechanism is enabled to constrain the spatial alignment of the multi-material graphs. The corresponding cross-intrinsic attention formula for this stage is simply substituted into the calculation: taking the feature dimension... Normal branch query vector Cross-modal concatenation of key vectors The result of calculating the dot product is: , divided by The unnormalized weights are obtained, and after softmax operation, attention weights are obtained. These attention weights are then multiplied by the value vector to obtain the updated normal branch features. The final generated normal map has no fragmented artifacts, and the color changes and geometric undulations corresponding to wrinkles are perfectly aligned at the pixel level.

[0045] UV space differential rendering supervision avoids occlusion artifacts from 2D rasterization, and the color and texture of the eyeglass frame are not incorrectly baked into the skin material. The final generated 4K 3D avatar asset renders without geometric distortion at a 60-degree side-rotation viewpoint, and the lighting transition is natural after changing to HDR ambient light. The material consistency between the occluded area and the surrounding skin reaches 98.7%, making it suitable for direct use in film-grade asset production.

[0046] Example 3: Cross-skin color outdoor face generalization reconstruction This embodiment uses 512×512 resolution African American outdoor face images from the CelebAMask-HQ dataset as input to verify the model's cross-skin color generalization ability.

[0047] During the model training phase, a multi-source heterogeneous data collaboration strategy was adopted, combining 100,000 outdoor 2D face images with 92 high-quality 3D scan textures. Using the BlenderCycles rendering engine, 100,000 synthetic training samples were generated with 2041 HDR environment maps, eliminating the need for additional real-world physically rendered data. During training, the low-rank material adaptation adapter employed a two-stage training strategy. The first stage involved 10,000 independent training steps, establishing basic material mapping capabilities for each branch. The second stage involved 10,000 joint training steps, enabling cross-attention and differential rendering constraints. The total training steps were 20,000, with a batch size of 8.

[0048] During the inference phase, the geometric reconstruction module accurately regresses the identity and expression parameters of dark-skinned faces without skin color bias. Entering the cascaded diffusion pipeline, the formula for the latent noise variable in this stage is simply substituted into the calculation: taking the noise level from step 10 of the texture completion stage. Target texture latent encoding Random noise Substituting into the formula, we get the latent noise variable for the current step as: Based on this latent variable, diffusion sampling is performed to generate the corresponding texture. The final generated reflectance map accurately reproduces the pigment distribution of dark skin, and the normal map clearly presents facial texture without oversmoothing artifacts.

[0049] During the UV spatial physical constraint stage, surface position update calculations are performed. The corresponding displacement update formula for this stage is simply substituted into the calculation: take the original world position of a certain pixel. Normalized geometric normal Predicted displacement map corresponding values Displacement scaling factor Substituting into the formula, the updated pixel world position is: ; After updating the surface position, perform tangent space orthogonalization and shading calculations to ensure that the material conforms to the rendering equation and that the specular reflection intensity and skin roughness conform to the real physical properties.

[0050] Quantitative test results show that the African American face assets reconstructed in this embodiment achieved a texture realism preference rate of 86.7% and a relighting quality preference rate of 90% in user studies, with a normalized mean square error of 1.492 mm. The difference in accuracy between the reconstructed face assets and those of Caucasian faces is less than 0.5%, demonstrating excellent cross-skin color generalization ability.

[0051] This invention employs a parameter-efficient adaptation paradigm combining a frozen unified backbone and cascaded low-rank adaptive adapters, effectively resolving the technical contradictions in existing technologies, such as the tendency of fully parameter-tuned diffusion models to forget pre-trained generation priors and the tendency of generative adversarial networks to suffer from pattern collapse and oversmoothing. This invention uses a shared, single frozen diffusion Transformer backbone to carry large-scale open-domain generation priors, switching between lightweight low-rank adaptive adapters across different subtasks such as texture completion, illumination homogenization, and material decomposition. With a training data scale of less than a hundred real 3D scan samples, it retains the generalization ability for hundreds of thousands of outdoor face images. Simultaneously, the number of parameters for a single adapter accounts for only 0.7% of the backbone network, reducing the number of parameters by two orders of magnitude compared to independently training multiple complete diffusion models. The open-domain priors are shared between subtasks without interference, overcoming the limitation imposed on large-scale neural network training by the scarcity of high-quality physically rendered material data at the architectural level.

[0052] This invention addresses the fundamental shortcomings of existing 2D screen-space rasterization supervision, which is prone to introducing occlusion artifacts and projection errors, as well as the instability of training and inconsistent material physical properties caused by the excessively large solution space in single-step inverse rendering, through UV-space differential physical rendering supervision and explicit lighting homogenization intermediate representation. This invention directly performs complete 3D light transmission calculations within the UV domain based on a fixed template geometric prior, avoiding the problem of occluders such as hair and glasses being incorrectly baked into the skin material. The fixed geometric prior eliminates the dependence on sample-by-sample specific geometry, making the physical supervision process stable and reproducible. Simultaneously, by explicitly inserting an intermediate stage of lighting homogenization, the originally unsuitable single-texture to multi-material inverse rendering problem is decoupled into two well-conditioned sub-problems: lighting removal and intrinsic decomposition. This significantly constrains the solution space, accelerates training convergence, and ensures that the generated materials exhibit consistent physical behavior after changing the lighting environment, without any residual lighting from the original environment.

[0053] This invention effectively solves the technical problems of high-frequency detail spatial misalignment and modality-specific overfitting in existing multi-material independent generation schemes by using a cross-intrinsic attention mechanism and a two-stage progressive training strategy. By batch-level cross-modal concatenation of key-value vectors from multiple material branches, this invention achieves feature interaction and constraints between different material branches, ensuring pixel-level precise alignment of high-frequency details such as wrinkles, pores, and pigmentation across multiple material maps including reflectivity, normals, and roughness. This effectively suppresses issues such as fragmented artifacts in normal maps, geometric undulations, and texture color misalignment. Combined with a mixed-viewpoint and lighting rendering sampling strategy and a multi-source heterogeneous data collaborative training scheme, the model exhibits stable reconstruction results for extreme poses, complex occlusions, and people with different skin tones. The generated high-resolution physically based material assets can be directly adapted to various industrial-grade physical rendering engines.

[0054] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A 3D head reconstruction system based on cascaded diffusion priors and UV spatial physical constraints, characterized in that, include: The geometric reconstruction module is used to perform multi-scale fusion of local and global semantic features on the input monocular face image, regress the identity, expression, lighting parameters and camera pose of the parameterized face model, project the input image pixels into the UV space, and generate incomplete initial textures with occluded areas. The cascaded diffusion texture generation module is based on a pre-trained DiT backbone network with full freeze and is configured with a cascaded switchable low-rank LoRA adapter. It sequentially performs UV space texture completion, illumination homogenization and intrinsic material decomposition to generate multiple sets of PBR material maps. The cross-intrinsic attention module, integrated into the DiT network structure in the intrinsic material decomposition stage, is used to realize cross-modal feature interaction and pixel-level spatial alignment constraints between multiple material branches. The UV space differential rendering supervision module constructs a differentiable micro-face shader based on the geometric prior calculated by the fixed template face, performs complete 3D light transport calculation in the 2DUV domain, generates physical constraint loss and backpropagates to optimize LoRA adapter parameters. The super-resolution enhancement module is used to perform resolution upsampling on the generated PBR material maps and combine them with the reconstructed 3D geometric model to output high-fidelity, relit 3D avatar assets.

2. The 3D head reconstruction system based on cascaded diffusion prior and UV spatial physical constraints according to claim 1, characterized in that, The texture completion and illumination homogenization stages of the cascaded diffusion texture generation module both employ a flow-matched latent diffusion framework, and the noise latent variables are constructed as follows: ； The model predicts the target velocity field by minimizing the flow matching loss, the loss function of which is: ； In the formula, The noise level is defined as follows, with a value range of [value range missing]. ; Latent encoding for the target texture; To follow a standard normal distribution Random noise; It has a mean of 0 and a covariance matrix that is the identity matrix. The standard normal distribution; For mathematical expectation operations; L2 norm operations; For the prediction function of the DiT network; Input textures for conditions.

3. The 3D head reconstruction system based on cascaded diffusion prior and UV spatial physical constraints according to claim 1, characterized in that, In the cross-intrinsic attention module, the query vector for each material branch is calculated independently. The key vector and value vector are obtained by concatenating the features of all material branches in the batch dimension. The attention output expression is: ； In the formula, For the first The query vector for each material branch; , , These are the key vectors for the reflectivity branch, normal branch, and compact reflectivity branch, respectively. This is the total key vector obtained by concatenating across modalities; , , These are the value vectors for the reflectivity branch, normal branch, and compact reflectivity branch, respectively. This is the total value vector obtained by concatenating across modalities; The dimension of the feature vector; It is a normalized exponential function; For attention operation functions; For the first The updated output features of each material branch.

4. The 3D head reconstruction system based on cascaded diffusion prior and UV spatial physical constraints according to claim 1, characterized in that, The UV space differential rendering supervision module adopts the Cook-Torrance micro-facet BRDF model, and the specular reflection term uses the simplified GGX distribution function. The specular BRDF expression is as follows: ； The rendering loss is a weighted combination of pixel-level MSE loss and perceptual loss, expressed as: ； In the formula, The normal vector of the surface micro-element; To observe the direction vector; Let be the incident light direction vector; This is the half-range vector between the observation direction and the incident light direction; It is the surface normal vector; For roughness parameters, satisfying , This represents the predicted material roughness value. Let GGX be the normal distribution function; This is the Fresnel reflection term; For Smith's visibility; For flow matching diffusion loss; Preset weights for pixel-level mean square error loss; Preset weights for perceived loss; Render textures for the target; The rendered texture predicted by the model; To learn the LPIPS perceptual loss function for perceptual image patch similarity.

5. The 3D head reconstruction system based on cascaded diffusion prior and UV spatial physical constraints according to claim 1, characterized in that, All LoRA adapters in the cascaded diffusion texture generation module have a rank of 32 and are injected into the attention layer and feedforward projection layer of DiT. The number of parameters of a single adapter accounts for 0.7% of the DiT backbone network. During the training phase, the DiT backbone network is frozen throughout, and only the LoRA adapter parameters are optimized.

6. A 3D head reconstruction method based on cascaded diffusion priors and UV spatial physical constraints according to the system of claim 1, characterized in that, Includes the following steps: S1 extracts local geometric features and global semantic prior features from the input monocular face image and performs multi-scale fusion. It regresses the identity, expression, lighting and camera pose parameters of the parameterized face model, projects the input image into the UV space, and generates an incomplete initial texture with occlusion. S2 is based on a pre-trained DiT backbone network with full freeze. By switching cascaded LoRA adapters, it sequentially performs UV space texture completion and illumination homogenization on the incomplete initial texture to obtain a homogenized intermediate texture with normalized illumination. S3 is based on a cross-intrinsic attention mechanism. It performs intrinsic material decomposition on homogenized intermediate textures through a multi-branch parallel LoRA adapter to generate multiple sets of initial PBR material maps. S4 performs differentiable micro-face rendering calculations in UV space based on the geometric priors pre-computed by the fixed template face, generates physical constraint loss and backpropagates to optimize LoRA adapter parameters. S5 performs super-resolution upsampling on the optimized PBR material maps and combines them with the reconstructed 3D geometric model to output high-fidelity, relit 3D avatar assets.

7. The 3D head reconstruction method based on cascaded diffusion prior and UV spatial physical constraints according to claim 6, characterized in that, Step S1 specifically includes the following steps: S1.1 uses the ConvNeXt V2 backbone network to extract local geometric features of the input monocular face image, and at the same time uses a frozen DINOv3 encoder to extract global semantic prior features of the input image. S1.2 performs multi-scale spatial fusion of local geometric features and global semantic prior features to regress the identity coefficient, expression coefficient, spherical harmonic illumination coefficient, and perspective camera pose of the Hifi3D++ parametric facial model. S1.3 Based on the geometric model obtained from regression and the camera pose, the visible skin region pixels of the input image are projected onto the predefined UV space through a differentiable renderer to generate an incomplete initial texture with occlusion gaps.

8. The 3D head reconstruction method based on cascaded diffusion prior and UV spatial physical constraints according to claim 6, characterized in that, Step S2 specifically includes the following steps: S2.1 loads the texture completion LoRA adapter, takes an incomplete initial texture as conditional input, predicts the pixel distribution of the occluded area based on the flow matching latent diffusion framework, and generates a visually complete completed texture with ambient lighting. S2.2 Switches to the LoRA adapter for homogenizing lighting, using texture completion as the conditional input, eliminates specular highlights and ambient shadows in the texture, maps the texture to a unified neutral lighting space, and generates a homogenized intermediate texture.

9. The 3D head reconstruction method based on cascaded diffusion prior and UV spatial physical constraints according to claim 6, characterized in that, Step S3 specifically includes the following steps: S3.1 simultaneously loads the reflectivity LoRA adapter, the normal LoRA adapter, and the compact reflectivity LoRA adapter, using homogenized intermediate texture as common input, and extracts the features of the corresponding materials respectively. S3.2 In each Transformer block of the DiT network, a cross-intrinsic attention mechanism is enabled. Each material branch generates a query vector independently. The features of all material branches are concatenated to generate a key vector and a value vector, thus completing cross-modal feature interaction. S3.3 The VAE decoder decodes the features of each branch, generating an initial reflectivity map, a normal map, and a compact reflectivity map that includes roughness, specular reflection, and displacement maps.

10. The 3D head reconstruction method based on cascaded diffusion prior and UV spatial physical constraints according to claim 6, characterized in that, Step S4 specifically includes the following steps: S4.1 Load the pre-calculated UV space world coordinate position map, geometric normal map and geometric tangent map of the fixed template face as geometric priors; S4.2 updates the pixel world position along the geometric normal of the predicted displacement map, performs Gram-Schmidt reorthogonalization on the tangent space normal, and constructs a corrected local shading framework; S4.3 performs differentiable GGX shading calculations based on the Cook-Torrance micro-facet model, generates rendering textures, calculates the blending loss between the rendering texture and the target texture, and optimizes the parameters of each LoRA adapter through backpropagation.

Images