A method and system for occluded face three-dimensional rapid reconstruction based on diffusion model

Abstract

The application provides a kind of based on diffusion model's occluded face three-dimensional fast reconstruction method and system, it is related to three-dimensional face reconstruction technical field, comprising: obtaining monocular two-dimensional face image containing occlusion, extracting image identity feature therefrom, obtaining identity condition by normalization;According to identity condition, use diffusion model to accelerate sampling and impose spectral self-adaptive projection constraint on ordinary differential equation, generate candidate solution set;Multi-modal feature fusion and candidate difference relationship amplification are carried out on candidate solution set and identity condition, and the normalized score of each candidate solution is obtained;The candidate solution with the highest score is taken as the global optimal solution, and the three-dimensional face grid is synthesized again according to the parameter;The application realizes low delay, high fidelity and optimal solution accurate retrieval ability of three-dimensional face geometry reconstruction in the scene of serious occlusion.

Description

Technical Field

[0001] This invention relates to the field of 3D face reconstruction technology, specifically to a method and system for rapid 3D reconstruction of occluded faces based on a diffusion model. Background Technology

[0002] 3D face reconstruction technology aims to recover the three-dimensional geometric structure of the face from two-dimensional images. It is a core supporting technology in fields such as face recognition, virtual reality, augmented reality, digital human-driven systems, and medical aesthetics. With the development of deep learning, methods based on convolutional neural networks (CNNs) and parametric models (such as FLAME) can achieve end-to-end reconstruction from monocular images to 3D faces. However, in real-world open environments, input images often lack key facial region information due to occlusions such as masks, sunglasses, hair, or hands. This makes it difficult for reconstruction models that rely solely on visual features to recover complete and accurate facial geometry, resulting in significant distortion or voids in the reconstruction results in the occluded areas.

[0003] To address the occlusion problem, existing research attempts to introduce generative diffusion models into 3D face reconstruction tasks, leveraging their powerful data distribution fitting capabilities to generate multiple candidate 3D face meshes to complete the geometry of occluded regions. However, these methods have significant limitations: First, diffusion models rely on Markov iterative denoising processes lasting hundreds to thousands of steps, resulting in significant inference time consumption and making it difficult to meet the demands of rapid reconstruction in industrial applications. Second, existing methods lack effective filtering mechanisms for redundant occlusion information in the feature space and fail to fully capture fine-grained geometric residuals, leading to insufficient accuracy of the reconstructed mesh in local regions. Third, after generating multiple candidate solutions, an isolated point-by-point scoring strategy is typically used to select the optimal solution, failing to utilize the interactive comparison information between candidate solutions. When the number of candidates is large, a "score dilution" phenomenon easily occurs, i.e., the score boundaries between superior and inferior candidates become blurred, making it difficult for the system to accurately select the globally optimal solution, severely restricting the final reconstruction quality.

[0004] Therefore, existing technologies suffer from slow sampling speed, severe distortion in reconstruction of occluded areas, and low accuracy in candidate solution selection, making it difficult to balance high fidelity and real-time requirements. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a method and system for rapid 3D reconstruction of occluded faces based on a diffusion model. This method achieves low-latency, high-fidelity 3D face geometric reconstruction with accurate optimal solution retrieval capabilities even in severely occluded scenarios.

[0006] According to some embodiments, the present invention adopts the following technical solution: A fast 3D reconstruction method for occluded faces based on a diffusion model includes: Obtain monocular 2D face images containing occlusions, extract image identity features from them, and obtain identity conditions through normalization; Based on the identity conditions, the diffusion model is used to accelerate sampling of ordinary differential equations and apply spectral adaptive projection constraints to generate a set of candidate solutions; Multimodal feature fusion and amplification of the differences between candidates are performed on the candidate solution set and identity conditions to obtain the normalized score of each candidate solution; The candidate solution with the highest score is taken as the global optimal solution, and a 3D face mesh is synthesized again based on this parameter.

[0007] According to some embodiments, the present invention adopts the following technical solution: A fast 3D reconstruction system for occluded faces based on a diffusion model includes: The feature extraction module is configured to: acquire a monocular two-dimensional face image containing occlusion, extract image identity features from it, and obtain identity conditions through normalization; The candidate generation module is configured to: generate a set of candidate solutions by using a diffusion model to accelerate sampling of ordinary differential equations based on identity conditions and applying spectral adaptive projection constraints. The candidate solution scoring module is configured to: perform multimodal feature fusion and amplify the difference relationship between candidates on the candidate solution set and identity conditions to obtain the normalized score of each candidate solution; The face reconstruction module is configured to take the candidate solution with the highest score as the global optimal solution and then synthesize a 3D face mesh based on this parameter.

[0008] According to some embodiments, the present invention adopts the following technical solution: A computer program product includes a computer program that, when executed by a processor, implements the aforementioned method for rapid 3D reconstruction of occluded faces based on a diffusion model.

[0009] According to some embodiments, the present invention adopts the following technical solution: A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the aforementioned method for rapid 3D reconstruction of occluded faces based on a diffusion model.

[0010] According to some embodiments, the present invention adopts the following technical solution: An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the aforementioned method for rapid 3D reconstruction of occluded faces based on a diffusion model.

[0011] Compared with the prior art, the beneficial effects of the present invention are as follows: 1) A TIE generation mechanism is proposed to address the mismatch between isotropic sampling and prior knowledge of 3D face deformation during the diffusion process, as well as the slow generation speed. This mechanism introduces SA-HEP, which stabilizes and accelerates the sampling process of ordinary differential equations by constructing dimension-adaptive geometric constraints. Combined with a solver, this invention can achieve high-fidelity, low-latency 3D geometric generation of occluded faces with only 20 sampling steps.

[0012] 2) An InterCompScoreNet is proposed, which reconstructs the selection process for optimal candidate solutions into a relationship modeling task across candidate solutions. This network achieves efficient alignment of multimodal features through a GFF module and utilizes an ERM module for joint inference among multiple candidate solutions. The two work together to effectively overcome the inherent "score dilution" defect of traditional isolated scoring mechanisms, significantly amplifying subtle geometric differences between highly similar candidate solutions.

[0013] 3) An HTK-Loss algorithm was designed and introduced, which utilizes a global distribution alignment and local asymmetric enhancement strategy to jointly optimize the scoring network. This loss function not only forces the network to learn the global ranking consistency of candidate solutions, but also imposes a discriminative penalty on the optimal solution, thereby sharpening the decision boundary and enabling the system to more accurately retrieve and output the optimal 3D face mesh with the global minimum error. Attached Figure Description

[0014] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0015] Figure 1 This is a diagram of the reconstructed network structure in Example 1. Detailed Implementation

[0016] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0017] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0018] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0019] Example 1 One embodiment of the present invention provides a fast 3D reconstruction method for occluded faces based on a diffusion model, comprising: Step 1: Obtain a monocular 2D face image containing occlusion, extract image identity features from it, and obtain identity conditions through normalization; Step 2: Based on the identity conditions, use the diffusion model to accelerate sampling of ordinary differential equations and apply spectral adaptive projection constraints to generate a candidate solution set; Furthermore, the method of using a diffusion model to accelerate sampling using ordinary differential equations specifically involves: By enhancing trajectory-aware reasoning, the generation process of the diffusion model is constructed as a numerical solution process of probabilistic flow ordinary differential equations. A high-order ordinary differential equation solver is used to perform large-step deterministic integration, compressing the number of denoising sampling steps to a preset number of steps, and generating multiple candidate 3D face shape parameters under the guidance of identity conditions.

[0020] Furthermore, the adaptive projection constraint of the spectrum is specifically as follows: By using spectrum-aware hyperellipsoid projection, the basis energy spectrum is extracted from the FLAME shape basis. After normalization, the boundary of the hyperellipsoid feasible region is constructed for each dimension. Element-wise adaptive projection truncation is performed on the terminal latent variables sampled by the ordinary differential equation to constrain the latent variables to the boundary of the hyperellipsoid feasible region. After decoding, candidate 3D face shape parameters are generated to obtain a set of candidate solutions.

[0021] Step 3: Perform multimodal feature fusion and amplify the difference relationship between candidates on the candidate solution set and identity conditions to obtain the normalized score of each candidate solution; Furthermore, the normalized score of each candidate solution is obtained through an interactive comparison scoring network, specifically: The candidate solutions and identity conditions are dynamically gated and fused to obtain each fused feature. All fused features are combined into a set tensor. Through explicit relation modeling, the geometric centroid of the candidate set is calculated and the difference residual between the relative centroids of each candidate is extracted to amplify the geometric differences between candidates. After scoring mapping, the normalized score of each candidate is output.

[0022] Furthermore, the training optimization of the interactive comparison scoring network adopts a hierarchical Top-...K The ranking loss function decouples the optimization into two stages: global distribution alignment and local asymmetric reinforcement.

[0023] Furthermore, the global distribution alignment specifically refers to: The geometric error between the candidate solution and the real 3D face scanning mesh is mapped to the target probability distribution, and the candidate scores output by the scoring network are mapped to the predicted probability distribution. The KL divergence between the two probability distributions is used as the loss constraint.

[0024] Furthermore, the local asymmetric enhancement specifically refers to: For the front with the smallest geometric error K For each candidate solution, a contrast penalty based on the margin is applied, with higher penalty weights assigned to candidate pairs whose error differences are smaller.

[0025] Step 4: Take the candidate solution with the highest score as the global optimal solution, and synthesize a 3D face mesh again based on this parameter.

[0026] As one embodiment, the present invention provides a fast 3D face reconstruction method based on a diffusion model, which achieves low latency, high fidelity, and accurate optimal solution retrieval capability for 3D face geometric reconstruction in severely occluded scenarios. The specific implementation process is described below: Since generative multi-hypothesis diffusion models are good at fitting complex data distributions and provide powerful occlusion completion structure priors, this embodiment combines the diffusion generation mechanism with the global interactive comparison scoring mechanism to complete the generation of high-quality candidate 3D faces and the accurate selection of the optimal solution. In order to accelerate the generation while ensuring high fidelity and solve the problem of difficulty in distinguishing between highly similar candidate solutions, this embodiment proposes a fast 3D face reconstruction network under occlusion based on diffusion model and interactive comparison mechanism.

[0027] The input to the reconstruction network is a monocular 2D face image containing occlusions. Output high-fidelity 3D face mesh geometric parameters That is, the global optimal solution, and further, through decoupling, the complete 3D face geometry is recovered. In the actual inference application stage, the specific execution process of 3D face reconstruction is as follows: (1) Input a two-dimensional image containing occlusion. Extracting identity features using the ArcFace encoder .

[0028] (2) Start the TIE module, and combine the SA-HEP adaptive constraints with the DPM-Solver++ solver to generate in parallel within 20 sampling steps. N =100 alternative shape parameters That is, the set of candidate solutions.

[0029] (3) Combine the candidate set with identity conditions The InterCompScoreNet scoring network is input, and after GFF feature fusion and explicit ERM relationship comparison, the output is a Softmax scoring probability vector. .

[0030] (4) The candidate solution with the highest score (Top-1) is retrieved and denoted as the global optimal shape parameter. .

[0031] (5) Optimal parameters The TIE module is input again to guide it to quickly synthesize decoupled facial expression parameters, and finally output a high-fidelity, distortion-free 3D face mesh structure.

[0032] The reconstructed network that achieves the above process, such as Figure 1 As shown, it mainly consists of two core network modules: the Trajectory Aware Inference Enhancement Module (TIE) and the InterCompScoreNet interactive comparison scoring network, and adopts a hierarchical Top- K The ranking loss function (HTK-Loss) is optimized during the training of the InterCompScoreNet network, which is explained in detail below: I. ArcFace Encoder Using a pre-trained ArcFace encoder to extract from an input occluded image Robust identity feature vectors are extracted and then subjected to L2 normalization to obtain identity conditions. The ArcFace encoder is implemented using a ResNet residual network.

[0033] The ArcFace encoder can be replaced with FaceNet, CosFace, SphereFace, or any visual backbone network with facial feature representation capabilities, such as VisionTransformer (ViT).

[0034] II. Trajectory Awareness Reasoning Enhancement Module (TIE) To address the long inference time of traditional Markov denoising sampling (such as DDPM's 1000-step sampling), and to overcome the mismatch between isotropic diffused noise and anisotropic deformation priors in the FLAME parameter space, this embodiment introduces the SA-HEP mechanism within the TIE module, and combines it with a high-order ordinary differential equation solver to form a complete efficient TIE sampling mechanism. SA-HEP utilizes the deformation spectrum to construct adaptive boundaries, suppressing long-tailed anomaly biases, enabling the network to quickly generate high-fidelity candidate solutions with only 20 sampling steps. Specifically: (1) Rapid sampling.

[0035] To address the issue of slow traditional sampling speeds, this embodiment constructs a TIE acceleration module. For example... Figure 1 As shown, the TIE acceleration module transforms the generation process of the diffusion model (i.e., the reverse denoising process) into a numerical solution process of the probability flow ordinary differential equation. Its core consists of a conditional U-Net network responsible for predicting noise and a solver responsible for state updates, which together form a large-step iterative loop.

[0036] Specifically, the generation process samples the initial tensor from the standard normal distribution. , This is the initial denoising time step; at each denoising time step... ， The current potential tensor The data is input into the U-Net network. To achieve precise guidance based on identity features, U-Net has a dedicated conditional module with the following processing logic: Current time step After sinusoidal position embedding mapping to temporal embedding, the identity conditions output by the ArcFace encoder are also included. Mapped as a context embedding; Next, the conditional module concatenates the temporal embedding with the contextual embedding and inputs it into the multilayer perceptron (MLP) to generate parameters for feature modulation.

[0037] Subsequently, U-Net uses the FiLM modulation mechanism to apply the above parameters to the input latent tensor. The noise is then processed by the attention module and finally output by U-Net as predicted noise.

[0038] After acquiring the prediction noise at the current time step, the TIE acceleration module uses the high-order solver DPM-Solver++ to perform large-step deterministic integration, thereby executing the "update" operation shown in the figure and deriving the latent tensor for the next time step. Through this iterative mechanism consisting of U-Net predicting noise and large-step updates of the solver, TIE significantly reduces the denoising process from 1000 steps to 20 steps.

[0039] When the sampling iteration reaches the termination time DPM Solver++ outputs latent tensors ,Should To complete the FLAME face parameters after all noise reduction.

[0040] Although the DPM-Solver++ algorithm is used in this embodiment to achieve efficient sampling in 20 steps, in other embodiments, this solver can be replaced with DPM-Solver, DDIM, Score-SDE, Euler solver or other equivalent accelerated sampling algorithms to accelerate diffusion model inference.

[0041] (2) Spectrum-aware hyperellipsoidal projection (SA-HEP).

[0042] The latent generation space of the diffusion model essentially corresponds to the shape parameter space of the FLAME model. The diffusion model outputs the shape parameters in this space through progressive denoising. However, due to the strongly anisotropic distribution of the PCA principal component basis energy in the FLAME shape space, which exhibits a sharp decay, i.e., low-order principal components control the main contours while high-order principal components correspond to small deformations with low amplitude, the diffusion model samples isotropic noise. This leads to physically unreasonable numerical biases accumulating in the low-energy dimensions of the FLAME during accelerated generation due to the large step size of the ordinary differential equation sampling. To address this, this embodiment designs an SA-HEP constraint module: First, the basis energy spectrum is extracted from the FLAME model by calculating the geometric deformation magnitude in each PCA principal component dimension. The specific extraction process is as follows: Assume the initial shape basis tensor read from the FLAME model is ,in, This represents the number of grid vertices, where 3 represents the three-dimensional spatial coordinates XYZ. D The principal component dimension of the PCA is used; flattening it along the vertex coordinate dimension yields a two-dimensional matrix. For the first i Extract the corresponding basis vectors from each PCA dimension. The initial basis energy of this dimension Based on the L2 norm of the vector, the following can be calculated:

[0043] Subsequently, based on the extracted energy spectrum Normalization is performed based on the maximum energy value:

[0044] in, i and j These are all index variables of the PCA principal component dimension, with values ​​ranging from 1 to... D ; , indicating that all D For each principal component dimension, the maximum basis energy value is extracted as the global normalization benchmark. Based on the normalized spectrum, SA-HEP constructs a dimension-adaptive hyperellipsoidal feasible domain boundary for each dimension. :

[0045] in, The global allowable offset is set to 2.0. Since the FLAME shape parameters, after standardization, approximately follow a standard normal distribution, this boundary can cover approximately... The range of principal component values; The minimum retention ratio for low-energy dimensions is set to 0.2; The constant for controlling the smoothness of the transition between high and low energy is set to 0.01.

[0046] FLAME face parameters obtained by fast sampling There may be values ​​that exceed the reasonable range of human facial physical dimensions, therefore, Enter SA HEP module, SA HEP Perform element-wise adaptive projection truncation and output the truncated FLAME face parameter vector. This represents a candidate solution. Through parallel execution... N A separate sampling trajectory, repeating the above sampling. Cut off the process and get N The final output includes: 10 alternative 3D face shape parameters. N One (in the embodiments) N = 100) Candidate solution set of alternative 3D face shape parameters .

[0047] SA The HEP module's dimension-wise adaptive projection truncation operation satisfies the following formula:

[0048] in, iThis refers to the dimension index of the shape parameters in the FLAME face PCA, corresponding to the aforementioned PCA principal component dimensions, with values ​​ranging from 1,..., D ; Output face parameter tensors to the solver The i Parameter values ​​before dimension truncation; This indicates that the face parameter is 1 after the truncation operation is completed. i Parameter values ​​in the dimension; This indicates that the aforementioned calculations are for the first... i The adaptive boundary thresholds in each dimension. SA-HEP serves as a lightweight geometric constraint for the sampling terminal, requiring no additional neural network computation overhead, and effectively suppresses out-of-range deviations from the long-tail distribution during the accelerated sampling process.

[0049] After the above SA All obtained after HEP constraint processing N candidate solutions Constructing a candidate solution set The candidate solution set will be input into the subsequent scoring network InterCompScoreNet to perform feature fusion and amplify the difference relationship between candidate solutions.

[0050] III. InterCompScoreNet (InterCompScoreNet) This embodiment integrates mechanisms such as Adaptive Gated Feature Fusion (GFF) and Explicit Relation Modeling (ERM) in the InterCompScoreNet interactive comparison scoring network. It aggregates multimodal features from the input occluded image and multiple generated candidate solutions, establishing cross-candidate solution relationships to obtain more accurate geometric contrast information. The Gated Feature Fusion (GFF) module projects high-dimensional grid features and image identity conditions onto a unified latent space, aligning and fusing multimodal features through a dynamic gating mechanism. The Explicit Relation Modeling (ERM) module performs joint reasoning within the candidate solution set, actively eliminating homogeneous common information and amplifying subtle geometric differences between candidate solutions by calculating the geometric center of the candidate set and extracting difference residuals. This strengthens the interactive comparison scoring module by utilizing explicit relationship comparisons across candidate solutions, integrating distributional knowledge from the entire candidate set, thereby significantly improving the discriminative power of optimal solution selection and effectively overcoming the "score dilution" problem caused by traditional isolated point-by-point scoring.

[0051] Specifically, for the generated candidate solution set InterCompScoreNet reconstructs isolated point-by-point scoring into joint relation reasoning within a set, which includes an adaptive gated feature fusion module (GFF) and an explicit relation modeling module (ERM).

[0052] (1) Adaptive Gated Feature Fusion Module (GFF) In the interactive comparative scoring network, the set of candidate solutions for alternative 3D face shape parameters generated by the TIE module. It was directly used as the initial high-dimensional grid feature. Input. To address high-dimensional grid features. Two-dimensional identity features The representation imbalance problem caused by direct splicing is addressed by GFF, which first addresses the single-dimensional representation imbalance. Perform context dimension expansion to align in the sample size dimension. The number of candidate solutions is determined, and then GFF linearly projects both onto a unified latent space, yielding the following results: and .

[0053] by To guide the signal, calculate the dynamic gating weight. :

[0054] in, For the Sigmoid activation function, This is a learnable weight matrix.

[0055] The fusion feature is expressed as:

[0056] in, This indicates element-wise multiplication.

[0057] In unobstructed areas, the network relies more on visual features; in severely occluded areas, the network automatically amplifies the mesh prior; to prevent the loss of original topological details from deep nonlinear projections, explicit residual connections are introduced and output through a feedforward neural network (FFN):

[0058] Finally, the fused feature vector of each candidate solution is obtained. .

[0059] (2) Explicit Relationship Modeling Module (ERM) Will N The fused feature vector of each candidate solution Combining into set tensors The basic representation is obtained after layer normalization. .

[0060] ERM constructs a horizontal joint comparator: (1) Cross-sample self-attention capture of global consensus: Multi-head self-attention mechanism (MHSA) is applied to the candidate solution dimension to calculate the global structural consensus features within the set:

[0061] Here, MHSA represents the multi-head attention calculation function. The three input variables in the formula... These correspond to the query in the attention mechanism. Q ), key K ) and Value V The matrix is ​​used because it employs a self-attention mechanism, aiming to uncover the interaction relationships between samples within the same candidate solution set. Therefore, the model's queries, keys, and values ​​all originate from the same basic representation. Specifically, the first one (As a Query): Represents the features of the current candidate solution to be evaluated, used to "query" other candidate solutions in the set; the second... (As a Key): Represents the matched features of other candidate solutions in the set, used to perform a dot product with the Query to obtain attention weights (i.e., the similarity or correlation between candidate solutions); the third (Value): Represents the actual content features of the candidate solutions. Finally, based on the attention weights calculated above, the Values ​​of all candidate solutions are weighted and aggregated to output features that contain global structural consensus. .

[0062] (2) Centroid elimination and difference residual amplification: Calculate the geometric centroid of the entire candidate solution set in the feature space. Extract the difference residual of each candidate solution relative to the centroid. And input it into the bottleneck multilayer perceptron for purification:

[0063] This structure forcibly strips away the homogeneous common features among candidate solutions, and explicitly amplifies the individualized fine-grained differences between each candidate solution.

[0064] (3) Dynamic fusion and scoring: using learnable scalar coefficients strictly constrained by Sigmoid. and Injection of dynamic control relationship features:

[0065]

[0066]

[0067] in, The first learnable scalar parameter is used to obtain the weights via a Sigmoid mapping. Used to control attention relationship features The injection intensity; The second learnable scalar parameter is used to obtain the weights via a Sigmoid mapping. Used to control the difference-enhancing relationship feature The injection intensity; The basic features to be integrated The fused features are obtained after dynamic feature fusion and layer normalization. The unnormalized raw score output by the multilayer perceptron is finally mapped to the normalized predicted probability score for each candidate solution using the Softmax function. .

[0068] In the interactive comparison scoring network, this embodiment extracts the difference residuals by calculating the geometric centroid and combining it with a multilayer perceptron. Alternatively, this relationship modeling process can also achieve explicit comparisons between candidate solutions using graph neural networks, fully cross-attention mechanisms, or distance metric matrices.

[0069] This embodiment uses a layered Top- K The ranking loss function (HTK-Loss) is used to optimize the training of the InterCompScoreNet interactive comparison scoring network, which makes up for the deficiency of the traditional global equal-weight ranking loss in that it cannot impose a discriminative penalty on the top-1 optimal solution.

[0070] Specifically, during the training phase, the pre-trained TIE module weights are kept frozen, and InterCompScoreNet is trained end-to-end. To address the problem that small geometric residuals between highly similar candidate solutions are difficult to differentiate, this embodiment designs HTK-Loss, which decouples optimization into two stages: global distribution alignment and local asymmetric enhancement.

[0071] (1) Global distribution alignment constraint Based on the geometric error between the candidate solution and the real 3D face scan mesh Calculate the target probability distribution The network prediction score is mapped to a probability distribution. ,in, This indicates that the aforementioned interactive comparison scoring network is NThe original score vectors (i.e., unnormalized scores Logits) of the candidate solutions' predicted outputs. A macroscopic ordinal consistency loss is constructed using KL divergence: [The text repeats itself here, so the translation only includes the first instance.]

[0072] In the global distribution alignment constraint, the aforementioned geometric error is measured. The method is not limited to the L1 distance (i.e., the average absolute error between 3D mesh vertices) or L2 distance (i.e., the mean square error between 3D mesh vertices) used in the embodiments, but can also be replaced by Huber Loss, cosine similarity loss or other measures that can reflect the spatial deviation of 3D mesh vertices.

[0073] (2) Local asymmetric strengthening loss For the smallest error K Given a set of head candidate solutions, a contrast penalty based on margin is introduced to calculate the weighted margin loss:

[0074] in, The preset sorting margins, The set of all valid candidate pairs, where "candidate pair" refers to the set of all pairs mentioned above. K The pairing of candidate solutions in the head candidate solution set is formed by combining each candidate solution in pairs. m , n ), and These respectively indicate that the aforementioned networks are candidate solutions. m and candidate solutions n Output predicted score. Joint adaptive weights. Driven by three asymmetric mechanisms: .

[0075] (1) Error-sensitive weights Candidate pairs with smaller error differences are assigned higher gradient weights, forcing the network to extract fine-grained discriminative features from extremely similar samples.

[0076] (2) Position-aware weights Based on high-quality solutions m The true ranking implements logarithmic decay, imposing a harsher penalty on ranking reversals at the top of the list than on those at the bottom.

[0077] (3) Top-1 significance incentive If the candidate pair contains the globally absolute optimal solution (i.e. If the penalty weight is doubled, it ensures that the absolute optimal solution is not misjudged.

[0078] (3) Overall loss function The total loss function is composed of a weighted combination of two parts:

[0079] in, This is the balance coefficient.

[0080] This embodiment uses median error (Med), mean error (Mean), and standard deviation of error (Std) as quantitative indicators to evaluate the accuracy of 3D facial geometric reconstruction. The smaller the value, the smaller the error and the higher the accuracy. At the same time, the end-to-end total inference time is used as an indicator to evaluate the computational efficiency of the system. This embodiment comprehensively compares and evaluates the proposed method with existing cutting-edge monocular 3D face reconstruction methods on the internationally authoritative public benchmark dataset NoWBenchmark. The results are shown in Tables 1 and 2. As can be seen from Tables 1 and 2, the method of this embodiment has achieved significantly better results in terms of reconstruction accuracy and inference speed.

[0081] Table 1. Comparison of this method with other methods - accuracy

[0082] Table 2 Comparison of this method with other methods - speed

[0083] Example 2 One embodiment of the present invention provides a rapid 3D reconstruction system for occluded faces based on a diffusion model, comprising: The feature extraction module is configured to: acquire a monocular two-dimensional face image containing occlusion, extract image identity features from it, and obtain identity conditions through normalization; The candidate generation module is configured to: generate a set of candidate solutions by using a diffusion model to accelerate sampling of ordinary differential equations based on identity conditions and applying spectral adaptive projection constraints. The candidate scoring module is configured to: perform multimodal feature fusion and amplify the difference relationship between candidates on the candidate solution set and identity conditions to obtain the normalized score of each candidate solution; The face reconstruction module is configured to take the candidate solution with the highest score as the global optimal solution and then synthesize a 3D face mesh based on this parameter.

[0084] Example 3 One embodiment of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method for rapid 3D reconstruction of occluded faces based on a diffusion model.

[0085] Example 4 In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided for storing computer instructions. When the computer instructions are executed by a processor, they implement the aforementioned method for rapid 3D reconstruction of occluded faces based on a diffusion model.

[0086] Example 5 One embodiment of the present invention provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the aforementioned method for rapid 3D reconstruction of occluded faces based on a diffusion model.

[0087] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0088] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0089] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A fast 3D reconstruction method for occluded faces based on a diffusion model, characterized in that, include: Obtain monocular 2D face images containing occlusions, extract image identity features from them, and obtain identity conditions through normalization; Based on the identity conditions, the diffusion model is used to accelerate sampling of ordinary differential equations and apply spectral adaptive projection constraints to generate a set of candidate solutions; Multimodal feature fusion and amplification of the differences between candidates are performed on the candidate solution set and identity conditions to obtain the normalized score of each candidate solution; The candidate solution with the highest score is taken as the global optimal solution, and a 3D face mesh is synthesized again based on this parameter.

2. The method for rapid 3D reconstruction of occluded faces based on a diffusion model as described in claim 1, characterized in that, The method of using a diffusion model to accelerate sampling for ordinary differential equations specifically involves: By enhancing trajectory-aware reasoning, the generation process of the diffusion model is constructed as a numerical solution process of probabilistic flow ordinary differential equations. A high-order ordinary differential equation solver is used to perform large-step deterministic integration, compressing the number of denoising sampling steps to a preset number of steps, and generating multiple latent variables under the guidance of identity conditions.

3. The method for rapid 3D reconstruction of occluded faces based on a diffusion model as described in claim 2, characterized in that, The adaptive projection constraint of the spectrum is specifically as follows: By using spectrum-aware hyperellipsoid projection, the basis energy spectrum is extracted from the FLAME shape basis. After normalization, the boundary of the hyperellipsoid feasible region is constructed for each dimension. Element-wise adaptive projection truncation is performed on the terminal latent variables sampled by the ordinary differential equation to constrain the latent variables to the boundary of the hyperellipsoid feasible region. After decoding, candidate 3D face shape parameters are generated to obtain a set of candidate solutions.

4. The method for rapid 3D reconstruction of occluded faces based on a diffusion model as described in claim 1, characterized in that, The normalized scores of each candidate solution are obtained through an interactive comparison scoring network, specifically: The candidate solutions and identity conditions are dynamically gated and fused to obtain each fused feature. All fused features are combined into a set tensor. Through explicit relation modeling, the geometric centroid of the candidate set is calculated and the difference residual between the relative centroids of each candidate is extracted to amplify the geometric differences between candidates. After scoring mapping, the normalized score of each candidate is output.

5. The method for rapid 3D reconstruction of occluded faces based on a diffusion model as described in claim 4, characterized in that, The training optimization of the interactive comparison scoring network adopts a hierarchical Top- K The ranking loss function decouples the optimization into two stages: global distribution alignment and local asymmetric reinforcement.

6. The method for rapid 3D reconstruction of occluded faces based on a diffusion model as described in claim 5, characterized in that, The global distribution alignment specifically refers to: The geometric error between the candidate solution and the real 3D face scanning mesh is mapped to the target probability distribution, and the candidate scores output by the scoring network are mapped to the predicted probability distribution. The KL divergence between the two probability distributions is used as the loss constraint.

7. The method for rapid 3D reconstruction of occluded faces based on a diffusion model as described in claim 5, characterized in that, The local asymmetric enhancement specifically refers to: For the front with the smallest geometric error K For each candidate solution, a contrast penalty based on the margin is applied, with higher penalty weights assigned to candidate pairs with smaller error differences.

8. A rapid 3D reconstruction system for occluded faces based on a diffusion model, characterized in that, include: The feature extraction module is configured to: acquire a monocular two-dimensional face image containing occlusion, extract image identity features from it, and obtain identity conditions through normalization; The candidate generation module is configured to: generate a set of candidate solutions by using a diffusion model to accelerate sampling of ordinary differential equations based on identity conditions and applying spectral adaptive projection constraints. The candidate scoring module is configured to: perform multimodal feature fusion and amplify the difference relationship between candidates on the candidate solution set and identity conditions to obtain the normalized score of each candidate solution; The face reconstruction module is configured to take the candidate solution with the highest score as the global optimal solution and then synthesize a 3D face mesh based on this parameter.

9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement a method for rapid 3D reconstruction of occluded faces based on a diffusion model as described in any one of claims 1-7.

10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform a method for rapid 3D reconstruction of occluded faces based on a diffusion model as described in any one of claims 1-7.

Images