Three-dimensional face generation method based on diffusion model

Through the three-dimensional face generation method based on diffusion model, combined with retopology processing, UV texture mapping and style enhancer, the accuracy and details of the CBCT scan data generation of three-dimensional face models are solved, and high-quality three-dimensional face model generation is achieved , suitable for medical diagnosis and treatment.

CN118941708APending Publication Date: 2024-11-12SOUTHEAST UNIV +1
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410924033.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-11
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Prior art When generating three-dimensional face models using CBCT scanning data, it is difficult to accurately capture complex facial details and establish effective mapping between different data sources, especially when patient movement or specific scanning postures are damaged, and soft tissue detail imaging There are defects.

Method used

Using a three-dimensional face generation method based on diffusion model, retopology processing and UV texture mapping are performed through high-resolution three-dimensional face scanning, combined with pre-trained CBCT encoder and StyleGAN2 network, a high-quality three-dimensional face model is generated, using style Enhancers improve the detail and fidelity of the model.

Benefits of technology

It significantly improves the accuracy and reality of three-dimensional face generation, enhances the visual quality and structural consistency of the model, and is suitable for medical imaging, helping clinicians better diagnose and treat facial deformities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941708A_ABST
    Figure CN118941708A_ABST
Patent Text Reader

Abstract

The invention discloses a three-dimensional face generation method based on a diffusion model, and the method comprises the steps: firstly, employing a pre-trained encoder for extracting a hidden space vector from CBCT scanning; the vectors are then input into a specific synthesis module, which can predict the shape and texture of the three-dimensional face. By using the diffusion model, efficient mapping from the CBCT data to the three-dimensional face parameter space is realized. In addition, material generation by using a StyleGAN2 network is explored, so that the details and reality of the final three-dimensional face model are improved. According to the method, the complexity of a traditional multi-stage processing flow is avoided, the reasoning speed is remarkably improved, and diagnosis and treatment design and doctor-patient communication in the fields of orthodontics, oral cavity and maxillofacial surgery, plastic surgery and the like are effectively supported through the generated three-dimensional face model. In addition, a fine-grained three-dimensional face data set is created, and necessary real data is provided for training and verification of the model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a method for generating a high-quality three-dimensional human face model by using CBCT scanning data, and in particular to a three-dimensional human face generation method based on a diffusion model. Background Art

[0002] 3D face generation involves generating accurate 3D face models from different data sources. The key challenge of this technology is how to accurately capture the complex details of the face and how to establish an effective mapping between different data sources and modalities. To solve this problem, scholars have proposed 3D deformable models, which allow the detailed features of individual faces to be fine-tuned and edited through a set of parameterized 3D face base models. In recent years, 3D deformable models have been widely used to restore 3D faces from a single image or multiple images, but the research on generating 3D faces from CBCT scans is very limited. However, CBCT technology has limitations in capturing soft tissue details and avoiding artifacts, especially the problem of image quality impairment when the patient moves or in a specific scanning posture. In order to improve the application effect of CBCT technology, scholars have explored a variety of improvement methods. For example, reducing artifacts through software algorithm optimization and combining other imaging techniques such as magnetic resonance imaging to enhance the visualization of soft tissues. In addition, with the development of 3D digital imaging technology, 3D photography and stereo mapping have become complementary means to solve the shortcomings of CBCT in soft tissue imaging. These methods help doctors understand the patient's facial structure more comprehensively by providing detailed 3D facial scans, especially showing unique advantages in complex dental surfaces and soft tissue changes. Therefore, it is of great significance to study methods for generating high-precision 3D human faces from CBCT data. Summary of the invention

[0003] The present invention discloses a method for generating a three-dimensional face based on a diffusion model, which specifically comprises the following steps:

[0004] Step 1: Select and use a set of high-resolution 3D face scans as a benchmark dataset, which includes detailed facial mesh and texture information.

[0005] Step 2: Apply 3DMM to retopology the high-resolution scan to generate a model with a unified topology for subsequent processing and analysis.

[0006] Step 3: For each retopological model, extract the UV texture map from the relevant texture image to create a high-quality texture map.

[0007] Step 4: Combine the retopological model and the corresponding UV texture mapping to form a complete 3D face dataset for training and verifying the synthesis module and texture generation network.

[0008] Step 5: Parse the input CBCT scan data through the pre-trained CBCT encoder to generate a latent space vector,

[0009] Step 6: Input the latent space vector obtained in step 1 into the diffusion model to predict the 3D face shape to improve the accuracy and realism of the generated model.

[0010] Step 7: Apply the StyleGAN2 network to generate textures corresponding to the predicted 3D shapes, further improving the details and realism of the model.

[0011] Step 8: Integrate shape and texture data to generate the final high-quality 3D face model.

[0012] The 3DMM retopology process in step 2 is described in detail as follows: First, the automatic face alignment technology is used to map the texture image T H The two-dimensional key points are calibrated and then mapped to the corresponding three-dimensional scan model S H On the surface, a set of three-dimensional key points is formed Next, a high-fidelity template shape S temp The template has 20481 vertices and 40832 triangles, and defines preset key points and facial area masks. temp Shape adjusted to fit S H The facial structure of the template is ensured to be as close as possible to the key points of S H The key points actually detected on In this process, the overall mesh deformation includes smoothing terms and landmark loss terms, which are used to optimize shape matching and reduce deformation errors. Finally, the adjusted template shape S temp Elaborate post-processing removes any sharp or irregular features caused by noise in the original scan and optimizes the mesh quality using a Laplace edit-based approach, resulting in a low-resolution baseline mesh S with consistent topology and enhanced visual quality. L This step ensures that the retopological model maintains structural consistency and high operability between different individuals, providing a solid foundation for subsequent 3D face generation.

[0013] The UV texture mapping process in step 3 is described in detail as follows: Using advanced image processing technology, texture information is extracted from high-resolution material images. This process relies on the retopological 3D model S LThe method focuses on the optimization of albedo and lighting, and through this focus, the texture can be more accurately adjusted to match the specific features of the model, such as facial contours and feature details. After texture optimization and super-resolution processing, the generated UV texture map has higher details and realism, ensuring the continuity and seamless matching of the texture of the generated 3D face model, significantly improving the visual quality and realism of the model.

[0014] The integration operation described in step 4 is specifically as follows: the entire 3D face dataset is randomly divided into a training set, a validation set, and a test set according to a ratio of 8:1:1, and the sets correspond one-to-one with the CBCT scan data of each 3D face entity.

[0015] The diffusion model in step 6 adopts the conditional generation method, which is as follows:

[0016] 3DMM models are generated from 3D polygonal mesh models in the training set using principal component analysis. Specifically, given m facial mesh models, each containing n vertices, an m×n tensor is constructed to represent all these vertices in the training set. Tucker decomposition is used to decompose the m×n tensor into a small PCA basis matrix B and an m′-dimensional factor representing the facial identity. Given an arbitrary 3DMM parameter s, a new set of vertices v representing the 3D facial shape can be generated:

[0017] v=B×s (1)

[0018] In this way, large-scale 3D facial shape data is mapped into an m′-dimensional parameter space, called S-space. Set m=327, m′=300, n=20481.

[0019] The shape production network is trained based on the denoising diffusion probability model, using CBCT embedding as a condition to guide the denoising process of 3D face synthesis. The specific formula used in this process is as follows:

[0020]

[0021] Among them, θ represents the parameters in the shape production network, d S is the conditional code for the input shape, x t is x after t steps of noise addition process 0 , is the predicted 3D face parameter, which will be fitted to the true value x 0In the denoising process of the diffusion model, Gaussian noise is continuously removed from the 3D face to which the noise is added, and the model parameters are updated based on the shape-related code to predict the 3D face. The forward process involves the noise addition process, where Gaussian noise is added to the true value through t steps, each step is encoded by the step encoder. Finally, all elements are fed into the decoder to predict the 3D face parameters.

[0022] After the forward noise adding process, the reverse process is performed, from Reverse deduction Therefore, we need to know the conditional distribution function in advance The training objective is defined as learning to predict the addition of x to the input 0 However, the model learns to predict the actual 3D face rather than the noise level in the data, as the result also depends on the input CBCT. By choosing this approach, the model is able to predict acceptable results even at the initial denoising step of the inference process, thus enabling faster sampling.

[0023] The shape generation network is trained using the following two types of losses: weighted l 1 loss and region-specific triplet loss. Through a differentiable 3DMM mapping module, the predicted 3DMM parameters can be transformed into the 3D positions of vertices. Since l is directly applied to all vertices 1 The loss results in an average result across the whole image, so a weighted mask is used to calculate the loss for different regions. 1 The formula of the loss function is:

[0024]

[0025] Among them, v i represents the vertex of the i-th region, and α i Represents the corresponding weights. The entire head mesh model is divided into four regions: (1) 68 facial landmarks; (2) nose and mouth; (3) forehead and cheeks; (4) back of the head with ears. The weights of these regions are set to 16:4:3:0. In order to enhance the diversity of the generated 3D shapes, region-specific triplet loss is used to train the model. The key idea behind it is to make the difference between the prediction and the positive example smaller, while the difference between the prediction and the negative example larger. The difference is measured by calculating the average Euclidean distance and applying weights to different regions. Specifically, four region masks are applied, and in each training iteration, a positive and negative sample pair of a region is randomly selected, and the region-specific triplet loss is calculated, as follows:

[0026]

[0027] in, is the vertex of the predicted i-th region, vi is the corresponding real vertex, and is its opposite sample. i and λ i Represent the interval and weight of the corresponding area respectively.

[0028] Among them, the StyleGAN2 network in step 7 is specially trained and adjusted to adapt to the generation of high-resolution texture maps from 3D shape information. This process uses StyleGAN2 as the backbone network, which adopts a generative adversarial network architecture to process and generate high-quality UV texture maps. In this application, StyleGAN2 accepts a latent space representation generated by CBCT data as a conditional input, as well as randomly generated latent codes. These inputs are mapped to the so-called W space, which is able to decouple and independently control different facial features, allowing finer texture detail generation.

[0029] During the training phase, the network uses UV texture maps extracted from the triangular mesh generated by 3DMM as target images. These texture maps are rich in detail and provide realistic skin texture and color for the generated 3D models. In this way, StyleGAN2 learns how to reconstruct and generate 2D image textures that match the quality of the original textures from the provided conditional codes. The settings of the loss function and hyperparameters remain the same as the standard StyleGAN2, ensuring that the network can be effectively optimized during the learning process to generate visually pleasing results.

[0030] In 2D image generation, it is often difficult to control the entire generation process using a single conditional code. Therefore, a style enhancer is introduced to increase the influence of the conditional code on material rendering.

[0031] Specifically, the style enhancer first transforms the input image features F i With conditional features Fusion:

[0032]

[0033] in

[0034]

[0035] here, refers to the last hidden state of the CBCT encoder, F′ i The offset Δw used to predict the style code i , denoted as w′ i =w+Δw i , and is used to adjust the current upsampling block:

[0036] w′i =w+Δw i =w+MLP(Conv 1×1 (F′ i )) (7)

[0037] Equipped with a style enhancer, the model is able to better embed CBCT information into the rendering process while maintaining image quality.

[0038] The present invention proposes a method for generating three-dimensional faces based on a diffusion model. The method first uses high-resolution three-dimensional face scanning data to establish a benchmark data set, including facial meshes and texture information. Retopology processing is performed through 3DMM technology to generate a model with a unified topological structure for subsequent processing and analysis. Subsequently, UV texture mapping is applied to each model to create a high-quality texture map, which is integrated with the retopological model to form a complete three-dimensional face data set. The pre-trained CBCT encoder is used to parse the input CBCT scanning data to generate a latent space vector, and the three-dimensional face shape is predicted through the diffusion model to improve the accuracy and realism of the generated model. In addition, the StyleGAN2 network is used to generate the corresponding texture to further enhance the details and fidelity of the model, and an innovative module of a style enhancer is proposed to ensure the influence of CBCT on the generation of three-dimensional faces. Finally, the shape and texture data are integrated to generate a high-quality three-dimensional face model. The core of the present invention lies in the cross-modal mapping directly from CBCT data to three-dimensional face parameters, which significantly improves the generation speed and fidelity, is suitable for medical imaging and other fields, and provides effective diagnostic and treatment tools for clinical practice. It can help clinicians identify and localize the source of deformity and its severity, avoid patient recalls and reduce appointment time, make treatment simulation and before-and-after comparisons easier, allowing clinicians to evaluate facial changes after orthodontic or surgical treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is the overall structure diagram of the model in the present invention;

[0040] Figure 2 Diagram of the style enhancer framework in the material generation network for the model. DETAILED DESCRIPTION

[0041] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0042] The three-dimensional face generation method based on the diffusion model proposed in the present invention includes a training part and a testing part. The method needs to be trained first, and the parameters of the shape generation network and the material generation network are continuously updated through training. After the training is completed, they are saved. In the test phase, the shape generation network and the material generation network are used to generate the shape and material of the three-dimensional face respectively, and finally integrated to obtain a high-precision three-dimensional face.

[0043] Example 1: See Figure 1 , 2 , a 3D face generation method based on a diffusion model, specifically comprising the following steps:

[0044] Step 1: Select and use a set of high-resolution 3D face scans as a benchmark dataset, which includes detailed facial mesh and texture information.

[0045] Step 2: Apply 3DMM to retopology the high-resolution scan to generate a model with a unified topology for subsequent processing and analysis.

[0046] Step 3: For each retopological model, extract the UV texture map from the relevant texture image to create a high-quality texture map.

[0047] Step 4: Combine the retopological model and the corresponding UV texture mapping to form a complete 3D face dataset for training and verifying the synthesis module and texture generation network.

[0048] Step 5: Parse the input CBCT scan data through the pre-trained CBCT encoder to generate a latent space vector,

[0049] Step 6: Input the latent space vector obtained in step 1 into the diffusion model to predict the 3D face shape to improve the accuracy and realism of the generated model.

[0050] Step 7: Apply the StyleGAN2 network to generate textures corresponding to the predicted 3D shapes, further improving the details and realism of the model.

[0051] Step 8: Integrate shape and texture data to generate the final high-quality 3D face model.

[0052] The 3DMM retopology process in step 2 is described in detail as follows: First, the automatic face alignment technology is used to map the texture image T H The two-dimensional key points are calibrated and then mapped to the corresponding three-dimensional scan model S H On the surface, a set of three-dimensional key points is formed Next, a high-fidelity template shape S tempThe template has 20481 vertices and 40832 triangles, and defines preset key points and facial area masks. temp Shape adjusted to fit S H The facial structure of the template is ensured to be as close as possible to the key points of S H The key points actually detected on In this process, the overall mesh deformation includes smoothing terms and landmark loss terms, which are used to optimize shape matching and reduce deformation errors. Finally, the adjusted template shape S temp Elaborate post-processing removes any sharp or irregular features caused by noise in the original scan and optimizes the mesh quality using a Laplace edit-based approach, resulting in a low-resolution baseline mesh S with consistent topology and enhanced visual quality. L This step ensures that the retopological model maintains structural consistency and high operability between different individuals, providing a solid foundation for subsequent 3D face generation.

[0053] The UV texture mapping process in step 3 is described in detail as follows: Using advanced image processing technology, texture information is extracted from high-resolution material images. This process relies on the retopological 3D model S L The method focuses on the optimization of albedo and lighting, and through this focus, the texture can be more accurately adjusted to match the specific features of the model, such as facial contours and feature details. After texture optimization and super-resolution processing, the generated UV texture map has higher details and realism, ensuring the continuity and seamless matching of the texture of the generated 3D face model, significantly improving the visual quality and realism of the model.

[0054] The integration operation described in step 4 is specifically as follows: the entire 3D face dataset is randomly divided into a training set, a validation set, and a test set according to a ratio of 8:1:1, and the sets correspond one-to-one with the CBCT scan data of each 3D face entity.

[0055] The diffusion model in step 6 adopts the conditional generation method, which is as follows:

[0056] 3DMM models are generated from 3D polygonal mesh models in the training set using principal component analysis. Specifically, given m facial mesh models, each containing n vertices, an m×n tensor is constructed to represent all these vertices in the training set. Tucker decomposition is used to decompose the m×n tensor into a small PCA basis matrix B and an m′-dimensional factor representing the facial identity. Given an arbitrary 3DMM parameter s, a new set of vertices v representing the 3D facial shape can be generated:

[0057] v=B×s (1)

[0058] In this way, large-scale 3D facial shape data is mapped into an m′-dimensional parameter space, called S-space. Set m=327, m′=300, n=20481.

[0059] The shape production network is trained based on the denoising diffusion probability model, using CBCT embedding as a condition to guide the denoising process of 3D face synthesis. The specific formula used in this process is as follows:

[0060]

[0061] Among them, θ represents the parameters in the shape production network, d S is the conditional code for the input shape, x t is x after t steps of noise addition process 0 , is the predicted 3D face parameter, which will be fitted to the true value x 0 In the denoising process of the diffusion model, Gaussian noise is continuously removed from the 3D face to which the noise is added, and the model parameters are updated based on the shape-related code to predict the 3D face. The forward process involves the noise addition process, where Gaussian noise is added to the true value through t steps, each step is encoded by the step encoder. Finally, all elements are fed into the decoder to predict the 3D face parameters.

[0062] After the forward noise adding process, the reverse process is performed, from Reverse deduction Therefore, we need to know the conditional distribution function in advance The training objective is defined as learning to predict the addition of x to the input 0 However, the model learns to predict the actual 3D face rather than the noise level in the data, as the result also depends on the input CBCT. By choosing this approach, the model is able to predict acceptable results even at the initial denoising step of the inference process, thus enabling faster sampling.

[0063] The shape generation network is trained using the following two types of losses: weighted l 1 loss and region-specific triplet loss. Through a differentiable 3DMM mapping module, the predicted 3DMM parameters can be transformed into the 3D positions of vertices. Since l is directly applied to all vertices 1 The loss results in an average result across the whole image, so a weighted mask is used to calculate the loss for different regions. 1 The formula of the loss function is:

[0064]

[0065] Among them, v i represents the vertex of the i-th region, and α i Represents the corresponding weights. The entire head mesh model is divided into four regions: (1) 68 facial landmarks; (2) nose and mouth; (3) forehead and cheeks; (4) back of the head with ears. The weights of these regions are set to 16:4:3:0. In order to enhance the diversity of the generated 3D shapes, region-specific triplet loss is used to train the model. The key idea behind it is to make the difference between the prediction and the positive example smaller, while the difference between the prediction and the negative example larger. The difference is measured by calculating the average Euclidean distance and applying weights to different regions. Specifically, four region masks are applied, and in each training iteration, a positive and negative sample pair of a region is randomly selected, and the region-specific triplet loss is calculated, as follows:

[0066]

[0067] in, is the vertex of the predicted i-th region, v i is the corresponding real vertex, and is its opposite sample. i and λ i Represent the interval and weight of the corresponding area respectively.

[0068] Among them, the StyleGAN2 network in step 7 is specially trained and adjusted to adapt to the generation of high-resolution texture maps from 3D shape information. This process uses StyleGAN2 as the backbone network, which adopts a generative adversarial network architecture to process and generate high-quality UV texture maps. In this application, StyleGAN2 accepts a latent space representation generated by CBCT data as a conditional input, as well as randomly generated latent codes. These inputs are mapped to the so-called W space, which is able to decouple and independently control different facial features, allowing finer texture detail generation.

[0069] During the training phase, the network uses UV texture maps extracted from the triangular mesh generated by 3DMM as target images. These texture maps are rich in detail and provide realistic skin texture and color for the generated 3D models. In this way, StyleGAN2 learns how to reconstruct and generate 2D image textures that match the quality of the original textures from the provided conditional codes. The settings of the loss function and hyperparameters remain the same as the standard StyleGAN2, ensuring that the network can be effectively optimized during the learning process to generate visually pleasing results.

[0070] In 2D image generation, it is often difficult to control the entire generation process using a single conditional code. Therefore, a style enhancer is introduced to increase the influence of the conditional code on material rendering.

[0071] Specifically, the style enhancer first transforms the input image features F i With conditional features Fusion:

[0072]

[0073] in

[0074]

[0075] here, refers to the last hidden state of the CBCT encoder, F′ i The offset Δw used to predict the style code i , denoted as w′ i =w+Δw i , and is used to adjust the current upsampling block:

[0076] w′ i =w+Δw i =w+MLP(Conv 1×1 (F′ i )) (7)

[0077] Equipped with a style enhancer, the model is able to better embed CBCT information into the rendering process while maintaining image quality.

[0078] Example 2: The steps of the present invention are described in detail below using the above-constructed data set as an example.

[0079] Experimental conditions: A computer is selected for network training. The computer is configured with an Intel(R) processor (3.2GHz) and 124GB random access memory (RAM), Ubuntu 14.04 64-bit operating system, and NVIDIA GTX 3090 (24GB) graphics card; the software environment is the deep learning framework pytorch1.13.1.

[0080] Experimental objects: The training data set was constructed by ourselves. The validation set, auxiliary set, and test set used in network training also came from this data set. They were strictly distinguished and contained 510 data pairs.

[0081] Experimental steps:

[0082] The specific structure of the 3D face generation model based on the diffusion model is shown in Table 1:

[0083] Table 1 Specific structure of 3D face generation based on diffusion model

[0084]

[0085]

[0086] (1) Training phase:

[0087] Step 1: Construct the experimental dataset. The training set contains 408 pairs of CBCT 3D face data. The validation set contains 51 pairs of CBCT 3D face data, and the test set contains 51 pairs of CBCT 3D face data. During training, the input size of the resampled CBCT is [1,512,512,512], where 1 is the size of each batch of data and 512 represents the dimension size of CBCT.

[0088] Step 2: Obtain the encoding of the input CBCT data through the pre-trained Vit model, which is trained on 96k CT data. Initialize the parameters in the shape generation network. First, randomly sample an integer time step from [1, T], representing the time step to be applied to the 3D face x 0 The number of noise steps is calculated and x is obtained according to the following formula t .

[0089]

[0090] Where t is the diffusion time step, β t is a constant noise at time step t, such that 0<β 1 <β 2 <…<β T <1. After the noise addition process, the reverse process needs to be calculated, so the conditional distribution function p(x t-1 ∣x t ). For the material generation network, we follow the hyperparameters and training settings of StyleGAN to train and test the mapping network and texture generator, and the resolution of the UV texture maps used for training is [512,512].

[0091] (2) Testing phase

[0092] After training is complete, the parameters of both networks are retained. During the testing phase, the shape generation network traverses backward through all diffusion time steps from T to 1, gradually improving the predictions of each inference step. During inference, compared to the training process, there is a lack of noise of real 3D faces as input to the model, so at inference time, randomly sampled noise from a uniform distribution is provided to the model.

[0093] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A 3D face generation method based on diffusion model, It is characterized in that The following steps are involved: Step 1: Select and use a set of high-resolution 3D face scans as a benchmark dataset, which includes detailed facial mesh and texture information. Step 2: Apply 3DMM to retopology the high-resolution scan to generate a model with a unified topology for subsequent processing and analysis. Step 3: For each retopological model, extract the UV texture map from the relevant texture image to create a high-quality texture map. Step 4: Combine the retopological model and the corresponding UV texture mapping to form a complete 3D face dataset for training and verifying the synthesis module and texture generation network. Step 5: Parse the input CBCT scan data through the pre-trained CBCT encoder to generate a latent space vector, Step 6: Input the latent space vector obtained in step 1 into the diffusion model to predict the 3D face shape to improve the accuracy and realism of the generated model. Step 7: Apply the StyleGAN2 network to generate textures corresponding to the predicted 3D shapes, further improving the details and realism of the model. Step 8: Integrate shape and texture data to generate a detailed 3D face model.

2. The method for generating a three-dimensional face based on a diffusion model according to claim 1, It is characterized in that The 3DMM retopology process in step 2 is described in detail as follows: First, an automated face alignment technique is used to map the texture image T H The two-dimensional key points are calibrated and then mapped to the corresponding three-dimensional scan model S H On the surface, a set of three-dimensional key points is formed Next, use a fine template shape S temp The template has 20481 vertices and 40832 triangles, and defines preset key points and facial area masks. Through the non-rigid iterative closest point (ICP) technology, S temp Shape adjusted to fit S H The facial structure of the template is ensured to be as close as possible to the key points of S H The key points actually detected on In this process, the overall mesh deformation includes smoothing terms and landmark loss terms, which are used to optimize shape matching and reduce deformation errors. Finally, the adjusted template shape S temp Elaborate post-processing removes any sharp or irregular features caused by noise in the original scan and optimizes the mesh quality using a Laplace edit-based approach, resulting in a low-resolution baseline mesh S with consistent topology and enhanced visual quality. L This step ensures that the retopological model maintains structural consistency and high operability among different individuals, providing a solid foundation for subsequent 3D face generation.

3. The method for generating a three-dimensional face based on a diffusion model according to claim 1, It is characterized in that The UV texture mapping process in step 3 is described in detail as follows: Using image processing technology, texture information is extracted from high-resolution material images. This process relies on the retopological 3D model S L The shape and posture parameters of the face are accurately fitted. After texture optimization and super-resolution processing, the generated UV texture map has higher details and realism, ensuring the continuity and seamless matching of the texture of the generated 3D face model, significantly improving the visual quality and realism of the model.

4. The open domain entity representation learning method based on attention mechanism according to claim 1, It is characterized in that The three-dimensional face dataset in step 4 randomly divides the entire three-dimensional face dataset into a training set, a validation set, and a test set according to a ratio of 8:1:1, and corresponds one-to-one with the CBCT scan data of each three-dimensional face entity.

5. The method for generating three-dimensional human faces based on a diffusion model according to claim 1, It is characterized in that The diffusion model in step 6 adopts the conditional generation method, as follows: Use principal component analysis to generate 3DMM models from 3D polygonal mesh models in the training set. Specifically, given m facial mesh models, each containing n vertices, construct an m×n tensor to represent all these vertices in the training set. Use Tucker decomposition to decompose the m×n tensor into a small PCA basis matrix B and an m′-dimensional factor representing the facial identity. Given any 3DMM parameters s, generate a set of new vertices v representing the 3D facial shape: v=B×s (1) In this way, large-scale 3D facial shape data is mapped into an m′-dimensional parameter space, called S-space, with m=327, m′=300, n=20481. The shape production network is trained based on the denoising diffusion probability model, using CBCT embedding as a condition to guide the denoising process of 3D face synthesis. The specific formula used in this process is as follows: Among them, θ represents the parameters in the shape production network, d S is the conditional code for the input shape, x t is x after t steps of noise addition process 0 , is the predicted 3D face parameter, which will be fitted to the true value x 0 superior, After the forward noise adding process, the reverse process is performed, from Reverse deduction Therefore, we need to know the conditional distribution function in advance The training objective is defined as learning to predict the addition of x to the input 0 However, the model learns to predict the actual 3D face rather than the noise level in the data, since the result also depends on the input CBCT. By choosing this approach, the model is able to predict acceptable results at the initial denoising step of the inference process, thus achieving faster sampling. The shape generation network is trained using the following two types of losses: weighted l 1 loss and region-specific triplet loss. Through a differentiable 3DMM mapping module, the predicted 3DMM parameters are transformed into the 3D positions of vertices. Since l is directly applied to all vertices 1 The loss will result in an overall average result, using weighted masks to calculate the loss of different regions, weighted l 1 The formula of the loss function is: Among them, v i represents the vertex of the i-th region, and α i Represents the corresponding weights. The entire head mesh model is divided into four regions: (1) 68 facial landmarks; (2) nose and mouth; (3) forehead and cheeks; (4) back of the head with ears. The weights of these regions are set to 16:4:3:

0. In order to enhance the diversity of the generated 3D shapes, a region-specific triplet loss is used to train the model. Four region masks are applied. In each training iteration, a positive and negative sample pair of a region is randomly selected, and the region-specific triplet loss is calculated. The formula is as follows: in, is the vertex of the predicted i-th region, v i is the corresponding real vertex, and is its opposite sample, m i and λ i Represent the interval and weight of the corresponding area respectively.

6. The method for generating a three-dimensional face based on a diffusion model according to claim 1, It is characterized in that The StyleGAN2 network in step 7 is specially trained and adjusted to adapt the generation of high-resolution texture maps from 3D shape information. This process uses StyleGAN2 as the backbone network, which adopts a generative adversarial network architecture to process and generate high-quality UV texture maps. In this application, StyleGAN2 accepts a latent space representation generated by CBCT data as a conditional input, as well as randomly generated latent codes. These inputs are mapped to W space, which is able to decouple and independently control different facial features, allowing finer texture detail generation. Specifically, the style enhancer first uses the cross-modal attention mechanism to convert the input image features F i With conditional features Fusion: in here, refers to the last hidden state of the CBCT encoder, and F′ is used to predict the offset Δw of the style code i , denoted as w′ i =w+Δw i , and is used to adjust the current upsampling block: w′ i =w+Δw i =w+MLP(Conv 1×1 (F′ i )) (7) Equipped with a style enhancer, the model is able to better embed CBCT information into the rendering process while maintaining image quality.

Citation Information

Cited By

  • Method and device for generating three-dimensional dental model

    CN119888089A

  • Single-image 3D portrait generation method and system based on mixed prior and noise resampling

    CN121305004A