A high-realism neural rendering method based on diffusion model

Abstract

The application discloses a high-realism neural rendering method based on a diffusion model, and belongs to the field of multi-modal visual generation. A diffusion model is used as a conditional generation model, a semantic encoder is used to compress a face model and a target face part image to a semantic space to extract conditional information, and then a rendering result of the face model under a semantic conditional hidden variable is generated, so that the high realism and accuracy of the generated face image are improved. By learning the mapping relationship of the three-dimensional face model to the two-dimensional image in a display manner, a guiding mechanism is added in the sampling process, the diversity of the generated effect is controlled, and animation editing in various modes is allowed. The application is suitable for the fields of movies, virtual reality and the like, is used for a three-dimensional face model rendering task in an animation production scene, avoids the disadvantages of unstable training of a generative adversarial network and lack of control sample diversity, improves the training efficiency, and enhances the realism of face model rendering.

Description

Technical Field

[0001] This invention relates to a highly realistic neural rendering method based on a diffusion model, belonging to the field of multimodal visual generation. Background Technology

[0002] High-fidelity neural rendering methods can render 3D facial models into 2D images based on a specific person's facial image. In the rendering process, using traditional methods, the textures and materials of the facial model typically require professional artists to create them, followed by computationally intensive physically based ray tracing and texture mapping to obtain the final rendering result. While current technology allows professional artists to create highly realistic skin and hair textures, the time and money involved are often immeasurable.

[0003] At this point, combining 2D images with neural rendering becomes a better solution. The goal of neural rendering methods is to learn the image rendering process through complex neural networks to replace traditional rendering methods, thereby enabling the rapid generation of high-quality images. During training, the neural network learns how to map 3D models onto high-quality rendered images by observing a large number of rendered images and their corresponding light source positions, material properties, etc. This type of method can better handle subtle facial expressions and dynamic changes, resulting in more natural and realistic facial images.

[0004] In principle, the process of learning to map a 3D face model to a 2D image can be achieved by establishing a conditionally constrained generative model. The conditions are determined by information provided by the 3D face model, and the generative model represents the joint probability distribution of the 2D image under the conditional variables. The rendered image is then sampled from the learned generative model. Currently, the commonly used generative model is the Generative Adversarial Network (GAN). The idea behind GANs is to leverage the development of mature discriminative models to advance more complex generative models. However, GANs have long faced training problems such as vanishing gradients and mode collapse, making it difficult to reach the theoretical equilibrium point. Furthermore, due to adversarial training, the diversity of samples generated by GANs is difficult to control. In most methods using GANs, multiple discriminators are often needed to simultaneously control the generator's learning, increasing the required training time. Summary of the Invention

[0005] To address the drawbacks of unstable training and lack of control over sample diversity in generative adversarial networks (GANs), the main objective of this invention is to provide a highly realistic neural rendering method based on a diffusion model. This method employs a diffusion model as the conditional generation model, while utilizing a semantic encoder to compress the face model and the target face image into a semantic space to extract conditional information, thereby improving the realism and accuracy of the generated face images. By explicitly learning the mapping relationship from the 3D face model to the 2D image, a guidance mechanism is added during the sampling process to control the diversity of generated effects, allowing for various animation editing methods.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] This invention discloses a highly realistic neural rendering method based on a diffusion model, comprising the following steps:

[0008] Step 1: Collect a large number of facial images containing multiple specific individuals, and use facial model reconstruction methods to extract facial images from the specific individuals. Each facial image Restore its facial model and camera parameters .

[0009] 1.1 Based on the face model and camera parameters Rendered as a grayscale image of the human face structure .

[0010] 1.2 Random sampling of specific individuals Face image , .

[0011] 1.3 Obtaining Data sets formed by data pairs.

[0012] Step 2: Utilize a pre-trained semantic encoder facial structure image With facial images Data pairs are compressed into a semantic space to obtain semantic conditional latent variables. :

[0013]

[0014] Step 3: Use the predicted conditional diffusion model as a neural rendering decoder. The initial weights, where For the time frame of the diffusion model, After denoising the diffusion model step by step along the reverse time axis Images generated at each time step are used to calculate a loss function to learn the conditional noise mean. The formula is shown below.

[0015]

[0016] in, This is the cutoff time for the diffusion process. express obey Uniform distribution; Indicating the diffusion process The image at any given time, i.e., the image before noise was added; express The initial distribution it follows; This refers to the noise added at each moment during the diffusion process. This indicates that the noise follows a standard normal distribution.

[0017] To better control the diversity of generated images, during the loss calculation process, a certain probability is used... semantic conditional hidden variables Replace with empty variable To learn the unconditional noise mean In the subsequent sampling process, according to The difference guides the generation of the image.

[0018] Step 4: Based on the loss Update the semantic encoder Model parameters The formula is as follows.

[0019]

[0020] in, Indicates the iteration round; Step size; Indicates the parameter The gradient.

[0021] Step 5: Repeat sampling from the dataset facial image As And obtain the semantic conditional latent variables according to step 2. Then, based on steps 3 and 4, train the semantic encoder. Until semantic space alignment.

[0022] Step 6: Based on the loss Update the neural rendering decoder Model parameters The formula is as follows.

[0023]

[0024] in, Indicates the iteration round; Step size; Indicates the parameter The gradient.

[0025] Step 7: Repeat sampling from the dataset facial image As And obtain the semantic conditional latent variables according to step 2. Following steps 3, 4, and 6, a semantic encoder is jointly trained. With Neural Render Decoder This continues until the iterative accuracy is achieved.

[0026] Step 8: Using the semantic encoder obtained through training and neural rendering decoder Generate face model Realistic rendering results This improves the realism and accuracy of generated facial images.

[0027] Step 8.1: From Medium sampling As a diffusion process Noise image at any given time.

[0028] Step 8.2: Through the neural rendering decoder ,calculate unconditional noise mean at time 1 ,in For the estimated diffusion process Noise image at any given time.

[0029] Step 8.3: Input face structure image With facial images Through semantic encoder Obtain semantic conditional latent variables .

[0030] Step 8.4: Through the Neural Render Decoder ,estimate Conditional noise mean at time 1 .

[0031] Step 8.5: Calculate conditional guided noise ,

[0032]

[0033] in, This is a guiding coefficient used to control the diversity of the generated results.

[0034] Step 8.6: Diffusion process The noisy image at each time point is sampled.

[0035]

[0036] in, Follows a standard normal distribution. , , Satisfy the following formula,

[0037]

[0038] Step 8.7, repeat steps 8.2-8.6, let from Decrease to .

[0039] Step 8.8, finally obtaining the face model. The actual infection results .

[0040] Beneficial effects:

[0041] 1. The present invention discloses a high-realism neural rendering method based on a diffusion model. The diffusion model is used as a conditional generation model to avoid the instability of generative adversarial networks, and the rendering results of face models under semantic latent variable conditions are generated, thereby improving the realism and accuracy of the generated face images.

[0042] 2. The present invention discloses a highly realistic neural rendering method based on a diffusion model, which explicitly learns the mapping relationship between a 3D face model and a 2D image, and adds a guidance mechanism during the sampling process to reliably control the diversity of the generated effects, enabling the face animation to be edited in a variety of ways. Attached Figure Description

[0043] Figure 1 This is a flowchart of a highly realistic neural rendering method based on a diffusion model disclosed in this invention;

[0044] Figure 2 This is a network structure diagram of the semantic encoder in this embodiment; Detailed Implementation

[0045] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. The technical problems solved by the present invention and its beneficial effects are also described. It should be noted that the described embodiments are only intended to facilitate understanding of the present invention and do not constitute any limitation thereof.

[0046] This embodiment discloses a highly realistic neural rendering method based on a diffusion model, applied to the rendering task of 3D face models in animation production scenes, such as... Figure 1 As shown, it includes the following steps:

[0047] Step 1: Obtain a large number of face images containing multiple specific individuals from the HDTF public dataset, and use the face model reconstruction method DECA to extract the faces of specific individuals. Each facial image Restore its facial model and camera parameters .

[0048] Furthermore, based on facial models and camera parameters Rendered as a grayscale image of the human face structure .

[0049] Random sampling of specific people Face image , .

[0050] Ultimately, what can be obtained Data sets formed by data pairs.

[0051] Step 2: Utilize a pre-trained semantic encoder facial structure image With facial images Data pairs are compressed into a semantic space to obtain semantic conditional latent variables. The formula is shown below.

[0052]

[0053] The network structure of a semantic encoder is as follows: Figure 2 As shown.

[0054] Step 3: Use the predicted conditional diffusion model as a neural rendering decoder. The initial weights, where For the time frame of the diffusion model, After denoising the diffusion model step by step along the reverse time axis Images generated at each time step are used to calculate a loss function to learn the conditional noise mean. The formula is shown below.

[0055]

[0056] Among the settings This is the cutoff time for the diffusion process. express obey Uniform distribution

[0057] in Indicating the diffusion process The image at time 10:00, i.e., the image before noise was added. Then it means The initial distribution it follows; This refers to the noise added at each moment during the diffusion process. This indicates that the noise follows a standard normal distribution.

[0058] Furthermore, to better control the diversity of generated images, during the loss calculation process, a certain probability is used... semantic conditional hidden variables Replace with empty variable To learn the unconditional noise mean In the subsequent sampling process, according to The difference guides the generation of the image.

[0059] Step 4: Based on the loss Update the semantic encoder Model parameters The formula is as follows.

[0060]

[0061] in Indicates the iteration round; Step size; Indicates the parameter The gradient.

[0062] Step 5: Repeat sampling from the dataset facial image As And obtain the semantic conditional latent variables according to step 2. Following steps 3-4, the semantic encoder is then trained. This continues until the parameters are updated 200,000 times.

[0063] Step 6: Based on the loss Update the neural rendering decoder Model parameters The formula is as follows.

[0064]

[0065] in, Indicates the iteration round; Step size; Indicates the parameter The gradient.

[0066] Step 7: Repeat sampling from the dataset , to take facial images As And obtain the semantic conditional latent variables according to step 2. Following steps 3-4 and 6, a semantic encoder is then jointly trained. With Neural Render Decoder This continues until the parameters are updated 60,000 times.

[0067] Step 8: Using the semantic encoder obtained through training and neural rendering decoder Generate face model Realistic rendering results This improves the realism and accuracy of generated facial images.

[0068] Step 8.1: From Medium sampling As a diffusion process Noise image at any given time.

[0069] Step 8.2: Through the neural rendering decoder ,calculate unconditional noise mean at time 1 ,in For the estimated diffusion process Noise image at any given time.

[0070] Step 8.3: Input face structure image With facial images Through semantic encoder Obtain semantic conditional latent variables .

[0071] Step 8.4: Through the Neural Render Decoder ,estimate Conditional noise mean at time 1 .

[0072] Step 8.5: Calculate conditional guided noise The formula is shown below.

[0073]

[0074] in, This is a guiding coefficient used to control the diversity of the generated results.

[0075] Step 8.6: Diffusion process The noisy image at time t is sampled using the following formula.

[0076]

[0077] in, Follows a standard normal distribution. , , Satisfy the following formula,

[0078]

[0079] Step 8.7, repeat steps 8.2-8.6, let from Decrease to .

[0080] Step 8.8, finally obtaining the face model. The actual infection results It accurately generates highly realistic facial images and allows for various animation editing methods.

[0081] This embodiment uses two full-reference evaluation metrics, PSNR and SSIM, and one no-reference evaluation metric, CPBD, to measure image visual quality.

[0082] PSNR (Peak Signal-to-Noise Ratio) is a commonly used image evaluation metric used to measure the similarity between a distorted image and the original image. PSNR is measured in decibels (dB), and a higher value indicates better image quality. SSIM (Structural Similarity Index) is another widely used image evaluation metric that considers brightness, contrast, and structure of an image simultaneously.

[0083] The SSIM value ranges from 0 to 1, where 1 indicates that the original image and the distorted image are completely identical, and 0 indicates that they have no similarity. Generally speaking, the closer the SSIM value is to 1, the less distortion there is and the better the image quality.

[0084] CPBD (Cumulative Probability of Blur Detection) is a metric used to evaluate image sharpness. Compared to PSNR and SSIM, it is a reference-image-free metric used to detect the degree of blur in an image. Generally speaking, the higher the CPBD value, the sharper the details and the weaker the blur.

[0085] This embodiment is compared with existing methods, and the test results of each method are shown in Table 1 below.

[0086] Table 1 Comparison of test results for various methods

[0087] Zhou et al. 23.2454 0.8020 0.1226 Zhang et al. 24.4174 0.8400 0.1530 Siarohin et al. 23.4079 0.8167 0.1345 Method of the present invention 25.5487 0.9349 0.2096

[0088] Based on the results of the three evaluation metrics, the PSNR, SSIM, and CPBD metrics of the method of this invention show significant improvements compared to other rendering methods. The improved PSNR and SSIM indicate that this method can obtain more accurate face images, while the improved CPBD indicates that the face images generated by this method have clearer details and a stronger sense of realism. Therefore, the test results of this embodiment demonstrate that the present invention can improve the realism and accuracy of generated face images and allows for various animation editing methods.

[0089] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A highly realistic neural rendering method based on a diffusion model, characterized in that: Includes the following steps, Step 1: Collect a large number of facial images containing multiple specific individuals, and use facial model reconstruction methods to extract facial images from each specific individual's face τ. i Recover its face model v i and camera parameters c i ; Step 2: Utilize a pre-trained semantic encoder facial structure image With facial images τ j The data is compressed into the semantic space to obtain the semantic conditional latent variable y: Step 3: Use the predicted conditional diffusion model as the neural rendering decoder. θ (u t The initial weights of u, t, y), where t is the time frame of the diffusion model, u, y ... t For the image generated at time t after progressive denoising along the inverse time axis of the diffusion model, calculate the loss function to learn the conditional noise mean ∈ (u t |y), its formula is shown below. Where T is the cutoff time of the diffusion process, t ~ [1-T] indicates that t follows a uniform distribution of 1-T; u0 represents the image at time 0 during the diffusion process, i.e., the image before the addition of noise; q(u0) represents the initial distribution of u0; ∈ represents the noise added at each time step during the diffusion process. This indicates that the noise follows a standard normal distribution; To better control the diversity of generated images, during the loss calculation process, the semantic conditional latent variable y is replaced with an empty variable with a certain probability p. Learning the unconditional noise mean In the subsequent sampling process, according to The difference guides the generation of the image; Step 4: Based on the loss Update semantic encoder The model parameter ω is given by the following formula. Where k represents the iteration round; r k Step size; This represents the gradient with respect to the parameter ω; Step 5: Repeat sampling from the dataset The face image τ i As u0, the semantic conditional latent variable y is obtained according to step 2; then, according to steps 3 and 4, the semantic encoder is trained. Until semantic space alignment; Step 6: Based on the loss Update Neural Render Decoder ∈ θ (u t The model parameter θ for (t, y) is given by the following formula. Where k represents the iteration round; s k Step size; This represents the gradient with respect to the parameter θ; Step 7: Repeat sampling from the dataset The face image τ i As u0, and obtain the semantic conditional latent variable y according to step 2; then train the semantic encoder together according to steps 3, 4 and 6. With neural rendering decoder ∈ θ Continue until the iteration accuracy is reached; Step 8: Using the semantic encoder obtained through training and neural rendering decoder ∈ θ (u t The realistic rendering result of the generated face model v (t,y) Improve the realism and accuracy of generated facial images.

2. The highly realistic neural rendering method based on a diffusion model as described in claim 1, characterized in that: The implementation method for step 1 is as follows: 1.1 Based on the face model v i and camera parameters c i Rendered as a grayscale image of the human face structure 1.2 Randomly sample facial images τ of a specific person ξ j i≠j; 1.3 Obtaining Data sets formed by data pairs.

3. The highly realistic neural rendering method based on a diffusion model as described in claim 2, characterized in that: The implementation method for step 8 is as follows: Step 8.1, from Mid-sampling u T , as the noise image at time T during the diffusion process; Step 8.2, via the neural rendering decoder ∈ θ Calculate the unconditional noise mean at time t. in This is the noise image at time t during the estimated diffusion process; Step 8.3: Input face structure image I v With facial image τ, via semantic encoder We obtain the semantic conditional latent variable y; Step 8.4, via the neural rendering decoder ∈ θ Estimate the mean of the conditional noise at time t Step 8.5: Calculate the guided noise under conditions Where s is the guiding coefficient, used to control the diversity of the generated results; Step 8.6: Sample the noise image at time t-1 during the diffusion process. Among them, z t Follows a standard normal distribution, α t =1-β t , β t Satisfy the following formula, Step 8.7: Repeat steps 8.2-8.6, letting t decrease from T to 1; Step 8.8: Finally, obtain the true infection result of the face model v.

Images