A system and method for restoring blind face images based on latent diffusion model

Abstract

The present invention discloses a system and method for restoring blind facial images based on a latent diffusion model, belonging to the field of image restoration technology. The method comprises the following steps: receiving a transmitted low-quality facial image; obtaining an edge distribution using an SRCNN network, and inputting the constructed edge distribution into a pre-trained latent diffusion model; using the SRCNN network as a diffusion estimator for image feature extraction; generating a latent facial image using the pre-trained latent diffusion model; and restoring the latent facial image using a VAE decoder. The present invention integrates the latent diffusion model with the SRCNN network to perform a transition distribution from the low-quality image to the intermediate state of the pre-trained diffusion model, thereby reducing pre-training complexity and increasing dataset diversity. By using a pre-trained U-net network and a scheduling algorithm as a generative model, an inverse diffusion process is performed in the latent space, simplifying the training process and reducing training time, thereby generating more accurate, high-quality facial images.

Description

Technical Field

[0001] The present invention belongs to the technical field of image restoration, and in particular relates to a system and method for restoring a blind face image based on a latent diffusion model. Background Art

[0002] Blind face restoration aims to restore high-quality faces from low-quality images that suffer from unknown degradations (such as low resolution, noise, blur, compression artifacts, etc.). In blind face restoration, traditional methods typically utilize face-specific priors in face restoration, and the prior information contained is limited. With the development and deepening of deep learning, the application of deep learning to image restoration has become the norm. Deep learning is a complex machine learning algorithm that has achieved results in image recognition that far exceed previous related technologies. It can obtain more effective hidden features in the data by learning the deep features of the data. These features can be used to process the data more efficiently, making it possible to train more complex models.

[0003] One solution for blind face restoration is to embed image features into a GAN network to generate high-quality, specified facial images. This model consists of a generator and a discriminator, which continuously compete with each other throughout the training process to learn and optimize the generator's output image. Another solution is to use a diffusion model to continuously add and remove noise from the image to train the decoder and encoder to restore blind face images. However, these methods have complex sampling processes and are expensive to train, and they also lack dataset diversity. Summary of the Invention

[0004] The purpose of the present invention is to provide a system and method for restoring blind face images based on a latent diffusion model to solve the problems raised in the above background technology.

[0005] The object of the present invention is achieved as follows: a system and method for restoring a blind face image based on a latent diffusion model, characterized in that: the system comprises an SRCNN network used as a diffusion estimator, a latent diffusion model used to generate a latent face image, and a VAE decoder used to restore the latent face image;

[0006] The SRCNN network is used as a diffusion estimator for image feature extraction.

[0007] Preferably, the SRCNN network includes an image feature extraction layer, a nonlinear mapping layer and a network reconstruction layer; the image feature extraction layer extracts the edge distribution of the image, the nonlinear mapping layer maps the constructed edge distribution to a pre-trained potential diffusion model, and the network reconstruction layer restores the mapped features to a high-quality image.

[0008] Preferably, the latent diffusion model comprises a U-net network, the U-net network comprising an encoder composed of ResNet blocks, a decoder composed of ResNet blocks and a cross attention layer;

[0009] The encoder compresses the potential image representation into a low-dimensional representation by downsampling, and the decoder restores the low-dimensional representation of the potential image by upsampling;

[0010] The cross attention layer is between the ResNet blocks of the encoder and decoder, and the correlation between the image edge distribution and Latents is extracted through the cross attention layer.

[0011] Preferably, the SRCNN network is trained using the FFHQ training set, which uses a typical degradation model to synthesize low-quality images to train the SRCNN network;

[0012] The expression of the typical degradation model is as follows:

[0013]

[0014] Among them, y is the low quality image (LQ), x is the high quality image (HQ), k l is a Gaussian kernel with a kernel width of l, n σ is Gaussian noise with standard deviation σ, ↓s and ↑s are bicubic sampling operators with scale s, and [·]JPEGq denotes the JPEG compression process with quality factor q.

[0015] A method for restoring a blind face image based on a latent diffusion model, characterized in that the method comprises the following steps:

[0016] Step S1: receiving a low-quality face image transmitted;

[0017] Step S2: Obtain the marginal distribution using the SRCNN network and input the constructed marginal distribution into the pre-trained latent diffusion model;

[0018] Step S3: Generate a latent face image using a pre-trained latent diffusion model;

[0019] Step S4: Use the VAE decoder to restore the potential face image.

[0020] Preferably, in step S2, the edge distribution is obtained using the SRCNN network, and the specific operations are as follows:

[0021] First, the low-quality face image is upsampled to the required size through bilateral interpolation. The edge feature blocks of the sampled image are extracted through the convolution operation of the image feature extraction layer, and each feature is represented as a high-dimensional vector.

[0022] The SRCNN network is used as a diffusion estimator f(·; W), constructing x N ~p(x N |y0) approximates a diffuse version of x0 N The marginal distribution of

[0023] Among them, x0 is the high-quality face image corresponding to the low-quality face image; x N is the diffuse version of x0.

[0024] Preferably, in step S2, the constructed edge distribution is input into a pre-trained potential diffusion model through a nonlinear mapping layer of the SRCNN network.

[0025] Preferably, in step S3, a pre-trained latent diffusion model is used to generate a latent face image. The latent diffusion model is a forward diffusion process and a reverse diffusion process in a latent space. The forward diffusion process is used to gradually add Gaussian noise to the sampled image, and the reverse diffusion process is used to gradually reduce the noise image. The specific operations are as follows:

[0026] Step S3-1: train the U-net network using the data set, cyclically update the U-net network, and use the U-net network to predict noise to obtain the previous noise result;

[0027] Step S3-2: Calculate the cross entropy loss between the predicted noise and the real noise, and update the parameters in the U-net network through the cross entropy loss;

[0028] Step S3-3: The U-net network outputs a predicted noise residual, where the predicted noise residual refers to the difference between the predicted noise sample and the true noise sample; the predicted noise sample is represented by Latents, and the predicted noise residual is represented by conditioned Latents;

[0029] Step S3-4: The amount of noise added is determined using a scheduling algorithm, and the denoised latent image representation is calculated using the scheduler algorithm. The conditioned latents are input into the scheduler, and the scheduler algorithm calculates a predicted denoised image representation based on the previous noise representation and the predicted noise residual. The calculated predicted denoised image representation is used as the new latents for the next cycle.

[0030] Compared with the existing technology, the present invention has the following improvements and advantages: 1. By fusing the potential diffusion model through the SRCNN network, the low-quality image is transitionally distributed to the intermediate state of the pre-trained diffusion model, which reduces the pre-training complexity and improves the diversity of the data set.

[0031] 2. By using the pre-trained U-net network and scheduling algorithm as the generation model, the inverse diffusion process is completed in the latent space, which further reduces the pre-training complexity, simplifies the training process and shortens the training time, thereby generating more accurate and high-quality face images. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of the process of the present invention.

[0033] Figure 2 This is the SRCNN network structure diagram. DETAILED DESCRIPTION

[0034] The present invention is further summarized below with reference to the accompanying drawings.

[0035] like Figure 1 As shown, a system for restoring blind facial images based on a latent diffusion model includes an SRCNN network used as a diffusion estimator, a latent diffusion model for generating a latent facial image, and a VAE decoder for restoring the latent facial image. The SRCNN network, used as a diffusion estimator, is used for image feature extraction. The SRCNN network includes an image feature extraction layer, a nonlinear mapping layer, and a network reconstruction layer. The image feature extraction layer extracts the edge distribution of the image. The nonlinear mapping layer maps the constructed edge distribution to the pre-trained latent diffusion model. The network reconstruction layer restores the mapped features to a high-quality image. The image feature extraction layer upsamples the input low-quality facial image to the desired size through bilateral interpolation. The sampled image is then passed through this layer for convolution to extract edge feature blocks. Each feature is represented as a high-dimensional vector, thereby extracting the edge distribution of the image.

[0036] Furthermore, the latent diffusion model includes a U-net network, which includes an encoder composed of ResNet blocks, a decoder composed of ResNet blocks and a cross-attention layer; the encoder compresses the latent image representation into a low-dimensional representation by downsampling, and the decoder restores the latent image of the low-dimensional representation by upsampling; the cross-attention layer is between the ResNet blocks of the encoder and decoder, and the cross-attention layer extracts the correlation between the image edge distribution and the Latents.

[0037] The edge distribution of the input image is mapped to the U-net layer through the cross-attention layer, and the image representation in Latents is merged to complete the embedding and fusion of image features, so that the next U-net layer can utilize the image information in this process.

[0038] A method for restoring blind face images based on a latent diffusion model, the method comprising the following steps:

[0039] Step S1: receiving a low-quality face image transmitted;

[0040] Step S2: Obtain the marginal distribution using the SRCNN network and input the constructed marginal distribution into the pre-trained latent diffusion model;

[0041] First, the low-quality face image is upsampled to the required size through bilateral interpolation. The edge feature blocks of the sampled image are extracted through the convolution operation of the image feature extraction layer, and each feature is represented as a high-dimensional vector. The SRCNN network is used as the diffusion estimator f(·; W) to construct x N ~p(x N |y0) approximates a diffuse version of x0 N The marginal distribution of

[0042] Among them, x0 is the high-quality face image corresponding to the low-quality face image; x N is the diffuse version of x0.

[0043] The SRCNN network is trained using the FFHQ training set, which uses a typical degradation model to synthesize low-quality images to train the SRCNN network;

[0044] The expression of the typical degradation model is as follows:

[0045]

[0046] Among them, y is the low quality image (LQ), x is the high quality image (HQ), k l is a Gaussian kernel with a kernel width of l, n σ is Gaussian noise with standard deviation σ, ↓s and ↑s are bicubic sampling operators with scale s, and [·]JPEGq denotes the JPEG compression process with quality factor q.

[0047] x and Gaussian kernel k l Perform convolution, downsample the convolution result with bicubic scale s, combine it with Gaussian noise, and then perform JPEG compression with a quality factor q. The compressed result is then upsampled with bicubic scale s to produce a low-quality image y. x represents data from the FFHQ dataset, and y represents the constructed training dataset.

[0048] In step S2, the constructed marginal distribution is input into the pre-trained latent diffusion model through the nonlinear mapping layer of the SRCNN network.

[0049] Use the traditional L2 loss function to make the generated image and the training image as similar as possible. The L2 loss function can be expressed as:

[0050]

[0051] where {x (i) ,y (i)} is a synthetic image pair, f(y (i) ; w) is the diffusion estimator.

[0052] Step S3: Generate a latent face image using a pre-trained latent diffusion model;

[0053] In step S3, a pre-trained latent diffusion model is used to generate a latent face image. The latent diffusion model is a forward diffusion process and a reverse diffusion process in the latent space. The forward diffusion process is used to gradually add Gaussian noise to the sampled image, and the reverse diffusion process is used to gradually reduce the noise image. The specific operations are as follows:

[0054] Step S3-1: train the U-net network using the data set, cyclically update the U-net network, and use the U-net network to predict noise to obtain the previous noise result;

[0055] Images are sampled from the FFHQ dataset, and randomly generated Gaussian noise is added to the sampled images. The noisy images are used as training samples. A certain number of images are sampled from the FFHQ dataset, and 10 training samples are generated for each image. The training samples are used to train the U-net network.

[0056] Step S3-2: Calculate the cross entropy loss between the predicted noise and the real noise, and update the U-net network parameters; use the forward diffusion process of the diffusion model to train the U-net network, and the trained U-net network is only used in the reverse process of the diffusion model.

[0057] Step S3-3: The U-net network outputs a predicted noise residual, where the predicted noise residual refers to the difference between the predicted noise sample and the true noise sample; the predicted noise sample is represented by Latents, and the predicted noise residual is represented by conditioned Latents;

[0058] Generate images using the inverse diffusion process of the latent diffusion model; convert Gaussian noise samples into latent image representations in the latent space. In the loop, the latents and the marginal distribution generated by the diffusion estimator are input into the trained U-net network together, and the conditioned latents are output.

[0059] Step S3-4: The amount of noise added is determined using a scheduling algorithm, and the denoised latent image representation is calculated using the scheduler algorithm. The conditioned latents are input into the scheduler, and the scheduler algorithm calculates a predicted denoised image representation based on the previous noise representation and the predicted noise residual. The calculated predicted denoised image representation is used as the new latents for the next cycle.

[0060] The pre-trained U-net network and scheduling algorithm serve as a generative model to complete the diffusion process in the latent space;

[0061] Step S4: Use the VAE decoder to restore the potential face image.

[0062] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A system for restoring blind face images based on a latent diffusion model, characterized by: The blind face image restoration system includes an SRCNN network used as a diffusion estimator, a potential diffusion model for generating a potential face image, and a VAE decoder for restoring the potential face image; The SRCNN network is used as a diffusion estimator for image feature extraction; The SRCNN network includes an image feature extraction layer, a nonlinear mapping layer, and a network reconstruction layer; the image feature extraction layer extracts the edge distribution of the image, the nonlinear mapping layer maps the constructed edge distribution to a pre-trained latent diffusion model, and the network reconstruction layer restores the mapped features to a high-quality image; The latent diffusion model includes a U-net network, the U-net network includes an encoder composed of ResNet blocks, a decoder composed of ResNet blocks and a cross attention layer; The encoder compresses the potential image representation into a low-dimensional representation by downsampling, and the decoder restores the low-dimensional representation of the potential image by upsampling; The cross attention layer is between the ResNet blocks of the encoder and decoder, and the correlation between the image edge distribution and Latents is extracted through the cross attention layer; The Latents represent the predicted noise samples.

2. The system for restoring blind face images based on a latent diffusion model according to claim 1, characterized in that: The SRCNN network is trained using the FFHQ training set, which uses a typical degradation model to synthesize low-quality images to train the SRCNN network; The expression of the typical degradation model is as follows: ； Among them, y is the low quality image (LQ), x is the high quality image (HQ), The kernel width is Gaussian kernel, The standard deviation is where ↓s and ↑s are the Gaussian noise, ↓s and ↑s are bicubic sampling operators with scale s, and [·]JPEGq denotes the JPEG compression process with quality factor q.

3. The method for restoring blind face images based on a latent diffusion model according to any one of claims 1 to 2, characterized in that: The method comprises the following steps: Step S1: receiving a low-quality face image transmitted; Step S2: Obtain the marginal distribution using the SRCNN network and input the constructed marginal distribution into the pre-trained latent diffusion model; Step S3: Generate a latent face image using a pre-trained latent diffusion model; Step S4: Use the VAE decoder to restore the potential face image.

4. The method according to claim 3, wherein: In step S2, the SRCNN network is used to obtain the edge distribution, and the specific operations are as follows: First, the low-quality face image is upsampled to the required size through bilateral interpolation. The edge feature blocks of the sampled image are extracted through the convolution operation of the image feature extraction layer, and each feature is represented as a high-dimensional vector. The SRCNN network is used as a diffusion estimator f(·; W), constructing approximate The diffusion version The marginal distribution of in, A high-quality face image corresponding to a low-quality face image; for The diffuse version of .

5. The method according to claim 3, wherein: In step S2, the constructed edge distribution is input into the pre-trained potential diffusion model through the nonlinear mapping layer of the SRCNN network.

6. The method according to claim 3, wherein: In step S3, a pre-trained latent diffusion model is used to generate a latent face image. The latent diffusion model is a forward diffusion process and a reverse diffusion process in a latent space. The forward diffusion process is used to gradually add Gaussian noise to the sampled image, and the reverse diffusion process is used to gradually reduce the noise image. The specific operations are as follows: Step S3-1: train the U-net network using the data set, cyclically update the U-net network, and use the U-net network to predict noise to obtain the previous noise result; Step S3-2: Calculate the cross entropy loss between the predicted noise and the real noise, and update the parameters in the U-net network through the cross entropy loss; Step S3-3: The U-net network outputs a predicted noise residual, where the predicted noise residual refers to the difference between the predicted noise sample and the true noise sample; the predicted noise residual is represented by conditioned latents; Step S3-4: The scheduling algorithm determines the amount of noise to add and the scheduler algorithm calculates the denoised latent image representation. The conditioned latents are input to the scheduler algorithm, which calculates the predicted denoised image representation based on the previous noise representation and the predicted noise residual. The calculated predicted denoised image representation is used as the new latents for the next cycle.

Images