Face image restoration model training method, face image restoration method and equipment

Abstract

The invention discloses a face image restoration model training method, and the method comprises the steps: training a preset diffusion model based on first training data, and obtaining a first diffusion model, the first training data comprising a simulated low-definition face image and a high-definition face image; inputting the low-definition face image into a first diffusion model to generate a first pseudo high-definition face image, constructing second training data by using the low-definition face image, the first pseudo high-definition face image, the low-definition face image and the high-definition face image, training the first diffusion model, updating model parameters, and obtaining a second diffusion model; and inputting the low-definition face image into a second diffusion model to generate a second pseudo high-definition face image, constructing third training data by using the low-definition face image, the second pseudo high-definition face image, the low-definition face image and the high-definition face image, and training the second diffusion model to update the model parameters so as to obtain a face image restoration model. According to the method, the difference between the training data and the real data is reduced, and the similarity between the repaired image and the real identity is effectively improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a training method for a face image restoration model, a face image restoration method, and a device. Background Technology

[0002] In security scenarios, identifying suspicious targets is a crucial procedure for solving cases. However, due to the environmental conditions in which surveillance cameras are captured, the image quality is often low, making it difficult to identify individuals from the images. Therefore, there is a need for image restoration that preserves the identity of the people in the images. Because of the inherent differences between the training data and the data to be restored, current neural network models perform poorly in image restoration, often resulting in altered identities after restoration. Summary of the Invention

[0003] The purpose of this application is to provide a training method for a face image restoration model, which reduces the difference between training data and real data and can effectively improve the similarity between the restored image and the real identity.

[0004] To achieve the above objectives, this application provides a method for training a face image restoration model, the method comprising:

[0005] The first diffusion model is obtained by training a preset diffusion model based on the first training data, wherein the first training data includes pairs of simulated low-resolution face images and high-resolution face images.

[0006] Multiple low-resolution face images are input into the first diffusion model to generate multiple first pseudo-high-resolution face images. Multiple pairs of low-resolution face images and first pseudo-high-resolution face images, as well as multiple pairs of low-resolution face images and high-resolution face images, are used to construct the second training data. The first diffusion model is trained based on the second training data to update the model parameters of the first diffusion model and obtain the second diffusion model.

[0007] Multiple low-resolution face images are input into the second diffusion model to generate multiple second pseudo-high-resolution face images. Multiple pairs of low-resolution face images and second pseudo-high-resolution face images, as well as multiple pairs of low-resolution face images and high-resolution face images, are used to construct the third training data. The second diffusion model is trained based on the third training data to update the model parameters of the second diffusion model, thereby obtaining the face image restoration model.

[0008] Furthermore, the preset diffusion model includes an encoder, a diffusion module, a conditional control module, and a decoder. The diffusion module is a stable diffusion model, and the conditional control module is a control network. The preset diffusion model is trained based on the first training data to obtain a first diffusion model, which includes:

[0009] Using an image degradation algorithm, a high-definition face image is degraded to obtain a corresponding simulated low-definition face image;

[0010] The high-definition face image is encoded by an encoder to obtain the latent space feature map corresponding to the high-definition face image;

[0011] Based on the sampling noise, the latent space feature map is noise-added through the forward diffusion process of the diffusion module to obtain a noisy latent space feature map.

[0012] The attention features are obtained by processing the simulated low-resolution face image based on the attention mechanism. The attention features are then input into the conditional control module for processing to obtain the conditional control features.

[0013] Using conditional control features as conditional information, the denoising network of the diffusion module is used to denoise the noisy latent space feature map, and the prediction noise is determined based on the denoised latent space feature map.

[0014] The loss function of the preset diffusion model is determined based on the predicted noise and the sampled noise. The preset diffusion model is trained until the output value of the loss function of the preset diffusion model is less than or equal to the preset threshold. Training is then stopped, and the first diffusion model is obtained.

[0015] Furthermore, the encoder and decoder are paired encoders and decoders, and the model training for the encoder and decoder includes:

[0016] High-resolution facial images are used as training data; the high-resolution facial images are input into a preset encoder.

[0017] The pre-defined encoder maps high-resolution face images to the latent space, obtaining the corresponding latent space feature map;

[0018] The latent space feature map is denoised by adding and removing noise using a pre-defined diffusion model to obtain a denoised latent space feature map.

[0019] The preset decoder reconstructs the image from the denoised latent space feature map;

[0020] Based on the training data, the preset encoder and preset decoder are trained to obtain the trained encoder and decoder. During the training of the preset diffusion model, the first diffusion model and the second diffusion model, the network parameters of the trained encoder and decoder are frozen.

[0021] Furthermore, the simulated low-resolution face image is processed using an attention mechanism to obtain attention features. These attention features are then input into a conditional control module for further processing to obtain conditional control features, including:

[0022] The first feature map is obtained by extracting features from the simulated low-resolution face image;

[0023] The first feature map is transformed using a multilayer perceptron to obtain the second feature map;

[0024] The second feature map is divided into multiple feature blocks, and these feature blocks are input into a linear embedding layer to obtain a token sequence.

[0025] Input the token sequence into the self-attention network to obtain self-attention features;

[0026] The self-attention features are processed using the UnPatchify function so that the dimension of the processed self-attention features is the same as the dimension of the first feature map.

[0027] Furthermore, the denoising network of the diffusion module is a Unet network, which includes a denoising encoder, intermediate layers, and a denoising decoder. The denoising encoder includes multiple cascaded denoising coding layers, and the denoising decoder includes multiple cascaded denoising decoding layers. The control network includes multiple cascaded conditional coding layers, intermediate layers, and multiple zero-convolutional layers. The multiple cascaded conditional coding layers correspond one-to-one with the multiple cascaded denoising decoding layers. Using conditional control features as conditional information, the denoising network of the diffusion module denoises the noisy latent space feature map. Based on the denoised latent space feature map, the predicted noise is determined, including:

[0028] The noisy latent space feature map is input into the denoising network, and the noisy latent space feature map is encoded by the denoising encoder to obtain the intermediate encoding vector.

[0029] The self-attention features are input into the control network, and the self-attention features are encoded sequentially through multiple cascaded conditional coding layers in the control network to obtain the conditional control features corresponding to each conditional coding layer.

[0030] The intermediate encoded vector is input into the denoising decoder. The intermediate encoded vector is decoded sequentially by multiple cascaded denoising decoding layers in the denoising decoder. Each denoising decoding layer decodes the vector input to the denoising decoding layer based on the conditional control features of the corresponding conditional encoding layer. The output of the last denoising decoding layer of the multiple cascaded denoising decoding layers is obtained to obtain the denoised latent space feature map.

[0031] The predicted noise is determined based on the denoised latent space feature map.

[0032] Furthermore, the first diffusion model is trained based on the second training data to update the model parameters of the first diffusion model, resulting in a second diffusion model, including:

[0033] The second training data is input into the first diffusion model to obtain the first prediction noise and the first predicted high-definition face image output by the first diffusion model based on the low-resolution face image and the first pseudo high-definition face image, and / or the low-resolution face image and the high-definition face image. The first predicted high-definition face image is input into the pre-trained identity classifier to obtain the identity prediction result corresponding to the first predicted high-definition face image. The images in the second training data have identity labels.

[0034] The loss function of the first diffusion model is determined based on the difference between random noise and first prediction noise in the forward diffusion process of the first diffusion model, and the difference between identity prediction results and identity labels.

[0035] The model parameters of the first diffusion model are iteratively adjusted using the loss function of the first diffusion model to obtain the trained second diffusion model.

[0036] Furthermore, multiple low-resolution face images are input into a first diffusion model to generate multiple first pseudo-high-resolution face images, including:

[0037] Obtain multiple facial features corresponding to multiple low-resolution facial images, where the multiple low-resolution facial images have the same identity information;

[0038] Randomly sampled noise and low-resolution face images are input into the first diffusion model;

[0039] The first diffusion model uses a denoising network and a conditional control module to denoise random sampling noise based on the current facial features corresponding to the low-resolution face image. Noise prediction is then performed based on the consistency constraint of multi-image identity information. The consistency constraint of multi-image identity information is as follows: at the same sampling time, the similarity between the facial features of the i-th first pseudo-high-resolution face image output by the i-th low-resolution face image and the facial features of the j-th first pseudo-high-resolution face image output by the j-th low-resolution face image is obtained. The gradient is calculated based on this similarity and fed back to the prediction noise at the sampling time to obtain the prediction noise at the sampling time.

[0040] The predicted latent space feature map is obtained by progressively subtracting the prediction noise corresponding to each sampling time from the random sampling noise.

[0041] The predicted latent space feature map is processed by image decoding to obtain the first pseudo-high-definition face image.

[0042] Furthermore, the second diffusion model is trained based on the third training data to update its model parameters, resulting in a face image restoration model, including:

[0043] The third training data is input into the second diffusion model to obtain the second prediction noise and the second predicted high-definition face image output by the second diffusion model based on the low-definition face image and the second pseudo high-definition face image, and / or the low-definition face image and the high-definition face image. The second predicted high-definition face image is input into the pre-trained image quality scorer and identity judge respectively to obtain the image quality score and identity preservation result corresponding to the second predicted high-definition face image respectively.

[0044] Based on the difference between random noise and second prediction noise in the forward diffusion process of the second diffusion model, the image quality score, and the identity preservation results, the loss function of the second diffusion model is determined.

[0045] The model parameters of the second diffusion model are iteratively adjusted using the loss function of the second diffusion model to obtain a trained face image restoration model.

[0046] Further, the model training for the image quality scorer includes:

[0047] Multiple low-resolution face images are input into the second diffusion model to obtain multiple second pseudo-high-resolution face images;

[0048] Based on a semi-supervised method, the quality of multiple second pseudo-high-definition face images is scored to obtain the quality score data corresponding to each second pseudo-high-definition face image;

[0049] Based on multiple quality score data and multiple second pseudo-high-definition face images, a model is trained on the preset image quality scorer to obtain the trained image quality scorer.

[0050] And / or,

[0051] The model training for the identity determiner includes:

[0052] Multiple low-resolution face images are input into the second diffusion model to obtain multiple second pseudo-high-resolution face images;

[0053] Based on a semi-supervised method, the similarity between the identity information of each second pseudo-high-definition face image and the real identity information of the corresponding low-definition face image is obtained, thus obtaining the identity similarity data corresponding to each second pseudo-high-definition face image.

[0054] Based on multiple identity similarity data and multiple second pseudo-high-definition face images, a model is trained on a preset identity determiner to obtain the trained identity determiner.

[0055] To achieve the above objectives, this application provides a method for face image restoration, the method comprising:

[0056] The target face image to be repaired and random sampled noise are input into a pre-trained face image repair model, which is trained based on the method described above.

[0057] Based on the target face image to be repaired, the random sampled noise is iteratively denoised to generate the repaired target face image.

[0058] For the purposes described above, this application provides a computer device, comprising:

[0059] The processor, communication interface, memory, and communication bus are connected, with the processor, communication interface, and memory communicating with each other via the communication bus.

[0060] Memory, used to store computer programs;

[0061] When a processor executes the computer program stored in memory, it implements the steps of the training method for the face image restoration model as described above, or the steps of the face image restoration method as described above.

[0062] This application uses low-resolution facial images from real-world scenarios to train the diffusion model, reducing the discrepancy between training and real data. The trained facial image restoration model can optimize the preservation of identity information in real degraded images and effectively improve the similarity between the restored image and the real identity. After processing by the facial image restoration model, images taken with low-resolution cameras can have their identity information determined, thereby improving the monitoring capabilities of areas with deployed old cameras. Attached Figure Description

[0063] Figure 1 This is a first flowchart of a training method for a face image restoration model provided according to an embodiment of this application;

[0064] Figure 2 This is a schematic diagram of the network structure of the preset diffusion model provided in the embodiments of this application;

[0065] Figure 3 This is a second flowchart of a training method for a face image restoration model provided according to an embodiment of this application;

[0066] Figure 4 This is a schematic diagram of the structure of the denoising network and the control network provided in the embodiments of this application;

[0067] Figure 5 This is a third flowchart of a training method for a face image restoration model provided according to an embodiment of this application;

[0068] Figure 6This is the fourth flowchart of the training method for the face image restoration model provided in the embodiments of this application;

[0069] Figure 7 This is a schematic diagram of the network structure of the first diffusion model and identity classifier provided in the embodiments of this application;

[0070] Figure 8 This is the fourth flowchart of the training method for the face image restoration model provided in the embodiments of this application;

[0071] Figure 9 This is a schematic diagram of the network structure of the second diffusion model provided in the embodiments of this application;

[0072] Figure 10 This is a first flowchart of a face image restoration method provided according to an embodiment of this application;

[0073] Figure 11 This is a system block diagram of a training device for a face image restoration model provided in the embodiments of this application;

[0074] Figure 12 This is a system block diagram of a face image restoration apparatus provided according to an embodiment of this application;

[0075] Figure 13 This is a schematic diagram of the structure of a computer device provided according to an embodiment of this application. Detailed Implementation

[0076] The present application will be described in detail below with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present application. Any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the protection scope of the present application.

[0077] In related technical solutions, training a face image restoration model requires acquiring paired low-resolution and high-resolution image data for model training. However, this data is generally unavailable, necessitating the degradation of high-resolution face images to low-resolution images using formulas after obtaining the high-resolution images. The low-resolution face images processed by this approach often differ from real-world low-resolution images. Using degraded low-resolution face images for model training limits the model's restoration performance and also affects the preservation of identity information in the restored image. This application uses real-world low-resolution face images for training the diffusion model, reducing the discrepancy between training and real data and effectively improving the similarity between the restored image and the real identity.

[0078] Please refer to Figure 1This application provides a method for training a face image restoration model. The training method includes steps S101-S103.

[0079] Step S101: Train the preset diffusion model based on the first training data to obtain the first diffusion model, wherein the first training data includes pairs of simulated low-resolution face images and high-resolution face images.

[0080] In this application embodiment, the high-definition face image is real image data, and the simulated low-definition face image is low-definition face image data obtained by applying a certain degradation mode to the high-definition face image, or is based on the constructed simulated low-definition face image data.

[0081] For example, a high-resolution face image is downsampled, subjected to a blur kernel convolution operation, and has noise added to degrade it into a simulated low-resolution face image. The image degradation process can be represented by the following formula:

[0082]

[0083] in, I represents a simulated low-resolution human face image. hq Represents a high-resolution facial image, ↓ s This represents a downsampling by a factor of s, where n is the number of samples. σ j represents Gaussian noise with standard deviation σ. peg -q indicates image compression with a compression ratio of q.

[0084] In this embodiment, the preset diffusion model, the first diffusion model, the second diffusion model, and the face image restoration model have the same network structure, but the network parameters of the above diffusion models are different. The preset diffusion model can be a stable diffusion model network with a conditional control network. The denoising network in the diffusion model can be an image denoising network based on stable diffusion, and the conditional control network can be a control network (ControlNet). The stable diffusion model includes a forward process (diffusion process) and a backward process (denoising process). The forward process (diffusion process) generally adds noise (standard Gaussian distribution) to the input data in an iterative manner to obtain the final noisy image. The backward process is the reverse process of the forward process, gradually restoring the original image without noise.

[0085] The preset diffusion model is trained using the first training data. During the model training process, the model parameters of the preset diffusion model are updated using the gradient of the loss function, thereby obtaining the trained first diffusion model.

[0086] Step S102: Input multiple low-resolution face images into the first diffusion model to generate multiple first pseudo-high-resolution face images. Construct multiple pairs of low-resolution face images and first pseudo-high-resolution face images, as well as multiple pairs of low-resolution face images and high-resolution face images, into second training data. Train the first diffusion model based on the second training data to update the model parameters of the first diffusion model and obtain the second diffusion model.

[0087] In this embodiment of the application, low-resolution face images refer to real image data, not low-resolution face image data obtained based on a certain degradation mode, nor simulated low-resolution face image data constructed.

[0088] Using a pre-trained first diffusion model, a low-resolution face image is input into the first diffusion model, and a first pseudo-high-resolution face image is output. Pairs of low-resolution face images and the first pseudo-high-resolution face image are input into the first diffusion model for training. Real low-resolution face images and high-resolution face images are also input into the first diffusion model for training. During the training process, the model parameters of the first diffusion model are updated, and the first diffusion model is further optimized.

[0089] Step S103: Input multiple low-resolution face images into the second diffusion model to generate multiple second pseudo-high-resolution face images. Construct multiple pairs of low-resolution face images and second pseudo-high-resolution face images, as well as multiple pairs of low-resolution face images and high-resolution face images, into third training data. Train the second diffusion model based on the third training data to update the model parameters of the second diffusion model and obtain the face image restoration model.

[0090] In this embodiment, the low-resolution face image refers to real image data. Using a pre-trained second diffusion model, the low-resolution face image is input into the second diffusion model, outputting a second pseudo-high-resolution face image. Pairs of low-resolution face images and second pseudo-high-resolution face images are input into the second diffusion model for training. Real low-resolution and high-resolution face images are also input into the second diffusion model for training. During training, the model parameters of the second diffusion model are updated, and the model is further optimized. After training the second diffusion model, a face image restoration model is obtained.

[0091] In this embodiment, low-resolution facial images from real-world scenes are used to train the diffusion model, reducing the difference between training data and real data. The trained facial image restoration model can optimize the preservation of identity information in real degraded images and effectively improve the similarity between the restored image and the real identity.

[0092] Please refer to Figure 2This application provides a network structure diagram of a preset diffusion model. The preset diffusion model includes an encoder, a diffusion module, a conditional control module, and a decoder. The diffusion module is a stable diffusion model, and the conditional control module is a control network. The denoising network in the diffusion module is a Unet network, including a denoising encoder and a denoising decoder, and the control network includes a conditional encoder.

[0093] For example, the encoder and decoder are paired encoders and decoders. Based on a pre-defined diffusion model, the encoder and decoder are trained to determine their model parameters. During training, high-resolution face images are used as training data. The high-resolution face images are input into the pre-defined encoder, which maps the high-resolution face images from the image space to the latent space, obtaining the corresponding latent space feature map. The latent space feature map is then subjected to noise addition and denoising processing through the pre-defined diffusion model to obtain a denoised latent space feature map. The pre-defined decoder reconstructs the image from the denoised latent space feature map. Based on the training data, the pre-defined encoder and decoder are trained to obtain trained encoders and decoders. During the training of the pre-defined diffusion model, the first diffusion model, and the second diffusion model, the network parameters of the trained encoder and decoder are frozen.

[0094] Please refer to Figure 3 and Figure 2 The steps for training a pre-defined diffusion model based on the first training data to obtain the first diffusion model include:

[0095] Step S301: Using an image degradation algorithm, the high-definition face image is degraded to obtain a corresponding simulated low-definition face image.

[0096] Image degradation algorithms can degrade images according to the degradation formulas described above.

[0097] Step S302: Encode the high-definition face image using an encoder to obtain the latent space feature map corresponding to the high-definition face image.

[0098] The high-definition face image is processed by an encoder, mapping the high-definition face image from the image space to the latent space.

[0099] Step S303: Based on the sampling noise, the latent space feature map is noise-added through the forward diffusion process of the diffusion module to obtain a noisy latent space feature map.

[0100] During the forward diffusion process of the diffusion module, sampling noise is added to the latent space feature map at each sampling time according to the sampling noise and diffusion duration until a pure noise image is obtained. For example, random noise is Gaussian noise.

[0101] Step S304: The simulated low-resolution face image is processed based on the self-attention mechanism to obtain self-attention features. The self-attention features are then input into the conditional control module for processing to obtain conditional control features.

[0102] In this embodiment, a simulated low-resolution face image is processed through a self-attention mechanism to obtain self-attention features, and the self-attention features are input into a conditional control module for processing to obtain conditional control features.

[0103] Step S305: Using the conditional control features as conditional information, the noisy latent space feature map is denoised through the denoising network of the diffusion module, and the prediction noise is determined based on the denoised latent space feature map.

[0104] During the denoising process of the diffusion module, a vector randomly sampled from a Gaussian distribution is used as the initial inverse diffusion vector. For each diffusion step, the accumulated noise is calculated based on the guiding conditions and the inverse diffusion vector of the previous diffusion step, and the inverse diffusion vector of each diffusion step is calculated based on the accumulated noise.

[0105] In this embodiment, the diffusion duration is sampled to obtain the sampling time. Using the conditional control feature as the guiding condition, the denoising network of the diffusion module is used to denoise the noisy latent space feature map to obtain the denoised latent space feature map corresponding to each sampling time. Noise prediction is performed on the denoised latent space feature map to obtain the predicted noise corresponding to each sampling time.

[0106] Step S306: Determine the loss function of the preset diffusion model based on the predicted noise and the sampled noise, train the preset diffusion model until the output value of the loss function of the preset diffusion model is less than or equal to the preset threshold, stop training, and obtain the first diffusion model.

[0107] In this embodiment, the mean square error between the sampling noise added during the forward diffusion process of the extended model and the prediction noise during the denoising process is used as the loss function.

[0108] For example, a preset diffusion model acquires the sampling noise and predicted noise at the same sampling time, and calculates the mean square error between the sampling noise and predicted noise at the same sampling time. If the mean square error between the sampling noise and predicted noise at the same sampling time is less than or equal to a preset threshold, the preset diffusion model is determined to have converged, indicating that the preset diffusion model training is complete, and the trained preset diffusion model is used as the first diffusion model. When the output value of the loss function is greater than the preset threshold, it is determined that the preset diffusion model has not yet converged, and the parameters of the preset diffusion model need to be adjusted again. Another set of training data is input into the preset diffusion model to continue training the preset diffusion model and to determine whether the updated loss function after training is less than or equal to the preset threshold. That is, the preset diffusion model is trained based on the first training data until the output value of the loss function of the preset diffusion model is less than or equal to the preset threshold, then training stops, and the trained preset diffusion model is determined as the first diffusion model.

[0109] For example, gradient descent can be used to backflip the loss value into a preset diffusion model to obtain the gradient, and then update the model parameters accordingly.

[0110] In one embodiment of this application, the diffusion module gradually subtracts the prediction noise corresponding to each sampling time from the noisy latent space feature map according to the prediction noise corresponding to each sampling time to obtain the predicted latent space feature map. The decoder performs image decoding processing on the predicted latent space feature map to obtain the predicted high-definition face image.

[0111] In one embodiment of this application, a simulated low-resolution face image is processed using a self-attention mechanism to obtain self-attention features, which are then used as input to a conditional control module. The simulated low-resolution face image is processed through convolution and other methods to extract features, resulting in a first feature map. This first feature map is then subjected to a dimensionality reduction transformation using a multilayer perceptron to obtain a second feature map. The second feature map is then divided into patches to obtain multiple feature blocks. These feature blocks are converted into a token sequence required by the self-attention network through a linear embedding layer. The token sequence is input into the self-attention network, and after processing by the network, self-attention features are obtained. The self-attention features are then processed using the UnPatchify function to ensure that the dimensionality of the processed self-attention features is the same as that of the first feature map.

[0112] In one embodiment of this application, the denoising network in the diffusion module is a Unet network denoising network. It includes a denoising encoder, intermediate layers, and a denoising decoder. The denoising encoder includes multiple cascaded denoising coding layers, and the denoising decoder includes multiple cascaded denoising decoding layers. The conditional control module is a ControlNet network, which includes multiple cascaded conditional coding layers, intermediate layers, and multiple zero-convolutional layers. Each cascaded conditional coding layer corresponds one-to-one with a cascaded denoising decoding layer in the denoising decoder. During decoding, each denoising decoding layer decodes the vector input to that layer based on the conditional control features output by the corresponding conditional coding layer. Figure 4 The diagram shown is a structural schematic of a denoising network and a control network provided in an embodiment of this application.

[0113] like Figure 5 As shown, using conditional control features as conditional information, the denoising network of the diffusion module denoises the noisy latent space feature map. Based on the denoised latent space feature map, the prediction noise is determined, including:

[0114] Step S501: Input the noisy latent space feature map into the denoising network, and encode the noisy latent space feature map through the denoising encoder in the denoising network to obtain the intermediate encoding vector.

[0115] Step S502: Input the self-attention features into the control network, and encode the self-attention features sequentially through multiple cascaded conditional coding layers in the control network to obtain the conditional control features corresponding to each conditional coding layer.

[0116] Step S503: Input the intermediate encoded vector into the denoising decoder. The intermediate encoded vector is decoded sequentially by multiple cascaded denoising decoding layers in the denoising decoder. Each denoising decoding layer decodes the vector input to the denoising decoding layer based on the conditional control features of the corresponding conditional coding layer. The output of the last denoising decoding layer of the multiple cascaded denoising decoding layers is obtained to obtain the denoised latent space feature map.

[0117] Step S504: Determine the predicted noise based on the denoised latent space feature map.

[0118] In one embodiment of this application, the denoised latent space feature map is input to the decoder of a preset diffusion model, and the image is reconstructed by the decoder to obtain the output predicted high-definition face image.

[0119] In one embodiment of this application, a first pseudo-high-definition face image is generated based on a pre-trained first diffusion model, specifically:

[0120] Input random sampled noise into the first diffusion model;

[0121] The denoising network and conditional control module of the first diffusion model are used to denoise the random sampling noise using low-resolution face images as conditions, and a predicted latent space feature map is generated.

[0122] The first pseudo-high-definition face image is obtained by performing image decoding processing on the predicted latent space feature map through the decoder of the first diffusion model.

[0123] In one embodiment of this application, generating a first pseudo-high-definition face image using a pre-trained first diffusion model further includes:

[0124] Obtain multiple facial features corresponding to multiple low-resolution facial images, where the multiple low-resolution facial images have the same identity information;

[0125] The randomly sampled noise and low-resolution face image are input into the first diffusion model;

[0126] The first diffusion model uses a denoising network and a conditional control module to denoise random sampling noise based on the current facial features corresponding to the low-resolution face image. Noise prediction is then performed based on the consistency constraint of multi-image identity information. The consistency constraint of multi-image identity information is as follows: at the same sampling time, the similarity between the facial features of the i-th first pseudo-high-resolution face image output by the i-th low-resolution face image and the facial features of the j-th first pseudo-high-resolution face image output by the j-th low-resolution face image is obtained. The gradient is calculated based on this similarity and fed back to the prediction noise at the sampling time to obtain the prediction noise at the sampling time.

[0127] The predicted latent space feature map is obtained by progressively subtracting the prediction noise corresponding to each sampling time from the random sampling noise.

[0128] The decoder performs image decoding processing on the predicted latent space feature map to obtain the first pseudo-high-definition face image.

[0129] In this implementation, multiple low-resolution facial images share the same identity information. A pre-trained facial recognition model can be used to perform facial recognition on these multiple low-resolution facial images, obtaining the facial features corresponding to each low-resolution facial image.

[0130] In this implementation, for multiple low-resolution images of the same identity information, when generating the first pseudo-high-resolution face image, the consistency of identity information is constrained by the consistency constraint of multiple image identity information. During inference, the consistency of identity information of a series of images of the same person is constrained, so that the model trained by a single image can use multiple image information for inference without training with multiple image data, thereby improving the model's ability to maintain identity information.

[0131] In a specific embodiment of this application, multiple low-resolution facial images with the same identity information are used. k , k∈I, This represents the first diffusion model, which, at the same sampling time t, acquires the facial features of the i-th pseudo-high-definition face image output from the i-th low-resolution face image. The facial features of the j-th pseudo-high-definition face image output from the j-th low-definition face image similarity between The gradient is calculated using this similarity, and the predicted noise at that sampling time is fed back. i The prediction noise at that sampling time is obtained. i The formula for the consistency constraint of identity information across multiple images is expressed as follows:

[0132]

[0133] ∈ k =N(0,1);

[0134]

[0135] in, This represents the first diffusion model, where s is the scaling factor. Represents the gradient. This represents the prediction result output by the first diffusion model at time t-1. This represents the prediction result output by the first diffusion model at time t=0, ∈ k This represents random sampling noise that follows a normal distribution.

[0136] Multiple pairs of low-resolution face images and a first pseudo-high-resolution face image, as well as multiple pairs of low-resolution face images and high-resolution face images, are used to construct a second training data set. Based on this second training data, a first diffusion model is trained to update its parameters, resulting in a second diffusion model. In one embodiment of this application, as... Figure 6 As shown, the first diffusion model is trained based on the second training data to update the model parameters of the first diffusion model, resulting in a second diffusion model, including:

[0137] Step S601: Input the second training data into the first diffusion model, obtain the first prediction noise and the first predicted high-definition face image output by the first diffusion model based on the low-resolution face image and the first pseudo high-definition face image, and / or the low-resolution face image and the high-definition face image, input the first predicted high-definition face image into the pre-trained identity classifier, and obtain the identity prediction result corresponding to the first predicted high-definition face image, wherein the images in the second training data have identity labels;

[0138] Step S602: Determine the loss function of the first diffusion model based on the difference between random noise and first prediction noise in the forward diffusion process of the first diffusion model, and based on the difference between identity prediction results and identity labels.

[0139] Step S603: Iteratively adjust the model parameters of the first diffusion model using the loss function of the first diffusion model to obtain the trained second diffusion model.

[0140] like Figure 7 The schematic diagram of the first diffusion model and identity classifier shown illustrates the process of training the first diffusion model by inputting the second training data into it. A first pseudo-high-definition face image is input into the encoder of the diffusion model, and a low-definition face image is input into the attention network as conditional information for the diffusion model. Alternatively, a high-definition face image can be input into the encoder of the diffusion model, and a low-definition face image can be input into the attention network as conditional information for the diffusion model. The network structure of the first diffusion model is the same as that of the preset diffusion model. Image processing of the first diffusion model outputs first predicted noise and a first predicted high-definition face image. This specific implementation process is similar to that of the preset diffusion model and will not be elaborated further here.

[0141] In this embodiment, the identity classifier has been pre-trained. For example, the training process of the identity classifier is as follows: using high-resolution and low-resolution face images as training objects, the facial features of each training object are obtained. Each training object has a corresponding identity label, such as identity information. Each facial feature is input into the identity classifier to be trained, and the corresponding identity prediction result is output. Using the pre-trained identity classifier, the identity prediction result corresponding to the first predicted high-resolution face image is output.

[0142] Based on the difference between the identity prediction result and the identity label, the loss value of the identity information during the training of the second diffusion model can be determined. Therefore, based on the difference between the random noise and the first prediction noise in the forward diffusion process of the first diffusion model, and based on the difference between the identity prediction result and the identity label, the loss function of the first diffusion model is determined. The model parameters of the first diffusion model are adjusted through this loss function until the model training convergence condition is met, and the trained first diffusion model, i.e., the second diffusion model, is obtained.

[0143] In this embodiment, during model training, a large amount of image data with known identities but poor image quality is used to train the diffusion model, thereby significantly improving the ability of the diffusion model to preserve identity information in the generated images. A first pseudo-high-definition face image is generated by the first diffusion model. Image pairs consisting of low-definition face images and the first pseudo-high-definition face image, as well as image pairs consisting of low-definition face images and high-definition face images, are used as training data for the first diffusion model. During model training, the first predicted high-definition face image output by the first diffusion model is input into the identity classifier to obtain the identity prediction result. The loss function of the first diffusion model is constrained based on the difference between the identity prediction result and the identity label of the image, thereby adjusting the parameters of the first diffusion model. This breaks the model's requirement for image pairs and helps to improve the model's identity preservation ability.

[0144] In one embodiment of this application, such as Figure 8 As shown, the second diffusion model is trained based on the third training data to update the model parameters of the second diffusion model, resulting in a face image restoration model, including:

[0145] Step S801: Input the third training data into the second diffusion model, obtain the second prediction noise and the second predicted high-definition face image output by the second diffusion model based on the low-definition face image and the second pseudo high-definition face image, and / or the low-definition face image and the high-definition face image, and input the second predicted high-definition face image into the pre-trained image quality scorer and identity judge respectively, and obtain the image quality score and identity preservation result corresponding to the second predicted high-definition face image respectively.

[0146] Step S802: Based on the difference between random noise and second prediction noise in the forward diffusion process of the second diffusion model, the image quality score, and the identity preservation result, determine the loss function of the second diffusion model;

[0147] Step S803: Iteratively adjust the model parameters of the second diffusion model using the loss function of the second diffusion model to obtain the trained face image restoration model.

[0148] like Figure 9The schematic diagram of the second diffusion model, image quality scorer, and identity detector shown illustrates the following: The second training data is input into the first diffusion model for training. The second pseudo-high-definition face image is input into the encoder of the diffusion model, and the low-definition face image is input into the attention network as conditional information for the diffusion model. Alternatively, a high-definition face image can be input into the encoder of the diffusion model, and a low-definition face image can be input into the attention network as conditional information for the diffusion model. The network structure of the second diffusion model is the same as that of the preset diffusion model. Image processing of the second diffusion model outputs the second predicted noise and the second predicted high-definition face image. This specific implementation process is similar to that of the preset diffusion model and will not be described in detail here.

[0149] Both the image quality scorer and the identity detector are pre-trained. The second predicted high-resolution face image is input into the pre-trained image quality scorer and identity detector, respectively, to obtain the image quality score and identity preservation result corresponding to the second predicted high-resolution face image. Based on the image quality score, the quality loss of the generated image is determined, and based on the identity preservation result, the loss for preserving the identity information of the image is determined. Together with the noise loss in the diffusion model, the loss function of the second diffusion model is determined. The model parameters of the second diffusion model are adjusted through this loss function until the model training convergence condition is met, resulting in the trained second diffusion model, i.e., the face image restoration model.

[0150] In this embodiment, during model training, an image quality scorer and an identity determiner are introduced. While improving image quality, the identity information of the image is maintained. Therefore, images captured by low-quality cameras can also help determine the identity information of the image after model processing, thereby improving the monitoring capability of the old camera area. Introducing human knowledge to evaluate the consistency of image quality and identity information can prompt the model processing process to summarize, and output images that conform to human visual habits for data without high-definition matching ground truth values. It also restricts the identity information of the generated image, especially in extremely poor scenarios, guiding a reasonable generation direction and providing stronger supervision information.

[0151] In one embodiment of this application, the image quality scorer model training process involves inputting multiple low-resolution face images into a second diffusion model to obtain multiple second pseudo-high-resolution face images. Based on a semi-supervised method, the quality of the multiple pseudo-high-resolution face images is scored, obtaining quality score data corresponding to each pseudo-high-resolution face image. Based on the multiple quality score data and the multiple pseudo-high-resolution face images, a preset image quality scorer model is trained to obtain the trained image quality scorer. The image quality of the pseudo-high-resolution face images and the similarity between the identity information of the pseudo-high-resolution face images and the actual identity information are ranked, and a certain number of pseudo-high-resolution face images are collected in each segment to form training data. The quality of the pseudo-high-resolution face images is scored manually; for example, the scoring can be divided into three types: conforming to human habits, having a few anomalies, and having a large number of anomalies. The image quality scorer is trained based on the manually scored data. In this embodiment, the second diffusion model trained based on real low-resolution data optimizes the preservation of identity information of real degraded images, thereby improving the quality of the image data to be screened during the semi-supervised image quality scoring process.

[0152] In one embodiment of this application, the model training process of the identity determiner involves inputting multiple low-resolution face images into a second diffusion model to obtain multiple second pseudo-high-resolution face images. Based on a semi-supervised method, the similarity between the identity information of each second pseudo-high-resolution face image and the real identity information of the corresponding low-resolution face image is obtained, resulting in identity similarity data for each second pseudo-high-resolution face image. Based on the multiple identity similarity data and the multiple second pseudo-high-resolution face images, a preset identity determiner model is trained to obtain the trained identity determiner. The second pseudo-high-resolution face images are sorted based on their image quality and the similarity between their identity information and the actual identity information. A certain number of second pseudo-high-resolution face images are collected in each segment to form training data. The identity information preservation of the second pseudo-high-resolution face images and their corresponding original low-resolution images is manually scored. For example, the scoring can be divided into three types: strictly preserving identity information, basically preserving but with some changes, and identity information changing significantly. The identity determiner is trained based on the manually scored data.

[0153] In this embodiment of the application, the image generation anomalies can be quantitatively evaluated through a semi-supervised approach. Therefore, the quality of generated images can be constrained directly during model training, which can further improve the generation performance of the model.

[0154] Please refer to Figure 10 This application provides a method for face image restoration, the method comprising the following steps:

[0155] Step S1001: Input the target face image to be repaired and random sampling noise into the pre-trained face image repair model. The pre-trained face image repair model is trained based on the training method described above.

[0156] Step S1002: Based on the target face image to be repaired, iteratively denoise the randomly sampled noise to generate the repaired target face image.

[0157] The denoising network of the face image restoration model starts with randomly generated Gaussian noise that follows a normal distribution. Within a time step t, it first predicts the noise added at time T during the denoising process, thus obtaining the predicted noise at time t. Removing this predicted noise from the noise image features at time t yields the noise image features at time t-1. Similarly, the denoising network predicts the noise added at time t-1 based on the noise image features at time t-1, obtaining the predicted noise at time t-1. Removing this predicted noise from the noise image features at time t-1 yields the noise image features at time t-2. This process is repeated until the prediction result at time step t=0 is output, thus obtaining the predicted latent space features. The decoder then decodes these predicted latent space features to obtain the restored target face image.

[0158] In one embodiment of this application, when generating the target face image to be repaired, the multi-image identity information consistency constraint condition in the above-described manner can be adopted.

[0159] like Figure 11 The diagram shows a training device for a face image restoration model, the device comprising:

[0160] The first diffusion model training module 1101 is used to train a preset diffusion model based on the first training data to obtain the first diffusion model, wherein the first training data includes pairs of simulated low-resolution face images and high-resolution face images.

[0161] The second diffusion model training module 1102 is used to input multiple low-resolution face images into the first diffusion model to generate multiple first pseudo-high-resolution face images, construct multiple pairs of low-resolution face images and first pseudo-high-resolution face images, as well as multiple pairs of low-resolution face images and high-resolution face images as second training data, and train the first diffusion model based on the second training data to update the model parameters of the first diffusion model and obtain the second diffusion model.

[0162] The face image restoration model training module 1103 inputs multiple low-resolution face images into the second diffusion model to generate multiple second pseudo-high-resolution face images. It constructs multiple pairs of low-resolution face images and second pseudo-high-resolution face images, as well as multiple pairs of low-resolution face images and high-resolution face images, into third training data. Based on the third training data, it trains the second diffusion model to update the model parameters of the second diffusion model and obtains the face image restoration model.

[0163] In this embodiment, low-resolution facial images from real-world scenes are used to train the diffusion model, which reduces the difference between training data and real data and can effectively improve the similarity between the restored image and the real identity.

[0164] Optionally, the preset diffusion model includes an encoder, a diffusion module, a conditional control module, and a decoder. The diffusion module is a stable diffusion model, the conditional control module is a control network, and the first diffusion model training module 1101 is specifically used for:

[0165] Using an image degradation algorithm, a high-definition face image is degraded to obtain a corresponding simulated low-definition face image;

[0166] The high-definition face image is encoded by an encoder to obtain the latent space feature map corresponding to the high-definition face image;

[0167] Based on the sampling noise, the latent space feature map is noise-added through the forward diffusion process of the diffusion module to obtain a noisy latent space feature map.

[0168] The attention features are obtained by processing the simulated low-resolution face image based on the attention mechanism. The attention features are then input into the conditional control module for processing to obtain the conditional control features.

[0169] Using conditional control features as conditional information, the denoising network of the diffusion module is used to denoise the noisy latent space feature map, and the prediction noise is determined based on the denoised latent space feature map.

[0170] The loss function of the preset diffusion model is determined based on the predicted noise and the sampled noise. The preset diffusion model is trained until the output value of the loss function of the preset diffusion model is less than or equal to the preset threshold. Training is then stopped, and the first diffusion model is obtained.

[0171] Optionally, the encoder and decoder are paired encoders and decoders, and the model training for the encoder and decoder includes:

[0172] High-resolution facial images are used as training data; the high-resolution facial images are input into a preset encoder.

[0173] The pre-defined encoder maps high-resolution face images to the latent space, obtaining the corresponding latent space feature map;

[0174] The latent space feature map is denoised by adding and removing noise using a pre-defined diffusion model to obtain a denoised latent space feature map.

[0175] The preset decoder reconstructs the image from the denoised latent space feature map;

[0176] Based on the training data, the preset encoder and preset decoder are trained to obtain the trained encoder and decoder. During the training of the preset diffusion model, the first diffusion model and the second diffusion model, the network parameters of the trained encoder and decoder are frozen.

[0177] Optionally, the first diffusion model training module 1101 is specifically used for:

[0178] The first feature map is obtained by extracting features from the simulated low-resolution face image;

[0179] The first feature map is transformed using a multilayer perceptron to obtain the second feature map;

[0180] The second feature map is divided into multiple feature blocks, and these feature blocks are input into a linear embedding layer to obtain a token sequence.

[0181] Input the token sequence into the self-attention network to obtain self-attention features;

[0182] The self-attention features are processed using the UnPatchify function so that the dimension of the processed self-attention features is the same as the dimension of the first feature map.

[0183] Optionally, the denoising network of the diffusion module is a Unet network. The denoising network includes a denoising encoder, intermediate layers, and a denoising decoder. The denoising encoder includes multiple cascaded denoising coding layers, and the denoising decoder includes multiple cascaded denoising decoding layers. The control network includes multiple cascaded conditional coding layers, intermediate layers, and multiple zero convolutional layers. The multiple cascaded conditional coding layers correspond one-to-one with the multiple cascaded denoising decoding layers. Specifically, the first diffusion model training module 1101 is used for:

[0184] The noisy latent space feature map is input into the denoising network, and the noisy latent space feature map is encoded by the denoising encoder to obtain the intermediate encoding vector.

[0185] The self-attention features are input into the control network, and the self-attention features are encoded sequentially through multiple cascaded conditional coding layers in the control network to obtain the conditional control features corresponding to each conditional coding layer.

[0186] The intermediate encoded vector is input into the denoising decoder. The intermediate encoded vector is decoded sequentially by multiple cascaded denoising decoding layers in the denoising decoder. Each denoising decoding layer decodes the vector input to the denoising decoding layer based on the conditional control features of the corresponding conditional encoding layer. The output of the last denoising decoding layer of the multiple cascaded denoising decoding layers is obtained to obtain the denoised latent space feature map.

[0187] The predicted noise is determined based on the denoised latent space feature map.

[0188] Optionally, the second diffusion model training module 1102 is specifically used for:

[0189] The second training data is input into the first diffusion model to obtain the first prediction noise and the first predicted high-definition face image output by the first diffusion model based on the low-resolution face image and the first pseudo high-definition face image, and / or the low-resolution face image and the high-definition face image. The first predicted high-definition face image is input into the pre-trained identity classifier to obtain the identity prediction result corresponding to the first predicted high-definition face image. The images in the second training data have identity labels.

[0190] The loss function of the first diffusion model is determined based on the difference between random noise and first prediction noise in the forward diffusion process of the first diffusion model, and the difference between identity prediction results and identity labels.

[0191] The model parameters of the first diffusion model are iteratively adjusted using the loss function of the first diffusion model to obtain the trained second diffusion model.

[0192] Optionally, the face image restoration model training module 1103 is specifically used for:

[0193] The third training data is input into the second diffusion model to obtain the second prediction noise and the second predicted high-definition face image output by the second diffusion model based on the low-definition face image and the second pseudo high-definition face image, and / or the low-definition face image and the high-definition face image. The second predicted high-definition face image is input into the pre-trained image quality scorer and identity judge respectively to obtain the image quality score and identity preservation result corresponding to the second predicted high-definition face image respectively.

[0194] Based on the difference between random noise and second predicted noise in the forward diffusion process of the second diffusion model, the image quality score, and the identity preservation result, the loss function of the second diffusion model is determined.

[0195] The model parameters of the second diffusion model are iteratively adjusted using the loss function of the second diffusion model to obtain a trained face image restoration model.

[0196] Optional, model training for the image quality scorer includes:

[0197] Multiple low-resolution face images are input into the second diffusion model to obtain multiple second pseudo-high-resolution face images;

[0198] Based on a semi-supervised method, the quality of multiple second pseudo-high-definition face images is scored to obtain the quality score data corresponding to each second pseudo-high-definition face image;

[0199] Based on multiple quality score data and multiple second pseudo-high-definition face images, a model is trained on a preset image quality scorer to obtain the trained image quality scorer.

[0200] Optional, the model training for the identity determiner includes:

[0201] Multiple low-resolution face images are input into the second diffusion model to obtain multiple second pseudo-high-resolution face images;

[0202] Based on a semi-supervised method, the similarity between the identity information of each second pseudo-high-definition face image and the real identity information of the corresponding low-definition face image is obtained, thus obtaining the identity similarity data corresponding to each second pseudo-high-definition face image.

[0203] Based on multiple identity similarity data and multiple second pseudo-high-definition face images, a model is trained on a preset identity determiner to obtain the trained identity determiner.

[0204] like Figure 12 A face image restoration device is shown, the device comprising:

[0205] The input module 1201 is used to input the target face image to be repaired and random sampling noise into a pre-trained face image repair model, which is obtained based on the above-mentioned device;

[0206] The denoising module 1202 is used to iteratively denoise the randomly sampled noise based on the target face image to be repaired, and generate the repaired target face image.

[0207] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements any of the above-described face image restoration model training methods or face image restoration methods.

[0208] Figure 13 This is a schematic diagram of the hardware structure of a computer device provided in an embodiment of this application. Figure 13The computer device shown includes: a processor 1301, a communication interface 1302, a memory 1303, and a communication bus 1304. The processor 1301, communication interface 1302, and memory 1303 communicate with each other via the communication bus 1304. Figure 13 The connection method between the processor 1301, communication interface 1302, and memory 1303 shown is merely exemplary. In the implementation process, the processor 1301, communication interface 1302, and memory 1303 may also communicate with each other using other connection methods besides the communication bus 1304.

[0209] Memory 1303 can be used to store computer programs, which may include instructions and data to implement the steps of any of the above-mentioned face image restoration model training methods or face image restoration methods. In this embodiment, memory 1303 can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, optical memory, and registers. Memory 1303 may include a hard disk and / or RAM.

[0210] Processor 1301 can be a general-purpose processor, which can be a processor that performs specific steps and / or operations by reading and executing a computer program (e.g., a computer program) stored in memory (e.g., memory 1303). The general-purpose processor may use data stored in memory (e.g., memory 1303) during the execution of said steps and / or operations. The general-purpose processor can be, for example, but not limited to, a central processing unit (CPU). Furthermore, processor 1301 can also be a special-purpose processor, which can be a processor specifically designed to perform specific steps and / or operations. Special-purpose processors can be, for example, but not limited to, ASICs and FPGAs. Additionally, processor 1301 can also be a combination of multiple processors, such as a multi-core processor.

[0211] Communication interface 1302 may include input / output (I / O) interfaces, physical interfaces, and logical interfaces for interconnecting devices within the network device, as well as interfaces for interconnecting the network device with other devices (e.g., network devices). The communication network may be Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. Communication interface 1302 may be a module, circuit, transceiver, or any device capable of enabling communication.

[0212] In implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware in the processor 1301 or by instructions in software form. The method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory (RAM), read-only memory (ROM), programmable read-only memory (ROM), electrically erasable programmable memory (EPR), registers, or other mature storage media in the art. This storage medium is located in memory 1303. The processor 1301 reads the information in memory 1303 and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0213] Although preferred embodiments of the present application have been disclosed for illustrative purposes, those skilled in the art will recognize that various modifications, additions, and substitutions are possible without departing from the scope and spirit of the present application as disclosed in the appended claims.

Claims

1. A method for training a face image inpainting model, characterized in that, The method comprises: training a preset diffusion model based on first training data to obtain a first diffusion model, wherein the first training data comprises pairs of simulated low-definition face images and high-definition face images; inputting a plurality of low-definition face images into the first diffusion model to generate a plurality of first pseudo-high-definition face images, constructing pairs of low-definition face images and first pseudo-high-definition face images and pairs of low-definition face images and high-definition face images as second training data, training the first diffusion model based on the second training data to update model parameters of the first diffusion model, and obtaining a second diffusion model; inputting a plurality of low-definition face images into the second diffusion model to generate a plurality of second pseudo-high-definition face images, constructing pairs of low-definition face images and second pseudo-high-definition face images and pairs of low-definition face images and high-definition face images as third training data, training the second diffusion model based on the third training data to update model parameters of the second diffusion model, and obtaining a face image restoration model. 2.The method of claim 1, wherein, The preset diffusion model comprises an encoder, a diffusion module, a conditional control module, and a decoder, the diffusion module is a stable diffusion model, the conditional control module is a control network, the preset diffusion model is trained based on first training data to obtain a first diffusion model, which comprises: degrading the high-definition face image using an image degradation algorithm to obtain a corresponding simulated low-definition face image; encoding the high-definition face image through the encoder to obtain a hidden space feature map corresponding to the high-definition face image; performing noise addition on the hidden space feature map through a forward diffusion process of the diffusion module according to a sampling noise to obtain a noise-added hidden space feature map; performing attention mechanism processing on the simulated low-definition face image to obtain an attention feature, and inputting the attention feature into the conditional control module for processing to obtain a conditional control feature; performing denoising on the noise-added hidden space feature map through a denoising network of the diffusion module using the conditional control feature as conditional information, and determining a predicted noise according to the denoised hidden space feature map; determining a loss function of the preset diffusion model according to the predicted noise and the sampling noise, training the preset diffusion model until an output value of the loss function of the preset diffusion model is less than or equal to a preset threshold, stopping the training, and obtaining a first diffusion model. 3.The method of Claim 2, wherein, The encoder and the decoder are a paired codec, and the model training of the encoder and the decoder comprises: inputting the high-definition face image into a preset encoder as training data; the preset encoder maps the high-definition face image to a hidden space to obtain a corresponding hidden space feature map; performing noise addition and denoising on the hidden space feature map through a preset diffusion model to obtain a denoised hidden space feature map; a preset decoder reconstructs the denoised hidden space feature map into an image; Model training is performed on the preset encoder and the preset decoder based on the training data, to obtain a trained encoder and a trained decoder, wherein network parameters of the trained encoder and the trained decoder are frozen during training of the preset diffusion model, the first diffusion model, and the second diffusion model. 4.The method of Claim 2, wherein, The analog low-resolution face image is processed based on an attention mechanism to obtain attention features, and the attention features are input into the conditional control module for processing to obtain conditional control features. The analog low-resolution face image is subjected to feature extraction to obtain a first feature map. The first feature map is subjected to dimension transformation by using a multi-layer perception to obtain a second feature map. The second feature map is divided into a plurality of feature blocks, and the plurality of feature blocks are input into a linear embedding layer to obtain a token sequence. The token sequence is input into a self-attention network to obtain self-attention features. The self-attention features are processed based on an UnPatchify function, so that the dimensions of the processed self-attention features are the same as those of the first feature map. 5.The method of claim 2, wherein, The denoising network of the diffusion module is a Unet network, the denoising network includes a denoising encoder, an intermediate layer, and a denoising decoder, the denoising encoder includes a plurality of cascaded denoising encoding layers, the denoising decoder includes a plurality of cascaded denoising decoding layers, the control network includes a plurality of cascaded conditional encoding layers, an intermediate layer, and a plurality of zero convolution layers, the plurality of cascaded conditional encoding layers correspond one-to-one to the plurality of cascaded denoising decoding layers, wherein the denoising processing of the noisy latent space feature map by the denoising network of the diffusion module based on the conditional control features as conditional information, and the determination of the predicted noise based on the denoised latent space feature map, include: The noisy latent space feature map is input into the denoising network, and the noisy latent space feature map is encoded by the denoising encoder to obtain an intermediate encoding vector. The self-attention features are input into the control network, and the self-attention features are sequentially encoded by the plurality of cascaded conditional encoding layers in the control network to obtain the conditional control features corresponding to each conditional encoding layer. The intermediate encoding vector is input into the denoising decoder, and the intermediate encoding vector is sequentially decoded by the plurality of cascaded denoising decoding layers in the denoising decoder, each denoising decoding layer decodes the vector input into the denoising decoding layer based on the conditional control features of the corresponding conditional encoding layer, and the output of the last denoising decoding layer of the plurality of cascaded denoising decoding layers is obtained to obtain the denoised latent space feature map. The predicted noise is determined based on the denoised latent space feature map. 6.The method of claim 1, wherein, The first diffusion model is trained based on the second training data to update the model parameters of the first diffusion model to obtain a second diffusion model, including: inputting the second training data into the first diffusion model, obtaining first predicted noise and a first predicted high-definition face image output by the first diffusion model based on the low-definition face image and the first pseudo high-definition face image, and / or the low-definition face image and the high-definition face image, inputting the first predicted high-definition face image into a pre-trained identity classifier, and obtaining an identity prediction result corresponding to the first predicted high-definition face image, wherein the images in the second training data have identity labels; determining a loss function of the first diffusion model based on a difference between random noise in a forward diffusion process of the first diffusion model and the first predicted noise, and a difference between the identity prediction result and the identity label; iteratively adjusting model parameters of the first diffusion model based on the loss function of the first diffusion model, to obtain a trained second diffusion model. 7.The method of claim 1, wherein, The inputting of the plurality of low-definition face images into the first diffusion model to generate a plurality of first pseudo high-definition face images comprises: obtaining a plurality of face features corresponding to a plurality of low-definition face images, wherein the plurality of low-definition face images have the same identity information; inputting random sampling noise and a low-definition face image into the first diffusion model; performing denoising processing on the random sampling noise based on a current face feature corresponding to the low-definition face image as a condition by using a denoising network and a condition control module of the first diffusion model, and performing noise prediction based on a multi-image identity information consistency constraint condition, wherein the multi-image identity information consistency constraint condition is that, at the same sampling time, a similarity between a face feature of an i-th first pseudo high-definition face image output by an i-th low-definition face image and a face feature of a j-th first pseudo high-definition face image output by a j-th low-definition face image is calculated, and the similarity is used for gradient calculation and returned to predicted noise at the sampling time to obtain the predicted noise at the sampling time; gradually subtracting the predicted noise corresponding to each sampling time from the random sampling noise to obtain a predicted latent space feature map; performing image decoding processing on the predicted latent space feature map to obtain a first pseudo high-definition face image. 8.The method of Claim 1, wherein The training of the second diffusion model based on the third training data to update model parameters of the second diffusion model to obtain a face image restoration model comprises: inputting the third training data into the second diffusion model, obtaining second predicted noise and a second predicted high-definition face image output by the second diffusion model based on the low-definition face image and the second pseudo high-definition face image, and / or the low-definition face image and the high-definition face image, inputting the second predicted high-definition face image into a pre-trained image quality scorer and an identity determiner respectively, and obtaining an image quality score and an identity retention result corresponding to the second predicted high-definition face image respectively; determining a loss function of the second diffusion model based on a difference between random noise in a forward diffusion process of the second diffusion model and the second predicted noise, the image quality score, and the identity retention result; and iteratively adjusting model parameters of the second diffusion model based on the loss function of the second diffusion model, to obtain a trained second diffusion model. The model parameters of the second diffusion model are iteratively adjusted based on a loss function of the second diffusion model, to obtain a trained face image inpainting model. 9.The method of Claim 8, wherein, The model training of the image quality scorer includes: The low-definition face images are input into the second diffusion model to obtain a plurality of second pseudo-high-definition face images; Based on a semi-supervised method, the quality of the plurality of second pseudo-high-definition face images is scored to obtain quality score data corresponding to each second pseudo-high-definition face image; Based on the plurality of quality score data and the plurality of second pseudo-high-definition face images, a preset image quality scorer is trained to obtain a trained image quality scorer. And / or, The model training of the identity determiner includes: The low-definition face images are input into the second diffusion model to obtain a plurality of second pseudo-high-definition face images; Based on a semi-supervised method, the similarity between the identity information of each second pseudo-high-definition face image and the real identity information of the corresponding low-definition face image is obtained to obtain identity similarity data corresponding to each second pseudo-high-definition face image; Based on the plurality of identity similarity data and the plurality of second pseudo-high-definition face images, a preset identity determiner is trained to obtain a trained identity determiner.

10. A face image inpainting method, characterized by, The method includes: A target face image to be inpainted and random sampling noise are input into a pre-trained face image inpainting model, and the pre-trained face image inpainting model is trained based on the method of any one of claims 1-6; Based on the target face image to be inpainted, the random sampling noise is iteratively denoised to generate an inpainted target face image.

11. A computer device, comprising: It includes: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus; The memory is used to store a computer program; The processor is used to execute the computer program stored on the memory to realize the steps of the face image inpainting model training method according to any one of claims 1-9, or realize the steps of the face image inpainting method according to any one of claim 10.

Images