Image inpainting model training methods, image inpainting methods, devices and electronic equipment

Abstract

This application provides an image restoration model training method, image restoration method, apparatus, and electronic device, belonging to the field of computer technology. The method includes: training a pattern generative adversarial network (PGA) based on a first training image set, and combining the PGA, a feature extraction network, and a discriminator to form an initial image restoration model; obtaining multiple target face images based on a second training image set, and inputting each target face image into the feature extraction network for feature extraction to obtain feature vectors; the PGA outputs predicted face images based on the feature vectors and noise information output from the coding layer; determining loss information based on each predicted face image and the second training image set, and iteratively optimizing the initial image restoration model based on the loss information until a preset condition is met; using the initial image restoration model that meets the preset condition as the target image restoration model, and stopping training. This application can improve the accuracy and practicality of face restoration.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to an image restoration model training method, an image restoration method, an apparatus, and an electronic device. Background Technology

[0002] With the development of artificial intelligence technology, people have begun to use neural network technology to restore images, such as old photos or blurry facial images.

[0003] In related technologies, to restore facial images, relatively clear areas in the image to be restored can be used to fill in and repair damaged or blurred areas. For example, a sample block-based restoration algorithm can be used to determine the priority of each region to be restored based on the damaged or blurred areas to be restored and the surrounding clear areas, and then restore each region to be restored in descending order of priority. Specifically, a matching criterion can be established based on the minimum absolute difference squared, and the matching block that best matches each region to be restored can be found from a large number of possible sample blocks and filled into each region to be restored, finally resulting in a restored facial image.

[0004] However, because related technologies require repairing blurry or missing areas based on the clearer regions in the image to be repaired, if the facial information in the image to be repaired is severely lacking or the image is uneven, the repaired face image will differ significantly from the original image. Therefore, this approach suffers from low accuracy and poor practicality in face restoration. Summary of the Invention

[0005] The purpose of this application is to provide an image restoration model training method, image restoration method, device, and electronic device that can improve the accuracy and practicality of face restoration.

[0006] The embodiments of this application are implemented as follows:

[0007] A first aspect of this application provides a method for training an image restoration model, the method comprising:

[0008] A pattern generative adversarial network is trained based on the first training image set, and the pattern generative adversarial network, the feature extraction network, and the discriminator are combined to form an initial image inpainting model. The feature extraction network includes multiple coding layers and fully connected layers connected in sequence.

[0009] Multiple target face images are obtained based on the second training image set, and each target face image is input into the feature extraction network for feature extraction to obtain a feature vector;

[0010] The pattern-generating adversarial network outputs at least one predicted face image based on the feature vector and noise information output by at least one coding layer in the feature extraction network;

[0011] Loss information is determined based on the predicted face images and the second training image set. The initial image restoration model is iteratively optimized based on the loss information until a preset condition is met. The initial image restoration model that meets the preset condition is taken as the target image restoration model, and training is stopped.

[0012] Optionally, the step of obtaining multiple target face images based on the second training image set, and inputting each of the target face images into the feature extraction network for feature extraction to obtain a feature vector, includes:

[0013] The initial face images in the second training image set are degraded to obtain each target face image, and the quality of each target face image is lower than that of each initial face image.

[0014] Each of the target face images is input into the first encoding layer of the feature extraction network, and then encoded sequentially by each encoding layer of the feature extraction network to obtain a feature map;

[0015] The feature map is dimensionally adjusted by the fully connected layer in the feature extraction network to obtain the feature vector.

[0016] Optionally, the step of degrading the initial face images in the second training image set to obtain each of the target face images includes:

[0017] The initial face image is filtered using a bilateral filtering algorithm, and then blurred using a convolution kernel.

[0018] The blurred features are downsampled and then subjected to Gaussian noise processing.

[0019] The features processed by Gaussian noise are compressed and upsampled to obtain the target face image.

[0020] Optionally, the style generation adversarial network includes multiple style generation modules, which are connected sequentially.

[0021] The generative adversarial network (GAN) outputs at least one predicted face image based on the feature vector and the noise information output by each coding layer in the feature extraction network, including:

[0022] The feature vector is mapped to latent space features through the mapping network in the pattern generative adversarial network.

[0023] The latent space features are input into each of the pattern generation modules, and the noise information output by at least one coding layer is input into each of the pattern generation modules according to the arrangement order of each coding layer in the initial image inpainting model, so as to obtain each of the predicted face images.

[0024] Optionally, the step of inputting noise information output from at least one coding layer into each of the style generation modules according to the arrangement order of the coding layers in the initial image inpainting model to obtain each of the predicted face images includes:

[0025] Determine the arrangement order of each style generation module;

[0026] If the arrangement order of the pattern generation modules is 1, then the latent space features, the preset constant terms, and random noise are input into the pattern generation module, and the pattern generation module outputs the decoding information.

[0027] If the arrangement order of the style generation modules is not 1 and is less than a preset value, then the latent space features, the decoding information output by the previous style generation module, and random noise are input into the style generation module, and the style generation module outputs the decoding information.

[0028] If the arrangement order of the style generation modules is equal to or greater than a preset value, then the latent space features, the feature map output by the encoding layer corresponding to the style generation module preceding the style generation module, and the information output by the style generation module preceding the style generation module are input into the style generation module, and the style generation module outputs the decoding information.

[0029] The decoded information output by the last style generation module is used as the predicted face image.

[0030] Optionally, determining the loss information based on each of the predicted face images and the second training image set includes:

[0031] Determine the initial face image corresponding to each of the target face images in the second training image set;

[0032] Each of the target face images, each of the initial face images, and each of the predicted face images are concatenated into multiple image pairs, and each of the image pairs is input into the discriminator;

[0033] The discriminator determines the loss information based on each of the image pairs.

[0034] Optionally, the step of determining the loss information by the discriminator based on each of the image pairs includes:

[0035] The discriminator calculates and generates adversarial loss values ​​based on each of the target face images and each of the predicted face images;

[0036] Calculate the absolute value loss (SmoothL1 Loss), additional corner edge loss (Arcface Loss), and gradient difference loss based on each of the initial face images and each of the predicted face images;

[0037] The loss information is obtained by weighted summation of the generative adversarial loss value, the absolute value loss value, the additional corner edge loss value, and the gradient difference loss value.

[0038] A second aspect of this application provides an image restoration method, the method comprising:

[0039] The image to be repaired is obtained and input into the image repair model. The image repair model is any target image repair model trained by the first aspect mentioned above. The image repair model includes a feature extraction network and a style generation adversarial network.

[0040] The feature extraction network extracts features from the image to be repaired to obtain a target feature vector;

[0041] The pattern generative adversarial network outputs the repaired image of the image to be repaired based on the target feature vector and the noise information output by each coding layer in the feature extraction network.

[0042] A third aspect of this application provides an image restoration model training apparatus, the apparatus comprising:

[0043] The component module is used to train a style generative adversarial network based on a first training image set, and to combine the style generative adversarial network, a feature extraction network, and a discriminator to form an initial image inpainting model. The feature extraction network includes multiple coding layers and fully connected layers connected in sequence.

[0044] The first extraction module is used to obtain multiple target face images based on the second training image set, and input each of the target face images into the feature extraction network for feature extraction to obtain a feature vector;

[0045] The first output module is used to output at least one predicted face image from the style generative adversarial network based on the feature vector and noise information output by at least one coding layer in the feature extraction network.

[0046] The optimization module is used to determine loss information based on each of the predicted face images and the second training image set, and to iteratively optimize the initial image restoration model based on the loss information until a preset condition is met. The initial image restoration model that meets the preset condition is then used as the target image restoration model, and training is stopped.

[0047] A fourth aspect of this application provides an image restoration apparatus, the apparatus comprising:

[0048] An input acquisition module is used to acquire the image to be repaired and input the image to be repaired into an image repair model, wherein the image repair model includes a feature extraction network and a style generation adversarial network;

[0049] The second extraction module is used to extract features from the image to be repaired by the feature extraction network to obtain a feature vector;

[0050] The second output module is used to output the repaired image of the image to be repaired by the style generative adversarial network based on the feature vector and the noise information output by each coding layer in the feature extraction network.

[0051] A fifth aspect of this application provides an electronic device, the electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the image restoration model training method of the first aspect and the image restoration method of the second aspect described above.

[0052] A sixth aspect of the embodiments of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image restoration model training method of the first aspect and the image restoration method of the second aspect described above.

[0053] The beneficial effects of the embodiments of this application include:

[0054] This application provides an image restoration model training method, wherein a pattern generative adversarial network is trained based on a first training image set, and the pattern generative adversarial network, a feature extraction network, and a discriminator are combined to form an initial image restoration model, which can obtain an initial image restoration model that can restore an input face image and output a restored face image.

[0055] Multiple target face images are obtained based on the second training image set, and each target face image is input into the feature extraction network for feature extraction to obtain a feature vector. In this way, each target face image can be converted into a feature vector, ensuring that after the feature extraction network inputs the feature vector into the pattern generative adversarial network, the pattern generative adversarial network can recognize and parse the feature vector.

[0056] The pattern-based generative adversarial network (GAN) outputs at least one predicted face image based on the feature vector and noise information from at least one coding layer in the feature extraction network. After inputting the feature vector and noise information into the GAN, it can generate a high-quality restored face image. This facilitates subsequent training of the initial image restoration model based on the predicted face image.

[0057] Loss information is determined based on each predicted face image and the second training image set. The initial image inpainting model is iteratively optimized based on the loss information until a preset condition is met. The initial image inpainting model that meets the preset condition is taken as the target image inpainting model, and training is stopped.

[0058] Since the loss information represents the difference between the predicted face image output by the initial image restoration model and the original image or target face image in the second training image set, iteratively correcting the initial image restoration model using this loss information can effectively reduce the error of the output of the initial image restoration model, thereby improving the accuracy of the image restoration model.

[0059] In other words, by training an initial image inpainting model based on a generative adversarial network, a target image inpainting model capable of accurately generating high-resolution face images can be obtained. This improves the accuracy and practicality of face inpainting. Attached Figure Description

[0060] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 A flowchart illustrating the first image restoration model training method provided in this application embodiment;

[0062] Figure 2 A flowchart illustrating the second image restoration model training method provided in this application embodiment;

[0063] Figure 3 A flowchart illustrating the third image restoration model training method provided in this application embodiment;

[0064] Figure 4 A flowchart illustrating the fourth image restoration model training method provided in this application embodiment;

[0065] Figure 5 A flowchart illustrating the fifth image restoration model training method provided in this application embodiment;

[0066] Figure 6 A flowchart illustrating the sixth image restoration model training method provided in this application embodiment;

[0067] Figure 7 A flowchart illustrating an image restoration method provided in this application embodiment;

[0068] Figure 8 This is a schematic diagram of the structure of an image restoration model provided in an embodiment of this application;

[0069] Figure 9 This is a schematic diagram of the structure of an image restoration model training device provided in an embodiment of this application;

[0070] Figure 10 This is a schematic diagram of the structure of an image restoration device provided in an embodiment of this application;

[0071] Figure 11 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0073] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0074] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0075] In the description of this application, it should be noted that the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0076] In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "set up," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0077] In related technologies, to restore facial images, relatively clear areas in the image to be restored can be used to fill in and repair damaged or blurred areas. For example, a sample block-based restoration algorithm can be used to determine the priority of each region to be restored based on the damaged or blurred areas to be restored and the surrounding clear areas, and then restore each region in descending order of priority. Specifically, a matching criterion can be established based on the minimum absolute difference squared, and the matching block that best matches each region to be restored can be found from a large number of possible sample blocks and filled into each region to be restored, finally resulting in a restored facial image.

[0078] However, because related technologies require repairing blurry or missing areas based on the clearer regions in the image to be repaired, if the facial information in the image to be repaired is severely lacking or the image is uneven, the repaired face image will differ significantly from the original image. Therefore, this approach suffers from low accuracy and poor practicality in face restoration.

[0079] To address this, this application provides an image inpainting model training method. A pattern generative adversarial network (GAN) is trained based on a first training image set. This GAN, along with a feature extraction network and a discriminator, forms an initial image inpainting model. Multiple target face images are obtained from a second training image set. Each target face image is input into the feature extraction network for feature extraction, yielding a feature vector. The GAN outputs at least one predicted face image based on the feature vector and noise information from at least one encoding layer in the feature extraction network. Loss information is determined based on each predicted face image and the second training image set. The initial image inpainting model is iteratively optimized based on this loss information until a preset condition is met. The initial image inpainting model that meets the preset condition is then used as the target image inpainting model, and training stops. This method can improve the accuracy and practicality of face inpainting.

[0080] This application uses an image restoration model training method applied in electronic devices as an example for illustration. However, it does not imply that this application's embodiments can only be applied to image restoration model training in electronic devices.

[0081] The image restoration model training method provided in the embodiments of this application will be explained in detail below.

[0082] Figure 1 A flowchart illustrating an image restoration model training method provided in this application is shown. This method can be applied to a computer device, which can be any terminal device or server. See also... Figure 1 This application provides an image restoration model training method, including:

[0083] Step 1001: Train a style generative adversarial network based on the first training image set, and combine the style generative adversarial network, feature extraction network and discriminator to form an initial image inpainting model.

[0084] Optionally, the first training image set may include multiple high-resolution face images. Each high-resolution face image may be a red-green-blue (RGB) image, and each high-resolution face image may have high quality, such as high contrast and signal-to-noise ratio, and low distortion.

[0085] Furthermore, this style of generative adversarial network can randomly generate high-resolution face images based on random vectors sampled from a Gaussian distribution and random noise. The random vector can be used to represent facial feature information in each high-resolution face image, and the random noise is used to represent facial details in each high-resolution face image.

[0086] During the training of the pattern generative adversarial network, the random vector can be concatenated with various random noises, or the latent space features corresponding to the random vector can be concatenated with various random noises before being input into the pattern generative adversarial network. This application does not limit this approach.

[0087] In addition, this style generation adversarial network can include multiple style generation modules, which are connected in sequence.

[0088] For example, after the corresponding random vector is input into the pattern generative adversarial network, each pattern generation module can decode or parse each vector to output the corresponding decoding information, and the decoding information output by the last pattern generation module is an image.

[0089] Furthermore, any possible training method can be used to train the generative adversarial network of this style, and the embodiments of this application do not limit this.

[0090] Optionally, the feature extraction network includes multiple sequentially connected coding layers and fully connected layers. This feature extraction network can extract features from an input face image and output a feature vector corresponding to the input face image.

[0091] In addition, the last coding layer in each coding layer can be connected to the fully connected layer, which can be connected to the pattern-generating adversarial network.

[0092] Each coding layer can extract features from the image or information input to the feature extraction network to output a corresponding tensor or vector. Each coding layer, except for the last one, can output a feature map to the next coding layer. The last coding layer can output a feature map to the fully connected layer.

[0093] For example, each coding layer can be viewed as a single encoder.

[0094] This fully connected layer can map the received feature maps to ensure that the generative adversarial network can accurately identify, parse, and process the information output by the feature extraction network.

[0095] The discriminator can determine the performance of the initial image inpainting model by judging the authenticity of the output of the style generative adversarial network, and this application embodiment does not limit this.

[0096] Optionally, the initial image inpainting model can be a network model for inpainting images. Optionally, the initial image inpainting model can refer to an image inpainting model that has not yet been trained or has not yet been trained.

[0097] Generally, the input to the initial image restoration model can be any low-quality, low-resolution face image, while the output of the initial image restoration model can be a high-quality face image obtained after restoration of the input face image. This application embodiment does not limit this.

[0098] It is worth noting that since this style generative adversarial network can generate high-quality, high-definition face images based on random vectors and random noise, and this feature extraction network can output the feature vector of the face image, it can be ensured that the initial image inpainting model can inpaint the input face image given an input face image, and then output an inpainted face image, so that the initial image inpainting model can be trained subsequently.

[0099] Step 1002: Obtain multiple target face images based on the second training image set, and input each target face image into the feature extraction network for feature extraction to obtain feature vectors.

[0100] Optionally, the second training image set may be the same as or different from the first training image set, and this application embodiment does not limit this.

[0101] For example, the second training image set may include only multiple high-definition face images, or it may include multiple high-definition face images and low-definition face images corresponding to each high-definition face image, or it may include multiple image pairs of high-definition face images and low-definition face images. This application embodiment does not limit this.

[0102] That is, if the second training image set includes multiple high-definition face images, then these multiple high-definition face images can be regarded as the original images of each target face image.

[0103] Optionally, each target face image may refer to a low-resolution face image of lower quality, and this application embodiment does not limit this.

[0104] Alternatively, each target face image can be sequentially input into the feature extraction network for feature extraction to obtain multiple feature vectors. That is, each feature vector can be a vector corresponding to a target face image, and the feature vector is used to represent the feature information and overall content of the face in the target face image.

[0105] The dimension of the feature vector can be (1, 512) or any other possible dimension, which is not limited in this application embodiment.

[0106] It's worth noting that since the target face images are low-resolution images of low quality, each feature vector is a vector used to represent the feature information and overall content of the face in the low-resolution image. This allows each target face image to be converted into a feature vector, ensuring that after the feature extraction network inputs this feature vector into the style generative adversarial network (GAN), the GAN can recognize and parse the feature vector, thus ensuring the practicality of the image restoration model training method.

[0107] Step 1003: The pattern generative adversarial network outputs at least one predicted face image based on the feature vector and the noise information output by at least one coding layer in the feature extraction network.

[0108] Optionally, the noise information may refer to the feature map output when encoding each target face image in the coding layer, and the noise information may be used to supplement the facial details in each target face image.

[0109] In addition, the noise information output by at least one coding layer in the feature extraction network can refer to the feature map output by a portion of the coding layers in the feature extraction network.

[0110] Optionally, the predicted face image may refer to the restored face image obtained by restoring the target face image through the initial image restoration model or the style generative adversarial network in the initial image restoration model.

[0111] It is worth noting that this style-based generative adversarial network (GAN) can generate high-resolution face images based on random vectors sampled from a Gaussian distribution and random noise. Therefore, after inputting the feature vector and noise information into this style-based GAN, it can generate a high-quality, restored face image. This facilitates subsequent training of the initial image restoration model based on the predicted face image.

[0112] Step 1004: Determine the loss information based on each predicted face image and the second training image set, and iteratively optimize the initial image inpainting model based on the loss information until a preset condition is met. Take the initial image inpainting model that meets the preset condition as the target image inpainting model and stop training.

[0113] Optionally, the loss information can be used to characterize the performance of the initial image inpainting model, or it can be used to characterize the difference between each predicted face image and the original image in the second training image set, or the difference between each predicted face image and the target face image obtained based on the second training image set.

[0114] Optionally, the preset condition may refer to the number of iterations reaching a certain threshold, or the loss information being less than or equal to a certain preset loss value. This application embodiment does not limit this.

[0115] Optionally, the target image restoration model may refer to the initial image restoration model that has been trained, or it may refer to the initial image restoration model that performs well when restoring face images with low quality or missing parts. This application embodiment does not limit this.

[0116] It is worth noting that since the initial image inpainting model in steps 1001-1003 has not yet been trained or optimized, the predicted face image output by the initial image inpainting model may have problems such as low quality, blurriness, or significant differences from the original image. Therefore, it is necessary to determine the performance of the initial image inpainting model by determining this loss information, and then adjust and optimize the parameters of the initial image inpainting model accordingly, so as to continuously improve the performance of the initial image inpainting model in repairing each target face image during the iterative optimization process.

[0117] It is worth noting that since this loss information represents the difference between the predicted face image output by the initial image restoration model and the original image or target face image in the second training image set, iteratively correcting the initial image restoration model using this loss information can effectively reduce the error of the initial image restoration model output, thereby improving the accuracy of the initial image restoration model.

[0118] This improves the reliability and accuracy of training the initial image restoration model, thereby enhancing the accuracy of the target image restoration model in restoring high-resolution face images from low-resolution input images.

[0119] In this embodiment, a pattern generative adversarial network (GAN) is trained based on a first training image set, and this GAN, along with a feature extraction network and a discriminator, forms an initial image inpainting model. Multiple target face images are obtained based on a second training image set, and each target face image is input into the feature extraction network for feature extraction to obtain a feature vector. The GAN outputs at least one predicted face image based on the feature vector and noise information from at least one encoding layer in the feature extraction network. Loss information is determined based on each predicted face image and the second training image set. The initial image inpainting model is iteratively optimized based on this loss information until a preset condition is met. The initial image inpainting model that meets the preset condition is then used as the target image inpainting model, and training stops.

[0120] Specifically, a pattern generative adversarial network is trained based on the first training image set, and this pattern generative adversarial network, along with a feature extraction network and a discriminator, are combined to form an initial image inpainting model. This initial image inpainting model can inpaint an input face image and output the inpainted face image.

[0121] Multiple target face images are obtained based on the second training image set, and each target face image is input into the feature extraction network for feature extraction to obtain a feature vector. In this way, each target face image can be converted into a feature vector, ensuring that after the feature extraction network inputs the feature vector into the pattern generative adversarial network, the pattern generative adversarial network can recognize and parse the feature vector.

[0122] The pattern-based generative adversarial network (GAN) outputs at least one predicted face image based on the feature vector and noise information from at least one coding layer in the feature extraction network. After inputting the feature vector and noise information into the GAN, it can generate a high-quality restored face image. This facilitates subsequent training of the initial image restoration model based on the predicted face image.

[0123] Loss information is determined based on each predicted face image and the second training image set. The initial image inpainting model is iteratively optimized based on the loss information until a preset condition is met. The initial image inpainting model that meets the preset condition is taken as the target image inpainting model, and training is stopped.

[0124] Since the loss information represents the difference between the predicted face image output by the initial image restoration model and the original image or target face image in the second training image set, iteratively correcting the initial image restoration model using this loss information can effectively reduce the error of the output of the initial image restoration model, thereby improving the accuracy of the image restoration model.

[0125] In other words, by training an initial image inpainting model based on a generative adversarial network, a target image inpainting model capable of accurately generating high-resolution face images can be obtained. This improves the accuracy and practicality of face inpainting.

[0126] In one possible implementation, see [link to relevant documentation]. Figure 2 Multiple target face images are obtained based on the second training image set, and each target face image is input into the feature extraction network for feature extraction to obtain a feature vector, including:

[0127] Step 1005: Degrade the initial face images in the second training image set to obtain target face images.

[0128] Optionally, in this application, degradation processing can refer to operations that reduce image quality. For example, degradation processing may include operations such as reducing image quality, increasing image blur, shrinking the image, or compressing the image. This application does not limit this.

[0129] Optionally, each initial face image can be one of the multiple high-resolution face images in the second training image set.

[0130] Optionally, the quality of each target face image is lower than the quality of each initial face image.

[0131] For example, image quality can generally be evaluated using parameters such as signal-to-noise ratio (SNR), contrast ratio, distortion level, and blur level. Generally, higher image quality means higher SNR and contrast ratio, and lower distortion and blur level. Conversely, lower image quality means lower SNR and contrast ratio, and higher distortion and blur level.

[0132] In this way, we can obtain low-quality, low-resolution images of each target face, which facilitates the subsequent extraction of feature vectors from each target face image for training the initial image restoration model.

[0133] Step 1006: Input each target face image into the first encoding layer of the feature extraction network, and encode it sequentially by each encoding layer of the feature extraction network to obtain a feature map.

[0134] Optionally, each coding layer may include a 3×3 convolutional layer and a non-linear activation layer.

[0135] For example, each coding layer can be regarded as an encoder E, and the fully connected layer can be represented by F. Then the feature vector output by the fully connected layer can be represented by z. lq =F(E(I) lq ))express.

[0136] Among them, I lq This refers to the target human face image.

[0137] Since the coding layers are connected sequentially, the first coding layer can refer to the first coding layer in the sequence.

[0138] Optionally, encoding by each coding layer in the feature extraction network in sequence can mean that each coding layer encodes the received feature map or information and outputs the encoded feature map or information to the next coding layer, until the last coding layer outputs the encoded feature map to the fully connected layer.

[0139] For example, after a target face image is input into the first encoding layer, the first encoding layer can output a tensor with dimensions (1, 512, 4, 4). After each encoding layer encodes in sequence, the last encoding layer can output a vector of dimensions (1, 512 * 4 * 4) = (1, 8192). That is, the feature map can be a vector of dimensions (1, 8192).

[0140] In this way, the encoding of each target face image can be completed, and the encoded feature map can be output to the fully connected layer.

[0141] Step 1007: The feature map is adjusted in dimension by the fully connected layer in the feature extraction network to obtain the feature vector.

[0142] Optionally, this dimensionality adjustment can refer to adjusting the dimension of the feature maps output by each encoding layer after encoding each target face image to the dimension required by the input generative adversarial network described above. Specifically, it can refer to a dimensionality reduction operation.

[0143] For example, if the feature map obtained in step 1006 is a (1, 8192) dimensional vector, after dimension adjustment, a (1, 512) dimensional feature vector can be obtained.

[0144] In this way, each target face image can be converted into a feature vector that represents the feature information and overall content of the face in the low-resolution face image. This ensures that after the feature extraction network inputs the feature vector into the style generative adversarial network, the style generative adversarial network can recognize and parse the feature vector, thereby ensuring the practicality of the image restoration model training method.

[0145] In one possible implementation, see [link to relevant documentation]. Figure 3 The initial face images in the second training image set are degraded to obtain target face images, including:

[0146] Step 1008: Filter the initial face image based on the bilateral filtering algorithm, and blur the initial face image using a convolution kernel.

[0147] Optionally, the bilateral filtering algorithm can be implemented based on a bilateral filter, and filtering the initial face image based on the bilateral filtering algorithm can simulate a skin smoothing effect.

[0148] Alternatively, the convolution kernel can be a convolution kernel used for blurring images.

[0149] Blur processing operations may include Gaussian blur and / or motion blur, but this application embodiment does not limit this.

[0150] This can improve the simulation effect of facial details in each target face image.

[0151] Step 1009: Perform downsampling on the blurred features and apply Gaussian noise.

[0152] Optionally, downsampling the blurred initial face image can refer to reducing the size of the initial face image. The scaling factor for reducing the initial face image can be set according to actual needs.

[0153] Gaussian noise processing can refer to adding Gaussian noise to the initial face image.

[0154] Step 1010: Compress and upsample the features after Gaussian noise processing to obtain the target face image.

[0155] Optionally, compression processing can refer to still image compression (Joint Photographic Experts Group, or JPEG for short). The compression ratio can be (100-q)%, where q can be any positive real number, and can be adjusted according to actual needs.

[0156] Optionally, the features after compression can be upsampled. Upsampling can be considered as amplifying the features after compression. Generally, the result of upsampling the features after compression is the target face image, but this embodiment does not limit this.

[0157] The magnification factor of the compressed features can be set according to actual needs. Generally, the product of the magnification factor and the reduction factor of the initial face image is 1. This ensures that the resulting target face images are the same size as the initial face images.

[0158] It should be noted that this degradation process can include: filtering based on a bilateral filtering algorithm, blurring, downsampling, and compression. Generally, upsampling can be considered as an operation performed after this degradation process to restore the image to its original size before downsampling.

[0159] For example, the initial face image can be degraded using the following formula:

[0160]

[0161] Among them, I hq Here is the initial face image as input, and k represents the convolution kernel used for blurring. ↓ s n represents the sampling scale. σ This indicates that Gaussian noise with variance σ is added to the initial face image. (JPEG) q This indicates that JPEG image compression with a compression rate of (100-q)% is applied to the original image, I lq BF represents the bilateral filter, which is the target face image corresponding to the initial face image.

[0162] Specifically, the filtering process of the bilateral filtering algorithm can be implemented using the following formula:

[0163]

[0164]

[0165] Where w(i,j,x,y) represents the bilateral filter weight value at point (i,j) in the spatial domain S, where point (i,j) can be any point in the spatial domain S, and point (x,y) is the coordinate of the center point of the spatial domain S. hq (i,j) represents the pixel value of the initial face image at point (i,j).

[0166] in addition, The variance of the Gaussian distribution in the spatial domain S is represented by . This represents the range variance.

[0167] This allows for the degradation of the initial face image. Furthermore, after degradation, appropriate upsampling can be used to ensure that each target face image is the same size as the initial face image, so that it can be subsequently input into the feature extraction network for processing.

[0168] In one possible implementation, see [link to relevant documentation]. Figure 4 The generative adversarial network (GAN) based on this pattern outputs at least one predicted face image according to the feature vector and the noise information output by each coding layer in the feature extraction network, including:

[0169] Step 1011: The feature vector is mapped to latent space features through the mapping network in the Generative Adversarial Network (GAN) of this style.

[0170] Optionally, the mapping network can be connected to the fully connected layer mentioned above, and can also be connected to the style generation modules in the style generation adversarial network.

[0171] The input to the mapping network is the feature vector output by the fully connected layer, and the output of the mapping network is the latent space feature.

[0172] Optionally, the latent space feature can generally be a tensor or vector of arbitrary dimensions, typically a 4-dimensional tensor. Furthermore, the dimensions of the aforementioned random noise can generally be the same as the dimensions of the latent space feature.

[0173] During the training of the pattern-generating adversarial network, the latent space features corresponding to the random vector can be concatenated with each random noise before being input into the network. For example, the latent space features corresponding to any random vector can be a (1, 512, 4, 4)-dimensional tensor, and each random noise can also be a (1, 512, 4, 4)-dimensional tensor. After concatenating the latent space features corresponding to any random vector with any random noise in the second dimension, a (1, 1024, 4, 4)-dimensional tensor can be obtained. This application does not limit this specific approach.

[0174] It is worth noting that, generally, this feature vector is a random vector that conforms to a uniform or Gaussian distribution. However, the distribution of various initial face images is very complex. Therefore, by mapping this feature vector to a more complex latent space through a mapping network and expressing it through the features of this latent space, the richness of the representation of the feature vector can be improved. In this way, the problem of inaccurate representation of the target face image due to the feature vector not conforming to a uniform or Gaussian distribution can be avoided.

[0175] Step 1012: Input the latent space features into each style generation module, and input the noise information output by at least one coding layer into each style generation module according to the arrangement order of each coding layer in the initial image inpainting model, to obtain each predicted face image.

[0176] Optionally, there is a correspondence between each coding layer and each style generation model. Generally, the first coding layer corresponds to the last style generation model, ..., and the last coding layer corresponds to the first style generation model.

[0177] For example, if there are 9 pattern generation models, 9 encoding layers are needed. Generally, the Nth encoding layer needs to output noise information to the 10th to Nth pattern generation models, where N is a positive integer less than or equal to 9.

[0178] In this way, each pattern generation module can generate predicted face images based on the latent space features and noise information. Furthermore, since the latent space features are more rich in expression, the accuracy and reliability of the generated predicted face images can be further improved.

[0179] In one possible implementation, noise information output from at least one coding layer is input into each style generation module according to the arrangement order of the coding layers in the initial image inpainting model, to obtain each predicted face image, including:

[0180] Determine the order in which the style generation modules are arranged.

[0181] Optionally, the arrangement order of the style generation modules can refer to the order in which the style generation modules are connected sequentially.

[0182] If the arrangement order of the pattern generation module is 1, then the latent space features, the preset constant term, and random noise are input into the pattern generation module, and the pattern generation module outputs the decoding information.

[0183] Optionally, the preset constant can be a constant used to enable the style generation module to perform normal decoding. The preset constant can be generated by any network layer or module and input into the style generation module.

[0184] Optionally, the decoding information can be used to characterize the image data obtained by the style generation module from decoding the latent space features, preset constants, and random noise.

[0185] It is worth noting that since there are no other style generation modules before the style generation module with the order of 1, this constant term needs to be input so that the first style generation module can decode normally.

[0186] If the arrangement order of the pattern generation modules is not 1 and is less than a preset value, then the latent space feature, the decoding information output by the previous pattern generation module, and random noise are input into the pattern generation module, and the pattern generation module outputs the decoding information.

[0187] Optionally, the preset value can be determined by relevant technical personnel according to actual needs. Generally, the preset value is less than the number of each style generation module, and the preset value can be set to a relatively small value.

[0188] It is worth noting that since there are other style generation modules before the style generation module whose order is not 1, the decoding information output by the style generation module preceding this style generation module needs to be output to these style generation modules. In this way, the encoded data of each encoding layer can be decoded accordingly.

[0189] In addition, since the style generation modules with an order less than the preset value still use random noise input that follows a Gaussian distribution, the random Gaussian noise input can be maintained, so that the style generation adversarial network maintains a high generation capability, thereby improving the quality of the generated preset face image.

[0190] If the arrangement order of the pattern generation modules is equal to or greater than a preset value, then the latent space features, the feature map output by the encoding layer corresponding to the pattern generation module preceding the pattern generation module, and the information output by the pattern generation module preceding the pattern generation module are input into the pattern generation module, and the pattern generation module outputs the decoding information.

[0191] It is worth noting that since each style generation module with an order equal to or greater than the preset value inputs the feature map output from the encoding layer corresponding to the style generation module preceding it, more feature information from the target face image can be added to the generated preset face image, thereby reducing the difference between the preset face image and the target face image.

[0192] The decoded information output by the last style generation module is used as the predicted face image.

[0193] It is worth noting that each style generation module can be regarded as a decoder as a whole. This decoder can re-parse the information contained in the feature vector and generate an image. Therefore, the decoding information output by the last style generation module can be an image, that is, the predicted face image.

[0194] In one possible implementation, see [link to relevant documentation]. Figure 5 The loss information is determined based on each predicted face image and the second training image set, including:

[0195] Step 1013: Determine the initial face images corresponding to each target face image in the second training image set.

[0196] In this way, the correspondence between each target face image and each initial face image can be accurately determined.

[0197] Step 1014: Concatenate each target face image, each initial face image, and each predicted face image into multiple image pairs, and input each image pair into the discriminator.

[0198] For example, if each initial face image is represented by I... lq This indicates that each predicted face image is represented by... This indicates that each initial face image is represented by I. hq This means that the resulting image pair after stitching can be used... It means, and G represents a generative adversarial network in the above style.

[0199] One possible approach is that if the second training image set includes multiple image pairs of high-resolution and low-resolution face images, that is, if the second training image set includes multiple initial image pairs consisting of target face images and corresponding initial face images, then the initial image pairs corresponding to each target face image and each predicted face image can be directly stitched together into multiple image pairs. This application does not limit this approach.

[0200] Step 1015: The discriminator determines the loss information based on each image pair.

[0201] In this way, loss information can be accurately determined to characterize the performance of the initial image inpainting model, the difference between each predicted face image and the original image in the second training image set, or the difference between each predicted face image and the target face image obtained based on the second training image set.

[0202] In one possible implementation, see [link to relevant documentation]. Figure 6 The discriminator determines the loss information based on each image pair, including:

[0203] Step 1016: The discriminator calculates and generates adversarial loss values ​​based on each target face image and each predicted face image.

[0204] For example, the generative adversarial loss value can be calculated according to the following formula:

[0205]

[0206] In this context, G and D represent the aforementioned generative adversarial network and the discriminator, respectively, and concat represents the concatenation operation.

[0207] Step 1017: Calculate the absolute value loss, additional corner edge loss, and gradient difference loss based on each initial face image and each predicted face image.

[0208] For example, the absolute value loss L can be calculated according to the following formula. smooth :

[0209]

[0210] The additional corner edge loss value L can be calculated using the following formula. id :

[0211]

[0212] Arcface refers to the publicly available Arcface network.

[0213] The gradient difference loss value can also be calculated using the following formula:

[0214]

[0215] in, and Representing I hq and The number of pixels, express The pixel value at coordinates (i-1, j).

[0216] In this way, the difference in horizontal and vertical gradients between each predicted face image and each initial face image can be determined by the gradient difference loss value, which facilitates the calculation of the difference values ​​of details such as hair, thereby improving the performance of the target image restoration model or the initial image restoration model in generating more detailed content.

[0217] Step 1018: The weighted sum of the generative adversarial loss value, the absolute loss value, the additional corner edge loss value, and the gradient difference loss value is used as the loss information.

[0218] In this way, the loss information can be accurately determined, which facilitates accurate, reliable and effective training of the initial image restoration model, thereby ensuring the performance of the target image restoration model in generating face images.

[0219] In one possible approach, when training a pattern generative adversarial network based on a first training image set, an initial generative adversarial network can be established based on a preset structure. Then, multiple low-resolution face images can be obtained based on each high-resolution face image in the first training image set. Random vectors are extracted from each low-resolution face image, and then the random vectors and random noise are input into each pattern generation module in the initial generative adversarial network.

[0220] Each style generation module can include a style modulation layer. After inputting the corresponding random vector into the style modulation layer, stylized features are obtained. The stylized features are then concatenated with random noise and subjected to style modulation with upsampling. The modulated result is then output to the next style generation module. This process continues until the last style generation module uses the modulated result as the face image output during training.

[0221] Then, the network loss information of the face image output by the last style generation module can be calculated using the corresponding loss function, and the initial generative adversarial network can be iteratively optimized based on the network loss information. The initial generative adversarial network that meets the performance requirements is used as the final style generative adversarial network that constitutes the initial image inpainting model.

[0222] In this way, the training of the style generation adversarial network can be completed.

[0223] Figure 7 A flowchart illustrating an image restoration model training method provided in this application is shown. This method can be applied to the aforementioned electronic device. See also... Figure 7 This application provides an image restoration method, including:

[0224] Step 2001: Obtain the image to be repaired and input it into the image repair model.

[0225] Optionally, the image to be repaired can be an image of any quality, any contrast, signal-to-noise ratio, degree of distortion and / or resolution, and the image to be repaired can include the entire face or a part of the face, which is not limited in this embodiment of the application.

[0226] Optionally, the image restoration model is a target image restoration model trained in any of the above embodiments, and the image restoration model includes a feature extraction network and a style generation adversarial network.

[0227] Specifically, inputting the image to be repaired into the image restoration model can mean inputting the image to be repaired into the first coding layer of the image restoration model.

[0228] Step 2002: The feature extraction network extracts features from the image to be repaired to obtain the target feature vector.

[0229] Optionally, the target feature vector can be obtained by sequentially encoding the image to be repaired through multiple encoding layers in the feature extraction network and outputting the encoding results, i.e., outputting a feature map, and then adjusting the dimension of the feature map output by the last encoding layer through a fully connected layer in the feature extraction network.

[0230] Step 2003: The pattern generative adversarial network outputs the repaired image of the image to be repaired based on the target feature vector and the noise information output by each coding layer in the feature extraction network.

[0231] Optionally, the restored image is obtained by restoring the image to be restored using the image restoration model. The restored image has higher signal-to-noise ratio, contrast, sharpness, and quality than the image to be restored.

[0232] It is worth noting that in step 2003, when the style generative adversarial network outputs the repaired image based on the target feature vector and the noise information output by each coding layer, it needs to use the feature to extract the noise information output by all coding layers in the network.

[0233] In one possible approach, the pattern-generating adversarial network outputs a repaired image of the image to be repaired based on the target feature vector and the noise information output by each coding layer in the feature extraction network, including:

[0234] The latent space features of the target are obtained by mapping the target feature vector.

[0235] Determine the order in which the style generation modules are arranged.

[0236] If the arrangement order of the pattern generation module is 1, then the target latent space features, the preset constant terms, and the noise information output by the last coding layer in each coding layer are input into the pattern generation module, and the pattern generation module outputs the target decoding information.

[0237] If the arrangement order of the pattern generation modules is not 1, then the latent space features of the target, the target decoding information output by the pattern generation module preceding the pattern generation module, and the noise information output by the coding layer corresponding to the pattern generation module in each coding layer are input into the pattern generation module, and the pattern generation module outputs the target decoding information.

[0238] For example, the nth encoding layer in each encoding layer corresponds to the (M+1)th to (n)th encoding layer in each style generation module. Here, M is the number of encoding layers or the number of style generation modules, and n is a positive integer less than or equal to M.

[0239] When the arrangement order of the pattern generation modules is not 1, the noise information output by the coding layer corresponding to the pattern generation module in each coding layer is input into each pattern generation module. This can mean that the noise information output by all coding layers except the last coding layer is input into the corresponding pattern generation module.

[0240] This ensures that the consistency of feature information between the image to be repaired and the repaired image can be maintained.

[0241] The target decoding information output by the last style generation module is used as the repaired image.

[0242] In this way, the image to be repaired can be repaired based on this image restoration model.

[0243] It should be noted that the image restoration method and the image restoration model training method provided in this application are corresponding. Some terms in the embodiments corresponding to the image restoration method have already been defined in the embodiments corresponding to the image restoration model training method, and can be found in any of the above embodiments.

[0244] It is understood that when executing the image restoration method, it should also be able to execute the various steps in the various embodiments of the corresponding image restoration model training method described above. That is, after the target image restoration model is trained and obtained, when applying the target image restoration model or the image restoration model, the various steps executed based on the initial image restoration model during the training process can be referred to for image restoration.

[0245] Furthermore, since the performance of the target image restoration model obtained through training is already high, it is not necessary to call the discriminator in the model to execute the loss function or iterative optimization process when applying the target image restoration model or the image restoration model.

[0246] Figure 8 This is a schematic diagram of the structure of an image restoration model provided in an embodiment of this application. See also... Figure 8 , Figure 8 The image inpainting model A shown includes a style generative adversarial network Y, a feature extraction network T, and a discriminator P.

[0247] As can be seen, the feature extraction network T includes an encoder B and a fully connected layer q. The encoder B also includes multiple sequentially connected coding layers b.

[0248] The style generation adversarial network Y includes a mapping network y and a decoder j, and the decoder j also includes multiple style generation modules s connected in sequence.

[0249] Specifically, the last coding layer b in each coding layer b is connected to the fully connected layer q, and the fully connected layer q is connected to the mapping network y, which in turn is connected to each style generation module s.

[0250] The first style generation module s in each style generation module is also used to input constant terms.

[0251] In addition, each encoding layer b is connected to each style generation module s in the order of arrangement, specifically the first encoding layer b is connected to the last style generation module s, and so on, the last encoding layer b is connected to the first style generation module s.

[0252] Image restoration model A can be the initial image restoration model, target image restoration model, or image restoration model described above. The working principle and beneficial effects of image restoration model A can be found in the descriptions of other embodiments above, and will not be repeated here.

[0253] The following describes the apparatus, device, and computer-readable storage medium used to implement the image restoration model training method provided in this application. The specific implementation process and technical effects are described above and will not be repeated below.

[0254] Figure 9 This is a schematic diagram of the structure of an image restoration model training device provided in an embodiment of this application. See also... Figure 9 The device includes:

[0255] Module 301 is used to train a style generative adversarial network based on the first training image set, and to combine the style generative adversarial network, feature extraction network and discriminator to form an initial image inpainting model.

[0256] Optionally, the feature extraction network includes multiple coding layers and fully connected layers connected in sequence.

[0257] The first extraction module 302 is used to obtain multiple target face images based on the second training image set, and input each target face image into the feature extraction network for feature extraction to obtain a feature vector.

[0258] The first output module 303 is used to output at least one predicted face image from the pattern generative adversarial network based on the feature vector and noise information output from at least one coding layer in the feature extraction network.

[0259] The optimization module 304 is used to determine loss information based on each predicted face image and the second training image set, and to iteratively optimize the initial image restoration model based on the loss information until a preset condition is met. The initial image restoration model that meets the preset condition is then used as the target image restoration model, and training is stopped.

[0260] Figure 10 This is a schematic diagram of the structure of an image restoration model training device provided in an embodiment of this application. See also... Figure 10 The device includes:

[0261] The input module 401 is used to acquire the image to be repaired and input the image to be repaired into the image repair model.

[0262] Optionally, the image restoration model includes a feature extraction network and a style generative adversarial network.

[0263] The second extraction module 402 is used to extract features from the image to be repaired by the feature extraction network to obtain a feature vector.

[0264] The second output module 403 is used to output the repaired image of the image to be repaired by the pattern generative adversarial network based on the feature vector and the noise information output by each coding layer in the feature extraction network.

[0265] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0266] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more microcontrollers, or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SoC).

[0267] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. See also... Figure 11 The electronic device includes: a memory 501 and a processor 502. The memory 501 stores a computer program that can run on the processor 502. When the processor 502 executes the computer program, it implements the steps in any of the above method embodiments.

[0268] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the various method embodiments described above.

[0269] Optionally, this application also provides a program product, such as a computer-readable storage medium, including a program that, when executed by a processor, performs any of the above-described image restoration model training methods or image restoration method embodiments.

[0270] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0271] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0272] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0273] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) or processor to execute partial steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0274] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0275] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for training an image restoration model, characterized in that, The method includes: A pattern generative adversarial network is trained based on the first training image set, and the pattern generative adversarial network, the feature extraction network, and the discriminator are combined to form an initial image inpainting model. The feature extraction network includes multiple coding layers and fully connected layers connected in sequence. Multiple target face images are obtained based on the second training image set, and each target face image is input into the feature extraction network for feature extraction to obtain a feature vector; The pattern-generating adversarial network outputs at least one predicted face image based on the feature vector and noise information output by at least one coding layer in the feature extraction network; Loss information is determined based on the predicted face images and the second training image set. The initial image restoration model is iteratively optimized based on the loss information until a preset condition is met. The initial image restoration model that meets the preset condition is taken as the target image restoration model, and training is stopped. The step of determining loss information based on each of the predicted face images and the second training image set includes: Determine the initial face image corresponding to each of the target face images in the second training image set; Each of the target face images, each of the initial face images, and each of the predicted face images are concatenated into multiple image pairs, and each of the image pairs is input into the discriminator; The discriminator determines the loss information based on each of the image pairs; The step of the discriminator determining the loss information based on each of the image pairs includes: The discriminator calculates and generates adversarial loss values ​​based on each of the target face images and each of the predicted face images; Calculate the absolute value loss, additional corner edge loss, and gradient difference loss based on each of the initial face images and each of the predicted face images; The loss information is obtained by weighted summation of the generative adversarial loss value, the absolute value loss value, the additional corner edge loss value, and the gradient difference loss value.

2. The image restoration model training method as described in claim 1, characterized in that, The process involves obtaining multiple target face images based on the second training image set, and inputting each target face image into the feature extraction network for feature extraction to obtain a feature vector, including: The initial face images in the second training image set are degraded to obtain each target face image, and the quality of each target face image is lower than that of each initial face image. Each of the target face images is input into the first encoding layer of the feature extraction network, and then encoded sequentially by each encoding layer of the feature extraction network to obtain a feature map; The feature map is dimensionally adjusted by the fully connected layer in the feature extraction network to obtain the feature vector.

3. The image restoration model training method as described in claim 2, characterized in that, The step of degrading the initial face images in the second training image set to obtain each of the target face images includes: The initial face image is filtered using a bilateral filtering algorithm, and then blurred using a convolution kernel. The blurred features are downsampled and then subjected to Gaussian noise processing. The features processed by Gaussian noise are compressed and upsampled to obtain the target face image.

4. The image restoration model training method as described in claim 1, characterized in that, The style generation adversarial network includes multiple style generation modules, which are connected in sequence. The generative adversarial network (GAN) outputs at least one predicted face image based on the feature vector and the noise information output by each coding layer in the feature extraction network, including: The feature vector is mapped to latent space features through the mapping network in the pattern generative adversarial network. The latent space features are input into each of the pattern generation modules, and the noise information output by at least one coding layer is input into each of the pattern generation modules according to the arrangement order of each coding layer in the initial image inpainting model, so as to obtain each of the predicted face images.

5. The image restoration model training method as described in claim 4, characterized in that, The step of inputting noise information output from at least one coding layer into each of the style generation modules according to the arrangement order of the coding layers in the initial image inpainting model to obtain each of the predicted face images includes: Determine the arrangement order of each style generation module; If the arrangement order of the pattern generation modules is 1, then the latent space features, the preset constant terms, and random noise are input into the pattern generation module, and the pattern generation module outputs the decoding information. If the arrangement order of the style generation modules is not 1 and is less than a preset value, then the latent space features, the decoding information output by the previous style generation module, and random noise are input into the style generation module, and the style generation module outputs the decoding information. If the arrangement order of the style generation modules is equal to or greater than a preset value, then the latent space features, the feature map output by the encoding layer corresponding to the style generation module preceding the style generation module, and the information output by the style generation module preceding the style generation module are input into the style generation module, and the style generation module outputs the decoding information. The decoded information output by the last style generation module is used as the predicted face image.

6. An image restoration method, characterized in that, The method includes: The image to be repaired is obtained and input into the image repair model. The image repair model is trained by the image repair model training method according to any one of claims 1-5. The image repair model includes: a feature extraction network and a style generation adversarial network. The feature extraction network extracts features from the image to be repaired to obtain a target feature vector; The pattern generative adversarial network outputs the repaired image of the image to be repaired based on the target feature vector and the noise information output by each coding layer in the feature extraction network.

7. An image restoration model training device, characterized in that, The device includes: The component module is used to train a style generative adversarial network based on a first training image set, and to combine the style generative adversarial network, a feature extraction network, and a discriminator to form an initial image inpainting model. The feature extraction network includes multiple coding layers and fully connected layers connected in sequence. The first extraction module is used to obtain multiple target face images based on the second training image set, and input each of the target face images into the feature extraction network for feature extraction to obtain a feature vector; The first output module is used to output at least one predicted face image from the style generative adversarial network based on the feature vector and noise information output by at least one coding layer in the feature extraction network. An optimization module is used to determine loss information based on each of the predicted face images and the second training image set, iteratively optimize the initial image restoration model based on the loss information until a preset condition is met, take the initial image restoration model that meets the preset condition as the target image restoration model, and stop training. The optimization module is specifically used for: Determine the initial face image corresponding to each of the target face images in the second training image set; Each of the target face images, each of the initial face images, and each of the predicted face images are concatenated into multiple image pairs, and each of the image pairs is input into the discriminator; The discriminator determines the loss information based on each of the image pairs; The discriminator calculates and generates adversarial loss values ​​based on each of the target face images and each of the predicted face images; Calculate the absolute value loss, additional corner edge loss, and gradient difference loss based on each of the initial face images and each of the predicted face images; The loss information is obtained by weighted summation of the generative adversarial loss value, the absolute value loss value, the additional corner edge loss value, and the gradient difference loss value.

8. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program that can run on the processor, and when the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

Images