Image processing method, image processing device, electronic device and readable storage medium

Abstract

The present disclosure provides an image processing method, an image processing device, an electronic device, and a readable storage medium. The image processing method includes: receiving an input image; processing the input image using a first generator to obtain an output image, wherein the output image has a higher definition than the input image; wherein the first generator is trained using at least two discriminators to train a generator to be trained. In the present disclosure, because the first generator used for image restoration is trained using at least two discriminators, the restored image can be enriched in detail, thereby improving the restoration effect.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image processing technology, and in particular to an image processing method, an image processing device, an electronic device, and a readable storage medium. Background Art

[0002] Image quality restoration technology is widely used in areas such as photo restoration and video sharpening. Most current algorithms use super-resolution reconstruction techniques to restore low-resolution images, but the resulting images are often smooth or prone to deformation of facial features during the restoration process. Therefore, improving image restoration is a pressing technical challenge. Summary of the Invention

[0003] The embodiments of the present disclosure provide an image processing method, an image processing device, an electronic device, and a readable storage medium, which are used to solve the problem that current image restoration methods have unsatisfactory restoration effects.

[0004] In order to solve the above technical problems, the present disclosure is implemented as follows:

[0005] In a first aspect, an embodiment of the present disclosure provides an image processing method, comprising:

[0006] receiving an input image;

[0007] Processing the input image using a first generator to obtain an output image, wherein the clarity of the output image is higher than the clarity of the input image;

[0008] The first generator is obtained by training the generator to be trained using at least two discriminators.

[0009] In a second aspect, an embodiment of the present disclosure provides an image processing method, including:

[0010] receiving an input image;

[0011] Performing face detection on the input image to obtain a face image;

[0012] Processing the facial image using the method of the first aspect to obtain a first restoration training image, wherein the clarity of the first restoration training image is higher than the clarity of the input image;

[0013] Processing the input image or the input image with the face image removed to obtain a second restoration training image, wherein the clarity of the second restoration training image is higher than the clarity of the input image;

[0014] The first restoration training image and the second restoration training image are fused to obtain a fused image, wherein the clarity of the fused image is higher than the clarity of the input image.

[0015] In a third aspect, an embodiment of the present disclosure provides an image processing device, including:

[0016] A receiving module, configured to receive an input image;

[0017] a processing module, configured to process the input image using a first generator to obtain an output image, wherein the clarity of the output image is higher than that of the input image;

[0018] The first generator is obtained by training the generator to be trained using at least two discriminators.

[0019] In a fourth aspect, an embodiment of the present disclosure provides an image processing device, including:

[0020] A receiving module, configured to receive an input image;

[0021] A face detection module, configured to perform face detection on the input image to obtain a face image;

[0022] A first processing module, configured to process the facial image using the method described in the first aspect to obtain a first restoration training image, wherein the clarity of the first restoration training image is higher than that of the input image;

[0023] a second processing module, configured to process the input image or the input image with the face image removed to obtain a second restoration training image, wherein the clarity of the second restoration training image is higher than that of the input image;

[0024] The first restoration training image and the second restoration training image are fused to obtain a fused image, wherein the clarity of the fused image is higher than the clarity of the input image.

[0025] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the image processing method described in the first aspect above, or, when executed by the processor, implements the steps of the image processing method described in the second aspect above.

[0026] In a sixth aspect, an embodiment of the present disclosure provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the image processing method as described in the first aspect above are implemented, or the steps of the image processing method as described in the second aspect above are implemented.

[0027] In the embodiment of the present disclosure, since the first generator used for image restoration is trained using at least two discriminators, the restored image details can be enriched, thereby improving the restoration effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present disclosure. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0029] Figure 1 1 is a flow chart of an image processing method according to an embodiment of the present disclosure;

[0030] Figure 2 Schematic diagram of a multi-scale first generator according to an embodiment of the present disclosure;

[0031] Figure 3 Schematic diagram of the flow of an image processing method according to another embodiment of the present disclosure;

[0032] Figure 4 Schematic diagram of the flow of an image processing method according to another embodiment of the present disclosure;

[0033] Figure 5 A schematic diagram of a key point extraction method according to an embodiment of the present disclosure;

[0034] Figure 6 Schematic diagram of a method for generating a key point mask image according to an embodiment of the present disclosure;

[0035] Figure 7 A schematic diagram of a multi-scale first generator according to another embodiment of the present disclosure;

[0036] Figure 8 Schematic diagram of various types of losses of a generator according to an embodiment of the present disclosure;

[0037] Figure 9 、 Figure 11 、 Figure 13 、 Figure 17 、 Figure 18 、 Figure 19 Schematic diagram of a generator training method according to an embodiment of the present disclosure;

[0038] Figure 10 、 Figure 12 、 Figure 14 Schematic diagram of a discriminator training method according to an embodiment of the present disclosure;

[0039] Figure 15 is a schematic diagram of an image of a facial part according to an embodiment of the present disclosure;

[0040] Figure 16 Schematic diagram of input and output of a generator and a discriminator according to an embodiment of the present disclosure;

[0041] Figure 20 A schematic diagram of a generator training method according to another embodiment of the present disclosure;

[0042] Figure 21 A schematic diagram of a discriminator training method according to another embodiment of the present disclosure;

[0043] Figure 22 Schematic diagram of input and output of a generator and a discriminator according to another embodiment of the present disclosure;

[0044] Figure 23 A schematic diagram of a generator training method according to another embodiment of the present disclosure;

[0045] Figure 24 A schematic diagram of a discriminator training method according to another embodiment of the present disclosure;

[0046] Figure 25 Schematic diagram of the flow of an image processing method according to another embodiment of the present disclosure;

[0047] Figure 26 Schematic diagram of the structure of an image processing device according to an embodiment of the present disclosure;

[0048] Figure 27 FIG. 4 is a structural diagram of an image processing device according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0050] Please refer to Figure 1 , an embodiment of the present disclosure provides an image processing method, including:

[0051] Step 11: Receive input image;

[0052] The input image may be an image to be processed, for example, an image with lower definition. The image to be processed may be a video frame extracted from a video, or a picture downloaded from the Internet or taken by a camera, or an image obtained by other means, and the embodiments of the present disclosure are not limited thereto. For example, there is a lot of noise in the input image, and the image quality is relatively blurry, so it is necessary to utilize the image processing method provided by the embodiments of the present disclosure to denoise and / or deblur, thereby improving clarity and achieving image quality enhancement. For example, when the input image is a color image, the input image may include a red (R) channel input image, a green (G) channel input image, and a blue (B) channel input image.

[0053] Step 12: Use the first generator to process the input image to obtain an output image, wherein the clarity of the output image is higher than the clarity of the input image; wherein the first generator is obtained by training the generator to be trained using at least two discriminators.

[0054] The first generator can be a pre-trained neural network. The generator to be trained can be a network built based on the convolutional neural network structure described above, but its parameters still require training. For example, the generator to be trained is used to train the first generator, and the parameters of the generator to be trained are greater than those of the first generator. For example, the parameters of a neural network include the weight parameters of each convolutional layer in the neural network. The larger the absolute value of a weight parameter, the greater the contribution of the neuron corresponding to that weight parameter to the output of the neural network, and thus the more important it is to the neural network. Generally, a neural network with more parameters is more complex and has a greater "capacity," which means that the neural network can complete more complex learning tasks. Compared to the generator to be trained, the first generator is simplified, with fewer parameters and a simpler network structure. This means that the first generator consumes fewer resources (e.g., computing resources, storage resources, etc.) during operation, making it suitable for use in lightweight terminals. Using this training method, the first generator can learn the reasoning capabilities of the generator to be trained, thereby enabling the first generator to possess a simple structure while also possessing strong reasoning capabilities.

[0055] It should be noted that in the embodiments of the present disclosure, "clarity" refers, for example, to the clarity of detailed image textures and their boundaries within an image. The higher the clarity, the better the perception of the human eye. The clarity of the restored training image is higher than that of the input image. This means, for example, that the input image is processed using the image processing methods provided in the embodiments of the present disclosure, such as by performing denoising and / or deblurring, so that the restored training image obtained after processing is clearer than the input image.

[0056] In the embodiment of the present disclosure, the input image may include a face image, that is, the first generator is used to perform face restoration. Of course, the input image may also be other types of images.

[0057] In the embodiment of the present disclosure, since the first generator used for image restoration is trained using at least two discriminators, the restored image details can be enriched, thereby improving the restoration effect.

[0058] In some embodiments of the present disclosure, the first generator optionally includes N restoration modules, which are used to perform denoising and / or deblurring on an input image of a specified scale to improve the clarity of the input image. Wherein, N is an integer greater than or equal to 2. In some preferred embodiments, N can be equal to 4. For further preferred embodiments, please refer to Figure 2 The four repair modules include: a 64*64 scale repair module, a 128*128 scale repair module, a 256*256 scale repair module, and a 512*512 scale repair module. Of course, the number of repair modules can also be other values, and the scales corresponding to each repair module are not limited to the four examples above.

[0059] In the embodiments of the present disclosure, scale refers to resolution.

[0060] In the embodiment of the present disclosure, optionally, the network structure adopted by the repair module is SRCNN or U-Net.

[0061] In the embodiment of the present disclosure, optionally, processing the input image by using the first generator to obtain the output image includes:

[0062] Processing the input image into images to be restored at N scales, wherein the scales of the image to be restored at a first scale to the image to be restored at an Nth scale increase in sequence;

[0063] The output image is obtained using the N restoration modules and the image to be restored at the N scales. In the disclosed embodiment, optionally, the scale of two adjacent scales in the N scales is twice the scale of the previous scale. For example, the N scales are 64*64, 128*128, 256*256, and 512*512.

[0064] In the embodiment of the present disclosure, optionally, processing the input image into images to be restored at N scales includes:

[0065] Determining the scale interval to which the input image belongs;

[0066] Processing the input image into an image to be restored at a j-th scale corresponding to the scale interval to which the input image belongs, where the j-th scale is one of the first scale to the N-th scale;

[0067] Upsampling and / or downsampling processing is performed on the image to be restored at the j-th scale to obtain images to be restored at the remaining N-1 scales.

[0068] The upsampling and downsampling in the above embodiments may be interpolation, such as bicubic interpolation.

[0069] That is, the input image can be first processed into an image to be restored at one of N scales, and then the image to be restored can be upsampled and / or downsampled to obtain images to be restored at other N-1 scales. Alternatively, the input image can be sequentially sampled into images to be restored at N scales.

[0070] Please refer to Figure 2 , Figure 2 In the illustrated embodiment, the scale interval to which the input image belongs is first determined. If the scale of the input image is less than or equal to 96*96, the input image is upsampled or downsampled to obtain a 64*64 scale training image to be repaired. The 64*64 scale training image to be repaired is then upsampled to obtain 128*128, 256*256, or 512*512 scale training images to be repaired. If the scale of the input image is greater than 96*96 and less than or equal to 192*192, the input image is upsampled or downsampled to obtain a 128*128 scale training image to be repaired. The 128*128 scale training image to be repaired is then downsampled and upsampled to obtain 64*64, 256*256, or 512*512 scale training images to be repaired. If the scale of the input image is greater than 192*192 and less than or equal to 384*384, the input image is upsampled or downsampled to obtain a 256*256 scale training image to be inpainted. Then, the 256*256 scale training image to be inpainted is downsampled and upsampled to obtain 64*64, 128*128, and 512*512 scale training images to be inpainted. If the scale of the input image is greater than 384*384, the input image is upsampled or downsampled to obtain a 512*512 scale training image to be inpainted. Then, the 512*512 scale training image to be inpainted is downsampled to obtain 64*64, 128*128, and 256*256 scale training images to be inpainted.

[0071] Of course, it should be noted that the above-mentioned numerical value for determining the interval to which the input image belongs can be selected as needed. In the above-mentioned scheme, the middle scale of two adjacent scales of the image to be repaired of N scales is taken. For example, the middle scale of the two adjacent scales of 64*64 and 128*128 is 96*96, and the middle scale of the two adjacent scales of 128*128 and 256*256 is 192*192, and so on. Of course, the specific scheme is not limited to the above-mentioned 96*96, 192*192, and 384*384.

[0072] In the above embodiment, upsampling or downsampling can be achieved by interpolation.

[0073] In some embodiments of the present disclosure, please refer to Figure 3 , using the N restoration modules and the N-scaled images to be restored, obtaining the output image includes:

[0074] Step 31: stitching the image to be repaired at a first scale and the random noise image at a first scale to obtain a first stitched image; inputting the first stitched image into a first repair module to obtain a repaired image at a first scale; and upsampling the repaired image at the first scale to obtain an upsampled image at a second scale.

[0075] The random noise image of the first scale may be randomly generated, or may be generated by upsampling or downsampling a random noise image of the same scale as the input image.

[0076] Still Figure 2 Take this as an example to illustrate, and get the 64*64 scale image to be repaired (i.e. Figure 2 After receiving the input 1) and the 64*64 scale random noise image, the 64*64 scale image to be repaired and the 64*64 scale random noise image are spliced to obtain a first spliced image, and the first spliced image is input into the first repair module to obtain a 64*64 scale repaired image, and then the 64*64 scale repaired image is upsampled to obtain a 128*128 scale upsampled image;

[0077] Step 32: stitching the i-th scale upsampled image, the i-th scale to-be-inpainted image, and the i-th scale random noise image to obtain an i-th stitched image; inputting the i-th stitched image into the i-th inpainting module to obtain an i-th scale inpainted image; upsampling the i-th scale inpainted image to obtain an i+1-th scale upsampled image; wherein i is an integer greater than or equal to 2;

[0078] The i-th repair module is a repair module located between the first repair module and the last repair module.

[0079] Still Figure 2 For example, for the second restoration module, first the 128*128 scale image to be restored (i.e. Figure 2 Input 2 in the above example), the 128*128 scale random noise image and the 128*128 scale upsampled image are stitched together to obtain a second stitched image, and the second stitched image is input into the second restoration module to obtain a 128*128 scale restoration image, and then the 128*128 scale restoration image is upsampled to obtain a 256*256 scale upsampled image; for the third restoration module, the obtained 256*256 scale image to be restored (i.e. Figure 2 Input 3), the 256*256 scale random noise image and the 256*256 scale upsampled image are stitched together to obtain a third stitched image, and the third stitched image is input into the third restoration module to obtain a 256*256 scale restoration image, and then the 256*256 scale restoration image is upsampled to obtain a 512*512 scale upsampled image;

[0080] Step 33: stitch the upsampled image of the Nth scale, the image to be repaired of the Nth size, and the random noise image of the Nth scale to obtain an Nth stitched image; input the Nth stitched image into the Nth repair module to obtain an Nth scale repaired image as the repair training image of the first generator.

[0081] Still Figure 2 As an example, for the last restoration module, first the 512*512 scale image to be restored (i.e. Figure 2 Input 4), a 512*512 scale random noise image and a 512*512 scale upsampled image are spliced to obtain a fourth spliced image, and the fourth spliced image is input into the last restoration module to obtain a 512*512 scale restoration image as the restoration training image of the first generator.

[0082] In this disclosed embodiment, random noise is added to the first generator during image restoration. This is because if a blurred image is fed alone into the first generator, the resulting restored image may appear overly smoothed due to a lack of high-frequency information. Adding random noise to the input of the first generator allows it to be mapped to high-frequency information in the restored image, thereby enriching the details of the restored image.

[0083] In some other embodiments of the present disclosure, please refer to Figure 4 , using the N restoration modules and the N-scaled images to be restored, obtaining the output image includes:

[0084] Step 41: For each scale of the image to be repaired, extract key points from the image to be repaired, generate multiple key point heat maps, merge and classify the key point heat maps, and obtain S key point mask images for each scale, where S is an integer greater than or equal to 2;

[0085] In the embodiment of the present disclosure, it is optional, please refer to Figure 5 , a 4-stack hourglass model can be used to extract key points from the image to be repaired, for example, 68 key points from a face image are extracted, and 68 key point heat maps are generated, where each key point heat map represents the probability that all pixels on the image are a certain key point (landmark). Then, please refer to Figure 6 , multiple keypoint heatmaps are merged (Merge) and classified (softmax) to obtain S keypoint mask images corresponding to different facial components. For example, S can be 5, and the corresponding facial components can be: left eye, right eye, nose, mouth, and outline. Of course, in other embodiments of the present disclosure, other keypoint extraction techniques are not excluded to extract keypoints in the image to be repaired, the extracted keypoints are not limited to 68, and the number of keypoint mask images is not limited to 5, that is, the number of facial components is not limited to 5.

[0086] Step 42: stitching the image to be inpainted at the first scale and the S keypoint mask images at the first scale to obtain a first stitched image; inputting the first stitched image into a first inpainting module to obtain an inpainted image at the first scale; and upsampling the inpainted image at the first scale to obtain an upsampled image at the second scale.

[0087] by Figure 7 For example, after obtaining a 64*64 scale image to be repaired and a 64*64 scale key point mask image, the 64*64 scale image to be repaired and the 64*64 scale key point mask image are spliced to obtain a first spliced image, and the first spliced image is input into the first repair module to obtain a 64*64 scale repaired image, and then the 64*64 scale repaired image is upsampled to obtain a 128*128 scale upsampled image;

[0088] Step 43: stitching the i-th scale upsampled image, the i-th scale image to be inpainted, and the S i-th scale keypoint mask images to obtain an i-th stitched image; inputting the i-th stitched image into the i-th inpainting module to obtain an i-th scale inpainted image; upsampling the i-th scale inpainted image to obtain an i+1-th scale upsampled image; wherein i is an integer greater than or equal to 2;

[0089] The i-th repair module is a repair module located between the first repair module and the last repair module.

[0090] by Figure 7 Taking an example to illustrate, for the second restoration module, the obtained 128*128 scale image to be restored, the 128*128 scale key point mask image and the 128*128 scale upsampled image are first spliced to obtain a second spliced image, and the second spliced image is input into the second restoration module to obtain a 128*128 scale restoration image, and then the 128*128 scale restoration image is upsampled to obtain a 256*256 scale upsampled image; for the third restoration module, the obtained 256*256 scale image to be restored, the 256*256 scale key point mask image and the 256*256 scale upsampled image are first spliced to obtain a third spliced image, and the third spliced image is input into the third restoration module to obtain a 256*256 scale restoration image, and then the 256*256 scale restoration image is upsampled to obtain a 512*512 scale upsampled image;

[0091] Step 44: stitch the upsampled image at the Nth scale, the image to be repaired at the Nth scale, and the S key point mask images at the Nth scale to obtain an Nth stitched image; input the Nth stitched image into the Nth repair module to obtain an Nth scale repaired image as the repair training image for the first generator.

[0092] Still Figure 7 Taking the last restoration module as an example, the 512*512 scale image to be restored, the 512*512 scale key point mask image and the 512*512 scale upsampled image are first spliced to obtain a fourth spliced image, and the fourth spliced image is input into the last restoration module to obtain a 512*512 scale restoration image as the restoration training image of the first generator.

[0093] In the disclosed embodiment, the facial key point heat map is introduced into the image sharpening process, which can reduce the degree of deformation of facial features while ensuring image sharpness, thereby improving the final image restoration effect.

[0094] The following describes the training method of the first generator in the embodiment of the present disclosure.

[0095] In an embodiment of the present disclosure, optionally, the first generator is obtained by training the generator to be trained using at least two discriminators, including: alternately training the generator to be trained and the at least two discriminators according to training images and verification images to obtain the first generator, wherein the clarity of the verification image is higher than the clarity of the training image, and when training the generator to be trained, the total loss of the generator to be trained includes at least one of the following: the first loss and the total adversarial loss of the at least two discriminators.

[0096] In some embodiments of the present disclosure, optionally, the first generator includes N repair modules, wherein N is an integer greater than or equal to 2. In some preferred embodiments, N may be equal to 4. For further preferred embodiments, please refer to Figure 2 , the four repair modules include: a 64*64 scale repair module, a 128*128 scale repair module, a 256*256 scale repair module and a 512*512 scale repair module. Of course, the number of the above repair modules can also be other values, and the scale corresponding to each repair module is not limited to the four examples mentioned above. The at least two discriminators include: N first-class discriminators with different network structures corresponding to the N repair modules; for example, if the first generator includes 4 repair modules, then the at least two discriminators include 4 first-class discriminators, please refer to Figure 8 , the four first-class discriminators can be respectively Figure 8 The discriminators 1, 2, 3 and 4 in the above dataset are trained with the first type of discriminators corresponding to multiple scales. The trained first generator can process and output facial images that are closer to real facial images than the first generator trained with a single discriminator at a single scale. The restoration effect is also better, with richer details and smaller deformation (it should be noted that Figure 8 The gray filling in the image is to cover the real face image or the real partial face image).

[0097] The training processes of the generator to be trained and the at least two discriminators are described below respectively.

[0098] Among them, please refer to Figure 9 , training the generator to be trained includes:

[0099] Step 91: Processing the training image into training images to be restored at N scales;

[0100] In the disclosed embodiment, the training image may first be processed into a training image to be restored at one of N scales, and then the training image to be restored may be upsampled and / or downsampled to obtain training images to be restored at other N-1 scales. Alternatively, the training image may be sequentially downsampled into training images to be restored at N scales.

[0101] by Figure 8 For example, the training image can be processed into four training images to be repaired with scales of 64*64, 128*128, 256*256 and 512*512.

[0102] Step 92: Inputting the N scales of the to-be-inpainted training images into the generator to be trained or the generator after the last training to obtain N scales of inpainted training images;

[0103] In the embodiment of the present disclosure, if it is the first time to train the generator to be trained, the training images to be repaired of the N scales are input into the generator to be trained; if it is not the first time to train, the training images to be repaired of the N scales are input into the generator after the last training.

[0104] The specific processing method of the trained generator for the N-scale training images to be repaired can be found in Figure 3 and Figure 4 The processing method in the illustrated embodiment will not be repeated here.

[0105] by Figure 8 For example, four to-be-restored training images of scales 64*64, 128*128, 256*256, and 512*512 are input into the generator to be trained or the generator after the last training to obtain four to-be-restored training images of scales 64*64, 128*128, 256*256, and 512*512.

[0106] Step 93: For each scale of the restored training image, set the restored training image of each scale to have a true value label, and input the restored training image with the true value label into the initial first-class discriminator or the first-class discriminator after the previous training to obtain a first identification result;

[0107] by Figure 8For example, the 64*64 scale repair training image is set to have a true value label, and the 64*64 scale repair training image with the true value label is input into the discriminator 1 to obtain the identification result of the discriminator 1; the 128*128 scale repair training image is set to have a true value label, and the 128*128 scale repair training image with the true value label is input into the discriminator 2 to obtain the identification result of the discriminator 2; the 256*256 scale repair training image is set to have a true value label, and the 256*256 scale repair training image with the true value label is input into the discriminator 3 to obtain the identification result of the discriminator 3; the 512*512 scale repair training image is set to have a true value label, and the 512*512 scale repair training image with the true value label is input into the discriminator 4 to obtain the identification result of the discriminator 4.

[0108] Step 94: Calculate a first adversarial loss based on the first identification result; the total adversarial loss includes the first adversarial loss.

[0109] Optionally, the first adversarial loss is the sum of the adversarial losses corresponding to the restored training images at each scale.

[0110] Step 95: Adjust the parameters of the generator to be trained according to the total adversarial loss.

[0111] Among them, please refer to Figure 10 , training the at least two discriminators comprises:

[0112] Step 101: Processing the training image into training images to be restored at N scales, and processing the verification image into verification images at N scales;

[0113] In the disclosed embodiment, the training image may first be processed into a training image to be restored at one of N scales, and then the training image to be restored may be upsampled and / or downsampled to obtain training images to be restored at other N-1 scales. Alternatively, the training image may be sequentially downsampled into training images to be restored at N scales.

[0114] In the disclosed embodiment, the verification image may be first processed into a verification image at one of N scales, and then the processed verification image may be upsampled and / or downsampled to obtain verification images at other N-1 scales. Alternatively, the verification image may be sequentially sampled into verification images at N scales.

[0115] by Figure 8For example, the training image can be processed into four training images to be restored with the scales of 64*64, 128*128, 256*256, and 512*512. The verification image can be processed into four verification images with the scales of 64*64, 128*128, 256*256, and 512*512.

[0116] Step 102: inputting the N scales of the to-be-inpainted training images into the generator to be trained or the generator after the last training to obtain N scales of inpainted training images;

[0117] The specific processing method of the trained generator for the N-scale training images to be repaired can be found in Figure 3 and Figure 4 The processing method in the illustrated embodiment will not be described again.

[0118] by Figure 8 For example, four to-be-restored training images of scales 64*64, 128*128, 256*256, and 512*512 are input into the generator to be trained or the generator after the last training to obtain four to-be-restored training images of scales 64*64, 128*128, 256*256, and 512*512.

[0119] Step 103: For each scale of the inpainted training image, set the inpainted training image of each scale to have a false value label, and input the inpainted training image with the false value label to the initial first-class discriminator or the first-class discriminator after the last training to obtain a third identification result; set the verification image of each scale to have a true value label, and input each verification image with the true value label to the first-class discriminator to obtain a fourth identification result;

[0120] by Figure 8For example, the 64*64 scale repair training image is set to have a false value label, and the 64*64 scale repair training image with the false value label is input into the discriminator 1 to obtain the third identification result of the discriminator 1, the 64*64 scale verification image is set to have a true value label, and the 64*64 scale verification image with the true value label is input into the discriminator 1 to obtain the fourth identification result of the discriminator 1; the 128*128 scale repair training image is set to have a false value label, and the 128*128 scale repair training image with the false value label is input into the discriminator 2 to obtain the third identification result of the discriminator 2, the 128*128 scale verification image is set to have a true value label, and the 128*128 scale verification image with the true value label is input into the discriminator 2 to obtain the fourth identification result of the discriminator 2; 2 The 56*256 scale repair training image is set to have a false value label, and the 256*256 scale repair training image with a false value label is input into the discriminator 3 to obtain the third identification result of the discriminator 3, and the 256*256 scale verification image is set to have a true value label, and the 256*256 scale verification image with a true value label is input into the discriminator 3 to obtain the fourth identification result of the discriminator 3; the 512*512 scale repair training image is set to have a false value label, and the 512*512 scale repair training image with a false value label is input into the discriminator 4 to obtain the third identification result of the discriminator 4, and the 512*512 scale verification image is set to have a true value label, and the 512*512 scale verification image with a true value label is input into the discriminator 4 to obtain the fourth identification result of the discriminator 4.

[0121] Step 104: Calculating a third adversarial loss based on the third identification result and the fourth identification result;

[0122] Step 105: Adjust the parameters of the first class discriminator according to the third adversarial loss to obtain an updated first class discriminator.

[0123] In some embodiments of the present disclosure, optionally, the at least two discriminators further include: a first type of discriminator and a second type of discriminator having N different network structures corresponding to the N restoration modules, wherein the second type of discriminator is configured to improve the clarity of the restoration of the local face of the training image by the first generator, so that the clarity of the local face features in the image output by the trained first generator is higher;

[0124] The following describes the training process of the generator to be trained and at least two discriminators respectively.

[0125] Please refer to Figure 11 , training the generator to be trained includes:

[0126] Step 111: Processing the training image into training images to be restored at N scales;

[0127] In the disclosed embodiment, the training image may first be processed into a training image to be restored at one of N scales, and then the training image to be restored may be upsampled and / or downsampled to obtain training images to be restored at other N-1 scales. Alternatively, the training image may be sequentially downsampled into training images to be restored at N scales.

[0128] by Figure 8 For example, the training image can be processed into four training images to be repaired with scales of 64*64, 128*128, 256*256 and 512*512.

[0129] Step 112: inputting the N scales of the to-be-inpainted training images into the generator to be trained or the generator after the last training to obtain N scales of inpainted training images;

[0130] The specific processing method of the trained generator for the N-scale training images to be repaired can be found in Figure 3 and Figure 4 The processing method in the illustrated embodiment will not be described again.

[0131] by Figure 8 For example, four to-be-restored training images of scales 64*64, 128*128, 256*256, and 512*512 are input into the generator to be trained or the generator after the last training to obtain four to-be-restored training images of scales 64*64, 128*128, 256*256, and 512*512.

[0132] Step 113: Obtain the first partial face image of the N-th scale restoration training image;

[0133] In some embodiments of the present disclosure, optionally, the first partial face image is an eye image. In the embodiment of the present disclosure, the eye image in the N-th scale restoration training image can be directly captured as the first partial face image.

[0134] Step 114: For each scale of the restored training image, set the restored training image of each scale to have a true value label, and input the restored training image with the true value label into the initial first-class discriminator or the first-class discriminator after the previous training to obtain a first identification result;

[0135] by Figure 8For example, the 64*64 scale repair training image is set to have a true value label, and the 64*64 scale repair training image with the true value label is input into the discriminator 1 to obtain an identification result of the discriminator 1; the 128*128 scale repair training image is set to have a true value label, and the 128*128 scale repair training image with the true value label is input into the discriminator 2 to obtain an identification result of the discriminator 2; the 256*256 scale repair training image is set to have a true value label, and the 256*256 scale repair training image with the true value label is input into the discriminator 3 to obtain an identification result of the discriminator 3; the 512*512 scale repair training image is set to have a true value label, and the 512*512 scale repair training image with the true value label is input into the discriminator 4 to obtain the first identification result of the discriminator 4.

[0136] Step 115: setting the first partial face image to have a true value label, and inputting the first partial face image with the true value label into the initial second type discriminator or the second type discriminator after the last training to obtain a second identification result;

[0137] by Figure 8 For example, Figure 8 The discriminator 5 in the example is a second type of discriminator, which sets the first face partial image to have a true value label, and inputs the first face partial image with the true value label into the discriminator 5 to obtain a second identification result of the discriminator 5;

[0138] Step 116: Calculate a first adversarial loss based on the first identification result; calculate a second adversarial loss based on the second identification result, where the total adversarial loss includes the first adversarial loss and the second adversarial loss;

[0139] Optionally, the first adversarial loss is the sum of the adversarial losses corresponding to the restored training images at each scale.

[0140] Step 117: Adjust parameters of the generator to be trained or the generator trained last time according to the total adversarial loss.

[0141] Please refer to Figure 12 , training the at least two discriminators comprises:

[0142] Step 121: Processing the training image into training images to be restored at N scales, and processing the verification image into verification images at N scales;

[0143] In the disclosed embodiment, the training image may first be processed into a training image to be restored at one of N scales, and then the training image to be restored may be upsampled and / or downsampled to obtain training images to be restored at other N-1 scales. Alternatively, the training image may be sequentially downsampled into training images to be restored at N scales.

[0144] In the disclosed embodiment, the verification image may be first processed into a verification image at one of N scales, and then the processed verification image may be upsampled and / or downsampled to obtain verification images at other N-1 scales. Alternatively, the verification image may be sequentially sampled into verification images at N scales.

[0145] by Figure 8 For example, the training image can be processed into four training images to be restored with the scales of 64*64, 128*128, 256*256, and 512*512. The verification image can be processed into four verification images with the scales of 64*64, 128*128, 256*256, and 512*512.

[0146] Step 122: Obtain a second partial face image of the verification image at the Nth scale;

[0147] In the embodiment of the present disclosure, optionally, the first partial face image and the second partial face image are eye images.

[0148] In the embodiment of the present disclosure, the eye image in the verification image of the Nth scale may be directly captured as the second partial face image.

[0149] Step 123: inputting the N scales of the to-be-inpainted training images into the generator to be trained or the generator after the last training to obtain N scales of inpainted training images;

[0150] The specific processing method of the trained generator for the N-scale training images to be repaired can be found in Figure 3 and Figure 4 The processing method in the illustrated embodiment will not be described again.

[0151] by Figure 8 For example, four to-be-restored training images of scales 64*64, 128*128, 256*256, and 512*512 are input into the generator to be trained or the generator after the last training to obtain four to-be-restored training images of scales 64*64, 128*128, 256*256, and 512*512.

[0152] Step 124: Obtain a first facial partial image of the N-th scale restoration training image;

[0153] In the embodiment of the present disclosure, the eye image in the restoration training image of the Nth scale may be directly captured as the first partial face image.

[0154] Step 125: For each scale of the restored training image, set the restored training image of each scale to have a false value label, and input the restored training image with the false value label to the initial first-class discriminator or the first-class discriminator after the previous training to obtain a third identification result; set the verification image of each scale to have a true value label, and input each verification image with the true value label to the first-class discriminator to obtain a fourth identification result;

[0155] Step 126: Setting the first partial face image to have a false value label, and inputting the first partial face image with the false value label to the initial second-category discriminator or the second-category discriminator after the previous training, to obtain a fifth identification result; setting the second partial face image to have a true value label, and inputting the second partial face image with the true value label to the initial second-category discriminator or the second-category discriminator after the previous training, to obtain a sixth identification result;

[0156] Step 127: Calculate a third adversarial loss based on the third identification result and the fourth identification result; calculate a fourth adversarial loss based on the fifth identification result and the sixth identification result;

[0157] Step 128: Adjust the parameters of the first class discriminator according to the third adversarial loss to obtain an updated first class discriminator; adjust the parameters of the second class discriminator according to the fourth adversarial loss to obtain an updated second class discriminator.

[0158] In the disclosed embodiment, since the eyes are the most important component of the human face, the training effect can be improved by increasing the adversarial loss of the eye image.

[0159] In some embodiments of the present disclosure, optionally, the at least two discriminators further include: X third-category discriminators; X is a positive integer greater than or equal to 1, and the third-category discriminators are configured to improve the first generator's restoration of details of the facial part of the training image, that is, compared with other training methods, the human eye image in the facial image output by the first generator trained with the third-category discriminator is clearer and has more details.

[0160] Please refer to Figure 13 , training the generator to be trained further includes:

[0161] Step 131: Processing the training image into training images to be restored at N scales;

[0162] The specific method of processing the training image into N-scale training images to be restored can be found in the description of the above embodiment and will not be described again.

[0163] Step 132: Inputting the N scaled training images to be restored into the generator to be trained or the generator after the last training to obtain N scaled restoration training images;

[0164] The processing process of the to-be-trained generator on the to-be-repaired training images of N scales can be referred to the description in the above embodiment and will not be repeated here.

[0165] Step 133: Performing face parsing processing on the N-th scaled inpainted image using a face parsing network to obtain X first face part images corresponding to the N-th scaled inpainted image, where if X is equal to 1, the first face part images contain one face part; and if X is greater than 1, the X first face part images contain different face parts.

[0166] In the disclosed embodiment, the face parsing network adopts a semantic segmentation network.

[0167] In an embodiment of the present disclosure, the face parsing network parses the face, and the output facial parts may include at least one of the following: background, facial skin, left eyebrow, right eyebrow, left eye, right eye, left ear, right ear, nose, teeth, upper lip, lower lip, clothes, hair, hat, glasses, neck, etc.

[0168] Step 134: setting all the X first face part images to have true value labels, and inputting each first face part image with a true value label into the initial third-category discriminator or the third-category discriminator after the last training to obtain a seventh identification result;

[0169] Step 135: Calculate a fifth adversarial loss based on the seventh identification result; the total adversarial loss includes the fifth adversarial loss;

[0170] Step 136: Adjust the parameters of the generator to be trained or the generator after the last training according to the total adversarial loss.

[0171] Please refer to Figure 14 , training the at least two discriminators comprises:

[0172] Step 141: Processing the training image into training images to be restored at N scales, and processing the verification image into verification images at N scales;

[0173] Step 142: Inputting the N scaled training images to be restored into the generator to be trained or the generator after the last training to obtain N scaled restoration training images;

[0174] Step 143: performing face parsing processing on the N-th scaled restoration image using a face parsing network to obtain X first face part images corresponding to the N-th scaled restoration image, wherein the X first face part images include different face parts; performing face parsing processing on the N-th scaled verification image using a face parsing network to obtain X second face part images corresponding to the N-th scaled verification image, wherein the X second face part images include different face parts;

[0175] In the disclosed embodiment, the face parsing network adopts a semantic segmentation network.

[0176] In an embodiment of the present disclosure, the face parsing network parses the face, and the output facial parts may include at least one of the following: background, facial skin, left eyebrow, right eyebrow, left eye, right eye, left ear, right ear, nose, teeth, upper lip, lower lip, clothes, hair, hat, glasses, neck, etc.

[0177] Please refer to Figure 15 , Figure 15 In the embodiment shown, X is equal to 1, and the third type of discriminator is configured to improve the first generator's restoration of the details of the facial skin in the training image, that is, compared with other training methods, the skin image in the facial image output by the first generator trained with the third type of discriminator is clearer and has more details.

[0178] Step 144: All of the X first facial part images are set to have false value labels, and the first facial part images with false value labels are input into the initial third-category discriminator or the last trained third-category discriminator to obtain an eighth identification result; all of the X second facial part images are set to have true value labels, and each second facial part image with a true value label is input into the initial third-category discriminator or the last trained third-category discriminator to obtain a ninth identification result;

[0179] Step 145: Calculating a sixth adversarial loss based on the eighth identification result and the ninth identification result;

[0180] Step 146: Adjust the parameters of the third class discriminator according to the sixth adversarial loss to obtain an updated third class discriminator.

[0181] Please refer to Figure 16 , Figure 16 Schematic diagram of the input and output of the generator and discriminator to be trained in the embodiment of the present disclosure, from Figure 16As can be seen from the figure, the input of the generator to be trained includes training images of N scales, random noise images of N scales (or key point mask images of N scales), and the output of the generator to be trained is the repaired training image; the discriminator includes the N first-class discriminators corresponding to the repair modules of the N scales, and X third-class discriminators. The input of the discriminator includes: the repaired training image of the generator to be trained, the verification image of N scales, X facial part images corresponding to the verification image of the Nth scale, and X facial part images corresponding to the repaired training image of the Nth scale.

[0182] In the disclosed embodiment, the facial features, skin and / or hair are segmented and input into the discriminator to distinguish true from false, so that there is a process of confrontation when the training generator repairs each part of the face, thereby enhancing the generator's ability to generate various parts of the face, thereby obtaining richer details.

[0183] In some embodiments of the present disclosure, optionally, the total loss of the generator to be trained further includes: face similarity loss;

[0184] Please refer to Figure 17 , training the generator to be trained further includes:

[0185] Step 171: Processing the training image into training images to be restored at N scales;

[0186] Step 172: Inputting the N scaled training images to be restored into the generator to be trained or the generator after the last training to obtain N scaled restoration training images;

[0187] Step 172: Perform key point detection on the N-th scale restored image using a key point detection network to obtain a first key point heat map corresponding to the N-th scale restored image;

[0188] Step 173: using a key point detection network to perform key point detection on the N-th scale training image to be restored, to obtain a second key point heat map corresponding to the N-th scale training image to be restored;

[0189] Step 174: Calculate the face similarity loss based on the first key point heat map and the second key point heat map.

[0190] Please refer to Figure 8 , Figure 8 The key point detection module in is the key point detection network, heat map_1 is the first key point heat map, and heat map_2 is the second key point heat map.

[0191] In the embodiment of the present disclosure, it is optional, please refer to Figure 5A 4-stack hourglass model can be used to extract key points from the N-th scale training image to be repaired and the repair training image. For example, 68 key points are extracted from a face image to generate 68 key point heat maps, where each key point heat map represents the probability that all pixels on the image are a certain key point (landmark).

[0192] In some embodiments of the present disclosure, optionally, the total loss of the generator to be trained further includes: average gradient loss;

[0193] Please refer to Figure 18 , training the generator to be trained further includes:

[0194] Step 181: Processing the training image into training images to be restored at N scales;

[0195] Step 182: Inputting the N scales of the to-be-inpainted training images into the generator to be trained or the generator after the last training to obtain N scales of inpainted training images;

[0196] Step 183: Calculate the average gradient loss of the N-th size restoration training image.

[0197] In the embodiment of the present disclosure, optionally, the calculation formula of the average gradient loss AvgG is as follows:

[0198]

[0199] Among them, m and n are the width and height of the N-th size repair training image, respectively, and f i,j is the pixel at position (i, j) of the N-th size repair training image, Indicates that f in the row direction i,j The difference between adjacent pixels, Indicates that f i,j The difference between adjacent pixels.

[0200] In some embodiments of the present disclosure, optionally, the first generator includes N restoration modules, and the loss used by the generator to be trained includes a first loss; in this embodiment, the first loss can be called a perceptual loss;

[0201] Please refer to Figure 19 , training the generator to be trained further includes:

[0202] Step 191: Processing the training image into training images to be restored at N scales, and processing the verification image into verification images at N scales;

[0203] Step 192: Inputting the N scaled training images to be restored into the generator to be trained or the generator after the last training to obtain N scaled restoration training images;

[0204] Step 193: Input the N-scaled restoration training images and the N-scaled verification images into the VGG network to obtain the loss of the restoration training image of each scale on the M target layers of the VGG network, where M is an integer greater than or equal to 1; the first loss includes the loss of the N-scaled restoration training images on the M target layers.

[0205] Optionally, the first loss includes: the loss of each scale of the restoration training image on M target layers is multiplied by the corresponding weight and then added, wherein the weights used in the target layer for the restoration training images of different scales are different.

[0206] For example, the generator to be trained includes four scale restoration modules, namely 64*64, 128*128, 256*256, and 512*512. The VGG network is a VGG19 network, and the M target layers are 2-2 layers, 3-4 layers, 4-4 layers, and 5-4 layers, respectively. The first loss (i.e., perceptual loss) L is calculated as follows:

[0207] L=L per_64 +L per_128 +L per_256 +L per_512

[0208]

[0209] Among them, L per_64 is the perceptual loss of the 64*64 scale restoration training image, L per_128 is the perceptual loss of the 128*128 scale restoration training image, L per_256 is the perceptual loss of the 256*256 scale restoration training image, L per_512 is the perceptual loss of the 512*512 scale restoration training image, Perceptual loss at layer 2-2 for inpainting training images at different scales, Perceptual loss at layers 3-4 for inpainting training images at different scales, Perceptual loss at layer 4-4 for repair training images of different scales, Perceptual loss at layer 5-4 for inpainting training images at different scales.

[0210] In the above example, due to the different clarity concerns at different scales, smaller resolution scales focus more globally, corresponding to shallower VGG layers, while larger resolution scales focus more locally, corresponding to deeper VGG layers.

[0211] Of course, in some embodiments of the present disclosure, the weights used in the target layer for restoration training images of different scales may also be the same. For example:

[0212]

[0213] In the embodiment of the present disclosure, optionally, the first loss further includes at least one of the following: L1 loss, second loss, and third loss;

[0214] When the first loss includes L1 loss, the training of the generator to be trained includes:

[0215] Processing the training image into training images to be restored at N scales, and processing the verification image into verification images at N scales;

[0216] Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images;

[0217] Obtaining an L1 loss by comparing the N scaled restoration training images with the N scaled verification images;

[0218] When the first loss includes the second loss, the training of the generator to be trained includes:

[0219] Processing the training image into training images to be restored at N scales, and processing the verification image into verification images at N scales;

[0220] Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images;

[0221] Obtaining a first eye image of an N-th scale restoration training image and a second eye image of an N-th scale verification image;

[0222] Inputting the first eye image and the second eye image into a VGG network to obtain a second loss of the first eye image on M target layers of the VGG network, where M is an integer greater than or equal to 1;

[0223] When the first loss includes the third loss, the training of the generator to be trained includes:

[0224] Processing the training image into training images to be restored at N scales, and processing the verification image into verification images at N scales;

[0225] Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images;

[0226] Obtaining a first face skin image of the restored training image at an Nth scale and a second face skin image of the verification image at an Nth scale;

[0227] The first facial skin image and the second facial skin image are input into a VGG network to obtain a third loss of the first facial skin image on the M target layers of the VGG network.

[0228] Through the above-mentioned second loss and third loss, the details of the eye area and skin area of the output image can be better improved.

[0229] In some embodiments of the present disclosure, the at least two discriminators include: a fourth type of discriminator and a fifth type of discriminator; the fourth type of discriminator is configured to maintain the structural features of the training image by the first generator, and specifically the output image of the first generator can retain more content information of the input image; the fifth type of discriminator is configured to improve the detail restoration of the training image by the first generator, and specifically compared with other training methods, the output image processed by the first generator trained with the fifth type of discriminator has more detail features and higher clarity.

[0230] Please refer to Figure 20 , training the generator to be trained includes:

[0231] Step 201: Processing the training image into training images to be restored at N scales;

[0232] Step 202: inputting the N scales of the to-be-inpainted training images into the generator to be trained or the generator after the last training to obtain N scales of inpainted training images;

[0233] Step 203: For each scale of the inpainted training image, the inpainted training image of each scale is set to have a true value label, and the inpainted training image with the true value label is input into the initial fourth-category discriminator or the fourth-category discriminator after the last training to obtain a tenth identification result;

[0234] Step 204: Calculating a seventh adversarial loss based on the tenth identification result;

[0235] Step 205: For each scale of the restored training image, set the restored training image of each scale to have a true value label, and input the restored training image with the true value label into the initial fifth class discriminator or the fifth class discriminator after the last training to obtain an eleventh identification result;

[0236] Step 206: Calculate an eighth adversarial loss based on the eleventh identification result; the total adversarial loss includes the seventh adversarial loss and the eighth adversarial loss.

[0237] Step 207: Adjust parameters of the generator to be trained or the generator trained last time according to the total adversarial loss.

[0238] Please refer to Figure 21 , training the at least two discriminators comprises:

[0239] Step 211: Processing the training image into training images to be restored at N scales; processing the verification image into verification images at N scales;

[0240] Step 212: inputting the N scales of the to-be-inpainted training images into the generator to be trained or the generator after the last training to obtain N scales of inpainted training images;

[0241] Step 213: For each scale of the inpainted training image, set the inpainted training image of each scale to have a false value label, and input the inpainted training image with the false value label to the initial fourth class discriminator or the fourth class discriminator after the last training, to obtain a twelfth identification result; for each scale of the to-be-inpainted training image, set the to-be-inpainted training image to have a true value label, and input the to-be-inpainted training image with the true value label to the initial fourth class discriminator or the fourth class discriminator after the last training, to obtain a thirteenth identification result;

[0242] Step 214: Calculating a ninth adversarial loss based on the twelfth identification result and the third identification result;

[0243] Step 215: Adjust the parameters of the fourth class discriminator according to the ninth adversarial loss to obtain an updated fourth class discriminator.

[0244] Step 216: For each scale of the restored training image, perform high-frequency filtering on the restored training image and the verification image of the corresponding scale to obtain a high-frequency filtered restored training image and verification image.

[0245] Step 217: For each scale of the high-frequency filtered restoration training image, the high-frequency filtered restoration training image is set to have a false value label, and the high-frequency filtered restoration training image with the false value label is input into the initial fifth class discriminator or the fifth class discriminator after the last training, to obtain a fourteenth identification result; for each scale of the Gaussian filtered verification image, the Gaussian filtered verification image is set to have a true value label, and the Gaussian filtered verification image with the true value label is input into the initial fifth class discriminator or the fifth class discriminator after the last training, to obtain a fifteenth identification result;

[0246] Step 218: Calculating a tenth adversarial loss based on the fourteenth identification result and the fifteenth identification result;

[0247] Step 219: Adjust the parameters of the fifth class discriminator according to the tenth adversarial loss to obtain an updated fifth class discriminator.

[0248] Please refer to Figure 22 , Figure 22 This is a schematic diagram of the input and output of the generator and discriminator to be trained according to another embodiment of the present disclosure, Figure 22 As can be seen from the figure, the input of the generator to be trained includes training images of N scales and random noise images of N scales (or key point mask images of N scales), and the output of the generator to be trained is the repaired training image. The fourth type of discriminator includes the N first type of discriminators corresponding to the above-mentioned repair modules of N scales, and the input of the fourth type of discriminator includes: the repaired training image of the generator to be trained and the training image of N scales. The fifth type of discriminator includes the N first type of discriminators corresponding to the above-mentioned repair modules of N scales, and the input of the fifth type of discriminator includes: the image of the repaired training image of the generator to be trained after high-frequency filtering and the image of the verification image of N scales after high-frequency filtering.

[0249] In the embodiment of the present disclosure, the verification image may be an image with the same content as the training image but a different definition, or may be an image with different content and a different definition than the training image.

[0250] In the above embodiment, two types of discriminators (the fourth type and the fifth type) are designed. The reason for this design is that the detailed texture is the high-frequency information in the image, and the high-frequency information in natural images has the characteristic of obeying a certain specific distribution. The fifth type of discriminator and the generator are trained against each other, so that the generator learns the distribution obeyed by the detailed texture, thereby being able to map the smooth low-definition image to the real natural image space with rich details. The fourth type of discriminator distinguishes the low-definition image from the corresponding restoration result, which can constrain the image to maintain its structural characteristics after passing through the generator and not deform.

[0251] In the embodiment of the present disclosure, optionally, the loss function of the fifth type of discriminator is as follows:

[0252] maxV(D1,G)=log[D1(HF(y))]+log[1-D1(HF(G(x))]

[0253] The loss function of the fourth class discriminator is as follows:

[0254] maxV(D2,G)=log[D2(x)]+log[1-D2(G(x))]

[0255] Among them, G represents the generator, D1 and D2 represent the fifth and fourth class discriminators respectively, HF represents the Gaussian high-frequency filter, x represents the training image input to the generator, and y represents the real high-definition verification image.

[0256] In the embodiment of the present disclosure, the total loss of the generator to be trained also includes: average gradient loss; that is, the total loss of the generator to be trained = the loss of the fourth discriminator + the loss of the fifth discriminator + the average gradient loss;

[0257] At this time, training the generator to be trained further includes:

[0258] Processing the training image into training images to be restored at N scales;

[0259] Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images;

[0260] Calculate the average gradient loss of the N-th size restoration training image.

[0261] That is, the loss function of the generator is as follows:

[0262] minV(D,G)=αlog[1-D1(G(x))]+βlog[1-D2(x)]+γAvgG(G(x))

[0263] Here, α, β, and γ represent the weights of each loss, and AvgG represents the average gradient loss. The average gradient can be used to evaluate the richness of detail texture in an image. The richer the image details, the faster the grayscale value changes in a certain direction, and the larger the average gradient value.

[0264] Optionally, the average gradient loss AvgG is calculated as follows:

[0265]

[0266] Among them, m and n are the width and height of the N-th size repair training image, respectively, and f i,j is the pixel at position (i, j) of the N-th size repair training image.

[0267] In some other embodiments of the present disclosure, the first generator includes N repair modules, and the at least two discriminators include: N first-type discriminators with different network structures corresponding to the N repair modules respectively;

[0268] Among them, please refer to Figure 23 , training the generator to be trained includes:

[0269] Step 231: Processing the training image into training images to be restored at N scales;

[0270] Step 232: For each scale of the training image to be repaired, extract key points from the training image to be repaired, generate multiple key point heat maps, merge and classify the key point heat maps, and obtain S key point mask images for each scale, where S is an integer greater than or equal to 2;

[0271] Step 233: inputting the N scales of the to-be-inpainted training images and the S keypoint mask images of each scale into the generator to be trained or the generator after the last training to obtain N scales of inpainted training images;

[0272] Step 234: For each scale of the restored training image, set the restored training image of each scale to have a true value label, and input the restored training image with the true value label into the initial first-class discriminator or the first-class discriminator after the previous training to obtain a first identification result;

[0273] Step 235: Calculate a first adversarial loss based on the first identification result; the total adversarial loss includes the first adversarial loss;

[0274] Step 236: Adjust parameters of the generator to be trained or the generator trained last time according to the total adversarial loss;

[0275] Please refer to Figure 24 , training the at least two discriminators comprises:

[0276] Step 241: Processing the training image into training images to be restored at N scales, and processing the verification image into verification images at N scales;

[0277] Step 242: For each scale of the training image to be repaired, extract key points from the training image to be repaired, generate multiple key point heat maps, merge and classify the key point heat maps, and obtain S key point mask images for each scale;

[0278] Step 243: inputting the N scales of the to-be-inpainted training images and the S keypoint mask images of each scale into the generator to be trained or the generator after the last training to obtain N scales of inpainted training images;

[0279] Step 244: For each scale of the inpainted training image, set the inpainted training image of each scale to have a false value label, and input the inpainted training image with the false value label to the initial first-class discriminator or the first-class discriminator after the previous training to obtain a third identification result; set the verification image of each scale to have a true value label, and input each verification image with the true value label to the first-class discriminator to obtain a fourth identification result;

[0280] Step 245: Calculating a third adversarial loss based on the third identification result and the fourth identification result;

[0281] Step 246: Adjust the parameters of the first class discriminator according to the third adversarial loss to obtain an updated first class discriminator.

[0282] In the embodiment of the present disclosure, optionally, the first generator includes N repair modules; the total loss of the generator to be trained = the loss of the first class discriminator + the first loss (perceptual loss);

[0283] At this time, training the generator to be trained includes:

[0284] Processing the training image into training images to be restored at N scales, and processing the verification image into verification images at N scales;

[0285] Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images;

[0286] Inputting the N-scale restoration training images and the N-scale verification images into the VGG network, and obtaining the loss of the restoration training image of each scale on the M target layers of the VGG network, where M is an integer greater than or equal to 1;

[0287] The first loss includes the loss of the N-scale restoration training images on M target layers.

[0288] Optionally, the first loss includes: the loss of each scale of the restoration training image on M target layers is multiplied by the corresponding weight and then added, wherein the weights used in the target layer for the restoration training images of different scales are different.

[0289] For example, the generator to be trained includes four scale restoration modules, namely 64*64, 128*128, 256*256, and 512*512. The VGG network is a VGG19 network, and the M target layers are 2-2 layers, 3-4 layers, 4-4 layers, and 5-4 layers, respectively. The first loss (i.e., perceptual loss) L is calculated as follows:

[0290] L=L per_64 +L per_128 +L per_256 +L per_512

[0291]

[0292]

[0293] Among them, L per_64 is the perceptual loss of the 64*64 scale restoration training image, L per_128 is the perceptual loss of the 128*128 scale restoration training image, L per_256 is the perceptual loss of the 256*256 scale restoration training image, L per_512 is the perceptual loss of the 512*512 scale restoration training image, Perceptual loss at layer 2-2 for inpainting training images at different scales, Perceptual loss at layers 3-4 for inpainting training images at different scales, Perceptual loss at layer 4-4 for repair training images of different scales, Perceptual loss at layer 5-4 for inpainting training images at different scales.

[0294] In the above example, due to the different clarity concerns at different scales, smaller resolution scales focus more globally, corresponding to shallower VGG layers, while larger resolution scales focus more locally, corresponding to deeper VGG layers.

[0295] Optionally, the loss used by the generator to be trained further includes: a pixel-by-pixel two-normal (L2) loss. That is, the total loss of the generator to be trained = the loss of the first class discriminator + the first loss (perceptual loss) + the pixel-by-pixel two-normal loss.

[0296] Among them, the calculation method of L2 loss is as follows: processing the training image into N-scaled training images to be repaired, and processing the verification image into N-scaled verification images; inputting the N-scaled training images to be repaired into the generator to be trained or the generator after the last training to obtain N-scaled repair training images; and obtaining the L2 loss by comparing the N-scaled repair training images and the N-scaled verification images.

[0297] In the embodiment of the present disclosure, optionally, the first generator includes N repair modules, each of which adopts the same network structure;

[0298] The training process of the generator to be trained includes a first training stage and a second training stage; the first training stage and the second training stage both include at least one training process of the generator to be trained;

[0299] In the first training stage, when adjusting parameters of each of the repair modules, all repair modules share the same parameters;

[0300] In the second training stage, each of the repair modules is independently adjusted.

[0301] Optionally, the learning rate used in the first training stage (for example, learning rate lr = 0.0001) is greater than the learning rate used in the second training stage (for example, learning rate lr = 0.00005). The larger the learning rate, the faster the training speed. Since the first training stage needs to quickly train the shared parameters, a larger learning rate is used, and the second training stage is a more refined training, so a smaller learning rate is used to fine-tune each restoration module. This is because the restoration module focuses on the structural information of the face at a lower scale and the detailed information of the face at a higher scale. After the first training stage, the shared parameters are decoupled so that the super-resolution module at each scale can pay more attention to the information at that scale, thereby achieving better detail restoration effects.

[0302] Please refer to Figure 25 , the embodiment of the present disclosure further provides an image processing method, including:

[0303] Step 251: receiving an input image;

[0304] Step 252: performing face detection on the input image to obtain a face image;

[0305] In an embodiment of the present disclosure, optionally performing face detection on the input image to obtain a face image includes: performing face detection on the input image to obtain a detection image, and performing standardized alignment on the detection image to obtain the face image.

[0306] Step 253: Processing the facial image using the method in any of the above embodiments to obtain a first restoration training image, wherein the clarity of the first restoration training image is higher than the clarity of the input image;

[0307] Step 254: Processing the input image or the input image with the face image removed to obtain a second restoration training image, wherein the second restoration training image has a higher definition than the input image.

[0308] Step 255: Fusing the first restoration training image and the second restoration training image to obtain a fused image, wherein the clarity of the fused image is higher than that of the input image.

[0309] In an embodiment of the present disclosure, optionally, processing the input image or the input image with the facial image removed to obtain a second restoration training image includes: using the method described in any of the above embodiments to process the input image or the input image with the facial image removed to obtain a second restoration training image.

[0310] Please refer to Figure 26 , this embodiment of the application further provides an image processing device 260, including:

[0311] Receiving module 261, configured to receive an input image;

[0312] The processing module 262 is used to process the input image using a first generator to obtain an output image, wherein the clarity of the output image is higher than the clarity of the input image; wherein the first generator is obtained by training the generator to be trained using at least two discriminators.

[0313] Optionally, the first generator includes N repair modules, where N is an integer greater than or equal to 2;

[0314] The processing module is configured to process the input image into images to be restored at N scales, wherein the scales of the image to be restored at a first scale to the image to be restored at an Nth scale increase in sequence; and obtain the output image using the N restoration modules and the images to be restored at N scales.

[0315] Optionally, for two adjacent scales in the N scales, the latter scale is twice the former scale.

[0316] Optionally, the processing module is used to determine the scale interval to which the input image belongs; process the input image into an image to be restored of the jth scale corresponding to the scale interval to which it belongs, where the jth scale is one of the first scale to the Nth scale; and upsample and / or downsample the image to be restored of the jth scale to obtain images to be restored of the remaining N-1 scales.

[0317] Optionally, the processing module is used to:

[0318] splicing the image to be repaired at a first scale and the random noise image at a first scale to obtain a first spliced image; inputting the first spliced image into a first repair module to obtain a repaired image at a first scale; and upsampling the repaired image at the first scale to obtain an upsampled image at a second scale;

[0319] splicing the i-th scale upsampled image, the i-th scale to be restored image, and the i-th scale random noise image to obtain an i-th spliced image; inputting the i-th spliced image into the i-th restoration module to obtain an i-th scale restored image; upsampling the i-th scale restored image to obtain an i+1-th scale upsampled image, where i is an integer greater than or equal to 2;

[0320] The upsampled image of the Nth scale, the image to be repaired of the Nth size, and the random noise image of the Nth scale are spliced together to obtain an Nth spliced image; the Nth spliced image is input into the Nth repair module to obtain an Nth scale repaired image as the output image of the first generator.

[0321] Optional,

[0322] The processing module is used to:

[0323] For each scale of the image to be repaired, extract the key points in the image to be repaired, generate multiple key point heat maps, merge and classify the key point heat maps, and obtain S key point mask images for each scale, where S is an integer greater than or equal to 2;

[0324] Splicing the image to be repaired at a first scale and S key point mask images at the first scale to obtain a first spliced image; inputting the first spliced image into a first repair module to obtain a repaired image at a first scale; and upsampling the repaired image at the first scale to obtain an upsampled image at a second scale;

[0325] splicing the i-th scale upsampled image, the i-th scale image to be inpainted, and the S i-th scale keypoint mask images to obtain an i-th spliced image; inputting the i-th spliced image into the i-th inpainting module to obtain an i-th scale inpainted image; upsampling the i-th scale inpainted image to obtain an i+1-th scale upsampled image; wherein i is an integer greater than or equal to 2;

[0326] The upsampled image of the Nth scale, the image to be repaired of the Nth scale, and S key point mask images of the Nth scale are spliced to obtain an Nth spliced image; the Nth spliced image is input into the Nth repair module to obtain an Nth scale repaired image as the output image of the first generator.

[0327] Optionally, a 4-stack hourglass model is used to extract key points in the image to be repaired.

[0328] Optionally, the device further includes:

[0329] A training module is used to alternately train the generator to be trained and the at least two discriminators based on training images and verification images to obtain the first generator, wherein the clarity of the verification image is higher than the clarity of the training image, and when training the generator to be trained, the total loss of the generator to be trained includes at least one of the following: the first loss and the total adversarial loss of the at least two discriminators.

[0330] Optionally, the first generator includes N restoration modules, where N is an integer greater than or equal to 2, and the at least two discriminators include: N first-type discriminators with different network structures corresponding to the N restoration modules, and a second-type discriminator, wherein the second-type discriminator is configured to improve the clarity of the local facial restoration of the training image by the first generator.

[0331] The training module includes a first training submodule;

[0332] The first training submodule is used to train the generator to be trained, including:

[0333] Processing the training image into training images to be restored at N scales;

[0334] Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images;

[0335] Obtain the first partial face image of the N-th scale restoration training image;

[0336] For each scale of the restored training image, setting the restored training image of each scale to have a true value label, and inputting the restored training image with the true value label into an initial first-class discriminator or a first-class discriminator after last training to obtain a first identification result;

[0337] Setting the first partial face image to have a true value label, and inputting the first partial face image with the true value label into an initial second-category discriminator or a second-category discriminator after last training to obtain a second identification result;

[0338] Calculating a first adversarial loss based on the first identification result; calculating a second adversarial loss based on the second identification result, wherein the total adversarial loss includes the first adversarial loss and the second adversarial loss;

[0339] Adjusting parameters of the generator to be trained or the generator trained last time according to the total adversarial loss;

[0340] The first training submodule is used to train the at least two discriminators, including:

[0341] Processing the training image into training images to be restored at N scales, and processing the verification image into verification images at N scales;

[0342] Obtaining a second partial face image of the verification image at the Nth scale;

[0343] Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images;

[0344] Obtain the first partial face image of the N-th scale restoration training image;

[0345] For each scale of the restored training image, the restored training image is set to have a false value label, and the restored training image with the false value label is input into the initial first-class discriminator or the first-class discriminator after the last training to obtain a third identification result; the verification image of each scale is set to have a true value label, and each verification image with the true value label is input into the first-class discriminator to obtain a fourth identification result;

[0346] Setting the first partial face image to have a false value label, and inputting the first partial face image with the false value label into an initial second-category discriminator or a second-category discriminator after last training, to obtain a fifth identification result; setting the second partial face image to have a true value label, and inputting the second partial face image with the true value label into the initial second-category discriminator or a second-category discriminator after last training, to obtain a sixth identification result;

[0347] Calculating a third adversarial loss based on the third identification result and the fourth identification result; calculating a fourth adversarial loss based on the fifth identification result and the sixth identification result;

[0348] The parameters of the first class discriminator are adjusted according to the third adversarial loss to obtain an updated first class discriminator; and the parameters of the second class discriminator are adjusted according to the fourth adversarial loss to obtain an updated second class discriminator.

[0349] Optionally, the first partial face image and the second partial face image are eye images.

[0350] Optionally, the at least two discriminators further include: X third-category discriminators; X is a positive integer greater than or equal to 1, and the third-category discriminators are configured to enhance the detail restoration of the facial part of the training image by the first generator.

[0351] Optionally, the first training submodule is used to train the generator to be trained, including:

[0352] Processing the training image into training images to be restored at N scales;

[0353] Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images;

[0354] Performing face parsing processing on the N-th scaled inpainted image using a face parsing network to obtain X first face part images corresponding to the N-th scaled inpainted image, wherein if X is equal to 1, the first face part images include one face part, and if X is greater than 1, the X first face part images include different face parts;

[0355] Setting the X first face part images as having true value labels, and inputting each first face part image having a true value label into an initial third-category discriminator or a third-category discriminator after last training to obtain a seventh identification result;

[0356] Calculating a fifth adversarial loss based on the seventh identification result; the total adversarial loss includes the fifth adversarial loss;

[0357] The first training submodule is used to train the at least two discriminators, including:

[0358] Processing the training image into training images to be restored at N scales, and processing the verification image into verification images at N scales;

[0359] Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images;

[0360] Performing face parsing processing on the N-th scaled restoration image using a face parsing network to obtain X first face part images corresponding to the N-th scaled restoration image, wherein the X first face part images include different face parts; performing face parsing processing on the N-th scaled verification image using a face parsing network to obtain X second face part images corresponding to the N-th scaled verification image, wherein the X second face part images include different face parts;

[0361] Setting the X first facial part images to have false value labels, and inputting the first facial part images with false value labels into an initial third-category discriminator or a third-category discriminator after last training, to obtain an eighth identification result; setting the X second facial part images to have true value labels, and inputting each second facial part image with a true value label into the initial third-category discriminator or a third-category discriminator after last training, to obtain a ninth identification result;

[0362] Calculating a sixth adversarial loss based on the eighth identification result and the ninth identification result;

[0363] Adjust parameters of the third class discriminator according to the sixth adversarial loss to obtain an updated third class discriminator.

[0364] Optionally, the face parsing network adopts a semantic segmentation network.

[0365] Optionally, X is equal to 1, and the third type of discriminator is configured to improve the first generator's restoration of facial skin details in the training image.

[0366] Optionally, the total loss of the generator to be trained further includes: face similarity loss;

[0367] The first training submodule is used to train the generator to be trained, including:

[0368] Processing the training image into training images to be restored at N scales;

[0369] Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images;

[0370] Performing key point detection on the N-th scale restored image using a key point detection network to obtain a first key point heat map corresponding to the N-th scale restored image;

[0371] Performing key point detection on the N-th scale training image to be restored using a key point detection network to obtain a second key point heat map corresponding to the N-th scale training image to be restored;

[0372] Calculate the face similarity loss based on the first key point heat map and the second key point heat map.

[0373] Optionally, the total loss of the generator to be trained further includes: average gradient loss;

[0374] The first training submodule is used to train the generator to be trained, including:

[0375] Processing the training image into training images to be restored at N scales;

[0376] Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images;

[0377] Calculate the average gradient loss of the N-th size restoration training image.

[0378] Optionally, the first generator includes N repair modules, where N is an integer greater than or equal to 2, and each of the repair modules adopts the same network structure;

[0379] The training process of the generator to be trained includes a first training stage and a second training stage, and the first training stage and the second training stage both include at least one training process of the generator to be trained;

[0380] In the first training stage, when adjusting parameters of each of the repair modules, all repair modules share the same parameters;

[0381] In the second training stage, each of the repair modules is independently adjusted.

[0382] Optionally, the learning rate used in the first training stage is greater than the learning rate used in the second training stage.

[0383] Optionally, the at least two discriminators include: a fourth category discriminator and a fifth category discriminator; the fourth category discriminator is configured to maintain the structural features of the training image by the first generator; and the fifth category discriminator is configured to enhance the detail restoration of the training image by the first generator.

[0384] Optionally, the training module further includes a second training submodule;

[0385] The second training submodule is used to train the generator to be trained, including:

[0386] Processing the training image into training images to be restored at N scales;

[0387] Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images;

[0388] For each scale of the restored training image, setting the restored training image of each scale to have a true value label, and inputting the restored training image with the true value label into an initial fourth-category discriminator or a fourth-category discriminator after last training to obtain a tenth identification result;

[0389] Calculating a seventh adversarial loss based on the tenth identification result;

[0390] For each scale of the restored training image, setting the restored training image of each scale to have a true value label, and inputting the restored training image with the true value label into the initial fifth class discriminator or the fifth class discriminator after the last training to obtain an eleventh identification result;

[0391] Calculating an eighth adversarial loss based on the eleventh identification result;

[0392] The total confrontation loss includes the seventh confrontation loss and the eighth confrontation loss;

[0393] Adjusting parameters of the generator to be trained or the generator trained last time according to the total adversarial loss;

[0394] The second training submodule is used to train the at least two discriminators, including:

[0395] Processing the training image into training images to be restored at N scales; processing the verification image into verification images at N scales;

[0396] Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images;

[0397] For each scale of the inpainted training image, the inpainted training image of each scale is set to have a false value label, and the inpainted training image with the false value label is input into the initial fourth class discriminator or the fourth class discriminator after the last training, so as to obtain a twelfth identification result; for each scale of the to-be-inpainted training image, the to-be-inpainted training image is set to have a true value label, and the to-be-inpainted training image with the true value label is input into the initial fourth class discriminator or the fourth class discriminator after the last training, so as to obtain a thirteenth identification result;

[0398] Calculating a ninth adversarial loss based on the twelfth identification result and the third identification result;

[0399] adjusting the parameters of the fourth class discriminator according to the ninth adversarial loss to obtain an updated fourth class discriminator; performing high-frequency filtering on the restored training image at each scale and the verification image at the corresponding scale to obtain a high-frequency filtered restored training image and verification image;

[0400] For each scale of the high-frequency filtered restoration training image, the high-frequency filtered restoration training image is set to have a false value label, and the high-frequency filtered restoration training image with the false value label is input into the initial fifth class discriminator or the fifth class discriminator after the previous training, to obtain a fourteenth identification result; for each scale of the Gaussian filtered verification image, the Gaussian filtered verification image is set to have a true value label, and the Gaussian filtered verification image with the true value label is input into the initial fifth class discriminator or the fifth class discriminator after the previous training, to obtain a fifteenth identification result;

[0401] Calculating a tenth adversarial loss based on the fourteenth identification result and the fifteenth identification result;

[0402] The parameters of the fifth class discriminator are adjusted according to the tenth adversarial loss to obtain an updated fifth class discriminator.

[0403] Optionally, the total loss of the generator to be trained further includes: average gradient loss;

[0404] The second training submodule is used to train the generator to be trained, including:

[0405] Processing the training image into training images to be restored at N scales;

[0406] Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images;

[0407] Calculate the average gradient loss of the N-th size restoration training image.

[0408] Optionally, the average gradient loss AvgG is calculated as follows:

[0409]

[0410] Among them, m and n are the width and height of the N-th size repair training image, respectively, and f i,j is the pixel at position (i, j) of the N-th size repair training image.

[0411] Optionally, the first generator includes N repair modules, and the at least two discriminators include: N first-type discriminators with different network structures corresponding to the N repair modules respectively;

[0412] The training module also includes a third training module;

[0413] The third training submodule is used to train the generator to be trained and includes:

[0414] Processing the training image into training images to be restored at N scales;

[0415] For each scale of the training image to be repaired, extract the key points in the training image to be repaired, generate multiple key point heat maps, merge and classify the key point heat maps, and obtain S key point mask images for each scale, where S is an integer greater than or equal to 2;

[0416] Inputting the N scales of the to-be-inpainted training images and the S key point mask images of each scale into the generator to be trained or the generator after the last training to obtain N scales of inpainted training images;

[0417] For each scale of the restored training image, setting the restored training image of each scale to have a true value label, and inputting the restored training image with the true value label into an initial first-class discriminator or a first-class discriminator after last training to obtain a first identification result;

[0418] Calculating a first adversarial loss based on the first identification result; the total adversarial loss includes the first adversarial loss;

[0419] Adjusting parameters of the generator to be trained or the generator trained last time according to the total adversarial loss;

[0420] The third training submodule is used to train the at least two discriminators, including:

[0421] Processing the training image into training images to be restored at N scales, and processing the verification image into verification images at N scales;

[0422] For each scale of the training image to be repaired, extract the key points in the training image to be repaired, generate multiple key point heat maps, merge and classify the key point heat maps, and obtain S key point mask images for each scale;

[0423] Inputting the N scales of the to-be-inpainted training images and the S key point mask images of each scale into the generator to be trained or the generator after the last training to obtain N scales of inpainted training images;

[0424] For each scale of the restored training image, the restored training image is set to have a false value label, and the restored training image with the false value label is input into the initial first-class discriminator or the first-class discriminator after the last training to obtain a third identification result; the verification image of each scale is set to have a true value label, and each verification image with the true value label is input into the first-class discriminator to obtain a fourth identification result;

[0425] Calculating a third adversarial loss based on the third identification result and the fourth identification result;

[0426] The parameters of the first class discriminator are adjusted according to the third adversarial loss to obtain an updated first class discriminator.

[0427] Optionally, the first generator includes N repair modules;

[0428] The third training submodule is used to train the generator to be trained, including:

[0429] Processing the training image into training images to be restored at N scales, and processing the verification image into verification images at N scales;

[0430] Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images;

[0431] Inputting the N-scale restoration training images and the N-scale verification images into the VGG network, and obtaining the loss of the restoration training image of each scale on the M target layers of the VGG network, where M is an integer greater than or equal to 1;

[0432] The first loss includes the loss of the N-scale restoration training images on M target layers.

[0433] Optionally, the first loss includes: the loss of each scale of the restoration training image on M target layers is multiplied by the corresponding weight and then added, wherein the weights used in the target layer for the restoration training images of different scales are different.

[0434] Optionally, the first loss also includes: pixel-by-pixel two-paradigm loss.

[0435] Optionally, the first generator includes four scale restoration modules, namely: a 64*64 scale restoration module, a 128*128 scale restoration module, a 256*256 scale restoration module, and a 512*512 scale restoration module.

[0436] Optionally, S is equal to 5, and the S key point mask images include: key point mask images of the left eye, the right eye, the nose, the mouth, and the contour.

[0437] Please refer to Figure 27 , the embodiment of the present disclosure further provides an image processing device, comprising:

[0438] Receiving module 271, configured to receive an input image;

[0439] A face detection module 272 is configured to perform face detection on the input image to obtain a face image;

[0440] a first processing module, configured to process the facial image using the image processing method described in any one of the above embodiments to obtain a first restoration training image, wherein the clarity of the first restoration training image is higher than that of the input image;

[0441] A second processing module 273 is configured to process the input image or the input image with the face image removed to obtain a second restoration training image, wherein the second restoration training image has a higher definition than the input image;

[0442] The fusion module 274 is configured to fuse the first restoration training image and the second restoration training image to obtain a fused image, wherein the clarity of the fused image is higher than that of the input image.

[0443] Optionally, the second processing module 273 is configured to process the input image or the input image with the face image removed by using the image processing method described in any of the above embodiments to obtain a second restored training image.

[0444] An embodiment of the present disclosure also provides an electronic device, including a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the image processing method described in any of the above embodiments.

[0445] An embodiment of the present disclosure further provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the image processing method in any of the above embodiments are implemented.

[0446] The processor is the processor in the terminal described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0447] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0448] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0449] The embodiments of the present disclosure are described above in conjunction with the accompanying drawings, but the present disclosure is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present disclosure, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present disclosure and the claims, all of which are protected by the present disclosure.

Claims

1. An image processing method, wherein: include: receiving an input image; Processing the input image using a first generator to obtain an output image, wherein the clarity of the output image is higher than the clarity of the input image; The first generator is trained by using at least two discriminators to train the training generator; The first generator includes N restoration modules, where N is an integer greater than or equal to 2; processing the input image using the first generator to obtain an output image includes: processing the input image into images to be restored at N scales, where the scales of the image to be restored at a first scale to the image to be restored at an Nth scale increase in sequence; and obtaining the output image using the N restoration modules and the images to be restored at the N scales; The method of obtaining the output image using the N restoration modules and the N scales of the to-be-restored image comprises: splicing the to-be-restored image of a first scale and the random noise image of a first scale to obtain a first spliced image; inputting the first spliced image into a first restoration module to obtain a restored image of a first scale; and upsampling the restored image of the first scale to obtain an upsampled image of a second scale. splicing the i-th scale upsampled image, the i-th scale to be restored image, and the i-th scale random noise image to obtain an i-th spliced image; inputting the i-th spliced image into the i-th restoration module to obtain an i-th scale restored image; upsampling the i-th scale restored image to obtain an i+1-th scale upsampled image, where i is an integer greater than or equal to 2; The upsampled image of the Nth scale, the image to be repaired of the Nth size, and the random noise image of the Nth scale are spliced together to obtain an Nth spliced image; the Nth spliced image is input into the Nth repair module to obtain an Nth scale repaired image as the output image of the first generator.

2. The image processing method according to claim 1, wherein: For two adjacent scales among the N scales, the latter scale is twice the former scale.

3. The image processing method according to claim 1, wherein: Processing the input image into images to be restored at N scales includes: Determining the scale interval to which the input image belongs; Processing the input image into an image to be restored at a j-th scale corresponding to the scale interval to which the input image belongs, where the j-th scale is one of the first scale to the N-th scale; Upsampling and / or downsampling processing is performed on the image to be restored at the j-th scale to obtain images to be restored at the remaining N-1 scales.

4. The image processing method according to claim 1, wherein: Obtaining the output image using the N restoration modules and the N-scaled images to be restored includes: For each scale of the image to be repaired, extract the key points in the image to be repaired, generate multiple key point heat maps, merge and classify the key point heat maps, and obtain S key point mask images for each scale, where S is an integer greater than or equal to 2; Splicing the image to be repaired at a first scale and S key point mask images at the first scale to obtain a first spliced image; inputting the first spliced image into a first repair module to obtain a repaired image at a first scale; and upsampling the repaired image at the first scale to obtain an upsampled image at a second scale; splicing the i-th scale upsampled image, the i-th scale image to be inpainted, and S i-th scale keypoint mask images to obtain an i-th spliced image; inputting the i-th spliced image into the i-th inpainting module to obtain an i-th scale inpainted image; upsampling the i-th scale inpainted image to obtain an i+1-th scale upsampled image; wherein i is an integer greater than or equal to 2; The upsampled image of the Nth scale, the image to be repaired of the Nth scale, and S key point mask images of the Nth scale are spliced to obtain an Nth spliced image; the Nth spliced image is input into the Nth repair module to obtain an Nth scale repaired image as the output image of the first generator.

5. The method according to claim 4, wherein: A 4-stack hourglass model is used to extract key points in the image to be repaired.

6. The method of claim 1, wherein: The first generator is trained using at least two discriminators to train the generator and includes: The generator to be trained and the at least two discriminators are alternately trained according to training images and verification images to obtain the first generator, wherein the clarity of the verification image is higher than the clarity of the training image, and when training the generator to be trained, the total loss of the generator to be trained includes at least one of the following: the first loss and the total adversarial loss of the at least two discriminators.

7. The method according to claim 6, wherein: The first generator includes N restoration modules, where N is an integer greater than or equal to 2, and the at least two discriminators include: N first-type discriminators with different network structures corresponding to the N restoration modules, and a second-type discriminator; wherein the second-type discriminator is configured to improve the clarity of the local facial restoration of the training image by the first generator.

8. The method of claim 7, wherein: Training the generator to be trained includes: Processing the training image into training images to be restored at N scales; Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images; Obtain the first partial face image of the N-th scale restoration training image; For each scale of the restored training image, setting the restored training image of each scale to have a true value label, and inputting the restored training image with the true value label into an initial first-class discriminator or a first-class discriminator after last training to obtain a first identification result; Setting the first partial face image to have a true value label, and inputting the first partial face image with the true value label into an initial second-category discriminator or a second-category discriminator after last training to obtain a second identification result; Calculating a first adversarial loss based on the first identification result; calculating a second adversarial loss based on the second identification result, wherein the total adversarial loss includes the first adversarial loss and the second adversarial loss; Adjusting parameters of the generator to be trained or the generator trained last time according to the total adversarial loss; Training the at least two discriminators comprises: Processing the training image into training images to be restored at N scales, and processing the verification image into verification images at N scales; Obtaining a second partial face image of the verification image at the Nth scale; Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images; Obtain the first partial face image of the N-th scale restoration training image; For each scale of the restored training image, the restored training image is set to have a false value label, and the restored training image with the false value label is input into the initial first-class discriminator or the first-class discriminator after the last training to obtain a third identification result; the verification image of each scale is set to have a true value label, and each verification image with the true value label is input into the first-class discriminator to obtain a fourth identification result; Setting the first partial face image to have a false value label, and inputting the first partial face image with the false value label into an initial second-category discriminator or a second-category discriminator after last training, to obtain a fifth identification result; setting the second partial face image to have a true value label, and inputting the second partial face image with the true value label into the initial second-category discriminator or a second-category discriminator after last training, to obtain a sixth identification result; Calculating a third adversarial loss based on the third identification result and the fourth identification result; calculating a fourth adversarial loss based on the fifth identification result and the sixth identification result; The parameters of the first class discriminator are adjusted according to the third adversarial loss to obtain an updated first class discriminator; and the parameters of the second class discriminator are adjusted according to the fourth adversarial loss to obtain an updated second class discriminator.

9. The method of claim 8, wherein: The first partial face image and the second partial face image are eye images.

10. The method of claim 7, wherein: The at least two discriminators further include: X third-category discriminators; X is a positive integer greater than or equal to 1, and the third-category discriminators are configured to enhance the first generator's restoration of details of the facial part of the training image.

11. The method according to claim 10, wherein: Training the generator to be trained further includes: Processing the training image into training images to be restored at N scales; Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images; Performing face parsing processing on the N-th scaled inpainted image using a face parsing network to obtain X first face part images corresponding to the N-th scaled inpainted image, where if X is equal to 1, the first face part images contain one face part; and if X is greater than 1, the X first face part images contain different face parts. Setting the X first face part images as having true value labels, and inputting each first face part image having a true value label into an initial third-category discriminator or a third-category discriminator after last training to obtain a seventh identification result; Calculating a fifth adversarial loss based on the seventh identification result; the total adversarial loss includes the fifth adversarial loss; Training the at least two discriminators further comprises: Processing the training image into training images to be restored at N scales, and processing the verification image into verification images at N scales; Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images; Performing face parsing processing on the N-th scaled restoration image using a face parsing network to obtain X first face part images corresponding to the N-th scaled restoration image, wherein the X first face part images include different face parts; performing face parsing processing on the N-th scaled verification image using a face parsing network to obtain X second face part images corresponding to the N-th scaled verification image, wherein the X second face part images include different face parts; Setting the X first facial part images to have false value labels, and inputting the first facial part images with false value labels into an initial third-category discriminator or a third-category discriminator after last training, to obtain an eighth identification result; setting the X second facial part images to have true value labels, and inputting each second facial part image with a true value label into the initial third-category discriminator or a third-category discriminator after last training, to obtain a ninth identification result; Calculating a sixth adversarial loss based on the eighth identification result and the ninth identification result; Adjust parameters of the third class discriminator according to the sixth adversarial loss to obtain an updated third class discriminator.

12. The method according to claim 10 or 11, wherein: X is equal to 1, and the third type of discriminator is configured to improve the first generator's restoration of facial skin details in the training image.

13. The method of claim 11, wherein: The face parsing network adopts a semantic segmentation network.

14. The method of claim 7, wherein: The total loss of the generator to be trained also includes: face similarity loss; Training the generator to be trained further includes: Processing the training image into training images to be restored at N scales; Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images; Perform key point detection on the N-th scale restored image using a key point detection network to obtain a first key point heat map corresponding to the N-th scale restored image; Performing key point detection on the N-th scale training image to be restored using a key point detection network to obtain a second key point heat map corresponding to the N-th scale training image to be restored; Calculate the face similarity loss based on the first key point heat map and the second key point heat map.

15. The method of claim 7, wherein: The total loss of the generator to be trained also includes: average gradient loss; Training the generator to be trained further includes: Processing the training image into training images to be restored at N scales; Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images; Calculate the average gradient loss of the N-th size inpainted training image.

16. The method of claim 6, wherein: The first generator includes N repair modules, where N is an integer greater than or equal to 2, and each of the repair modules adopts the same network structure; The training process of the generator to be trained includes a first training stage and a second training stage, and the first training stage and the second training stage both include at least one training process of the generator to be trained; In the first training stage, when adjusting parameters of each of the repair modules, all the repair modules share the same parameters; In the second training stage, each of the repair modules is independently adjusted.

17. The method of claim 16, wherein: The learning rate used in the first training phase is greater than the learning rate used in the second training phase.

18. The method of claim 6, wherein: The at least two discriminators include: a fourth type of discriminator and a fifth type of discriminator; the fourth type of discriminator is configured to maintain the structural features of the training image produced by the first generator; and the fifth type of discriminator is configured to enhance the detail restoration of the training image produced by the first generator.

19. The method of claim 18, wherein: Training the generator to be trained includes: Processing the training image into training images to be restored at N scales; Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images; For each scale of the restored training image, setting the restored training image of each scale to have a true value label, and inputting the restored training image with the true value label into an initial fourth-category discriminator or a fourth-category discriminator after last training to obtain a tenth identification result; Calculating a seventh adversarial loss based on the tenth identification result; For each scale of the restored training image, setting the restored training image of each scale to have a true value label, and inputting the restored training image with the true value label into an initial fifth-category discriminator or a fifth-category discriminator after last training to obtain an eleventh identification result; Calculating an eighth adversarial loss based on the eleventh identification result; The total confrontation loss includes the seventh confrontation loss and the eighth confrontation loss; Adjusting parameters of the generator to be trained or the generator trained last time according to the total adversarial loss; Training the at least two discriminators comprises: Processing the training image into training images to be restored at N scales; processing the verification image into verification images at N scales; Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images; For each scale of the inpainted training image, the inpainted training image of each scale is set to have a false value label, and the inpainted training image with the false value label is input into the initial fourth class discriminator or the fourth class discriminator after the last training, so as to obtain a twelfth identification result; for each scale of the to-be-inpainted training image, the to-be-inpainted training image is set to have a true value label, and the to-be-inpainted training image with the true value label is input into the initial fourth class discriminator or the fourth class discriminator after the last training, so as to obtain a thirteenth identification result; Calculating a ninth adversarial loss based on the twelfth identification result and the third identification result; adjusting the parameters of the fourth class discriminator according to the ninth adversarial loss to obtain an updated fourth class discriminator; performing high-frequency filtering on the restored training image at each scale and the verification image at the corresponding scale to obtain a high-frequency filtered restored training image and verification image; For each scale of the high-frequency filtered restoration training image, the high-frequency filtered restoration training image is set to have a false value label, and the high-frequency filtered restoration training image with the false value label is input into the initial fifth class discriminator or the fifth class discriminator after the previous training, to obtain a fourteenth identification result; for each scale of the Gaussian filtered verification image, the Gaussian filtered verification image is set to have a true value label, and the Gaussian filtered verification image with the true value label is input into the initial fifth class discriminator or the fifth class discriminator after the previous training, to obtain a fifteenth identification result; Calculating a tenth adversarial loss based on the fourteenth identification result and the fifteenth identification result; The parameters of the fifth class discriminator are adjusted according to the tenth adversarial loss to obtain an updated fifth class discriminator.

20. The method of claim 18, wherein: The total loss of the generator to be trained also includes: average gradient loss; Training the generator to be trained further includes: Processing the training image into training images to be restored at N scales; Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images; Calculate the average gradient loss of the N-th size inpainted training image.

21. The method according to claim 15 or 20, wherein: The calculation formula of average gradient loss AvgG is as follows: Among them, m and n are the width and height of the N-th size repair training image, respectively, and f i,j is the pixel at position (i, j) of the N-th size repair training image.

22. The method of claim 6, wherein: The first generator includes N repair modules, where N is an integer greater than or equal to 2, and the at least two discriminators include: N first-type discriminators with different network structures corresponding to the N repair modules.

23. The method of claim 22, wherein: Training the generator to be trained includes: Processing the training image into training images to be restored at N scales; For each scale of the training image to be repaired, extract the key points in the training image to be repaired, generate multiple key point heat maps, merge and classify the key point heat maps, and obtain S key point mask images for each scale, where S is an integer greater than or equal to 2; Inputting the N scales of the to-be-inpainted training images and the S key point mask images of each scale into the generator to be trained or the generator after the last training to obtain N scales of inpainted training images; For each scale of the restored training image, setting the restored training image of each scale to have a true value label, and inputting the restored training image with the true value label into an initial first-class discriminator or a first-class discriminator after last training to obtain a first identification result; Calculating a first adversarial loss based on the first identification result; the total adversarial loss includes the first adversarial loss; Adjusting parameters of the generator to be trained or the generator trained last time according to the total adversarial loss; Training the at least two discriminators comprises: Processing the training image into training images to be restored at N scales, and processing the verification image into verification images at N scales; For each scale of the training image to be repaired, extract the key points in the training image to be repaired, generate multiple key point heat maps, merge and classify the key point heat maps, and obtain S key point mask images for each scale; Inputting the N scales of the to-be-inpainted training images and the S key point mask images of each scale into the generator to be trained or the generator after the last training to obtain N scales of inpainted training images; For each scale of the restored training image, the restored training image is set to have a false value label, and the restored training image with the false value label is input into the initial first-class discriminator or the first-class discriminator after the last training to obtain a third identification result; the verification image of each scale is set to have a true value label, and each verification image with the true value label is input into the first-class discriminator to obtain a fourth identification result; Calculating a third adversarial loss based on the third identification result and the fourth identification result; The parameters of the first class discriminator are adjusted according to the third adversarial loss to obtain an updated first class discriminator.

24. The method of claim 6 or 22, wherein: Training the generator to be trained includes: Processing the training image into training images to be restored at N scales, and processing the verification image into verification images at N scales; Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images; Inputting the N-scale restoration training images and the N-scale verification images into the VGG network, and obtaining the loss of the restoration training image of each scale on the M target layers of the VGG network, where M is an integer greater than or equal to 1; The first loss includes the loss of the N-scale restoration training images on M target layers.

25. The method of claim 24, wherein: The first loss includes: the loss of each scale of the restoration training image on M target layers is multiplied by the corresponding weight and then added, wherein the weights used in the target layer for the restoration training images of different scales are different.

26. The method of claim 22, wherein: The first loss includes: a pixel-by-pixel two-normal loss.

27. The method of claim 6, wherein: The first loss also includes at least one of the following: L1 loss, second loss and third loss; When the first loss includes L1 loss, the training of the generator to be trained includes: Processing the training image into training images to be restored at N scales, and processing the verification image into verification images at N scales; Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images; Obtaining an L1 loss by comparing the N scaled restoration training images with the N scaled verification images; When the first loss includes the second loss, the training of the generator to be trained includes: Processing the training image into training images to be restored at N scales, and processing the verification image into verification images at N scales; Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images; Obtaining a first eye image of an N-th scale restoration training image and a second eye image of an N-th scale verification image; Inputting the first eye image and the second eye image into a VGG network to obtain a second loss of the first eye image on M target layers of the VGG network, where M is an integer greater than or equal to 1; When the first loss includes the third loss, the training of the generator to be trained includes: Processing the training image into training images to be restored at N scales, and processing the verification image into verification images at N scales; Inputting the N scales of the to-be-restored training images into the generator to be trained or the generator after the last training to obtain N scales of the restored training images; Obtaining a first face skin image of the restored training image at an Nth scale and a second face skin image of the verification image at an Nth scale; The first facial skin image and the second facial skin image are input into a VGG network to obtain a third loss of the first facial skin image on the M target layers of the VGG network.

28. The method of claim 1, wherein The first generator includes four scale restoration modules, namely: a 64*64 scale restoration module, a 128*128 scale restoration module, a 256*256 scale restoration module, and a 512*512 scale restoration module.

29. The method of claim 4 or 23, wherein: S is equal to 5, and the S key point mask images include: key point mask images of the left eye, the right eye, the nose, the mouth, and the outline.

30. The method of claim 1, 4, 7, 16 or 22, wherein The network structure adopted by the repair module is SRCNN or U-Net.

31. An image processing method, wherein: include: receiving an input image; Performing face detection on the input image to obtain a face image; Processing the facial image using the method according to any one of claims 1 to 30 to obtain a first restoration training image, wherein the clarity of the first restoration training image is higher than the clarity of the input image; Processing the input image or the input image with the face image removed to obtain a second restoration training image, wherein the clarity of the second restoration training image is higher than the clarity of the input image; The first restoration training image and the second restoration training image are fused to obtain a fused image, wherein the clarity of the fused image is higher than the clarity of the input image.

32. The method of claim 31, wherein Processing the input image or the input image with the face image removed to obtain a second restoration training image includes: The input image or the input image with the face image removed is processed using the method according to any one of claims 1 to 30 to obtain a second repair training image.

33. An image processing device, wherein: include: A receiving module, configured to receive an input image; a processing module, configured to process the input image using a first generator to obtain an output image, wherein the clarity of the output image is higher than that of the input image; The first generator is trained by using at least two discriminators to train the training generator; The first generator includes N restoration modules, where N is an integer greater than or equal to 2; processing the input image using the first generator to obtain an output image includes: processing the input image into images to be restored at N scales, where the scales of the image to be restored at a first scale to the image to be restored at an Nth scale increase in sequence; and obtaining the output image using the N restoration modules and the images to be restored at the N scales; The method of obtaining the output image using the N restoration modules and the N scales of the to-be-restored image comprises: splicing the to-be-restored image of a first scale and the random noise image of a first scale to obtain a first spliced image; inputting the first spliced image into a first restoration module to obtain a restored image of a first scale; and upsampling the restored image of the first scale to obtain an upsampled image of a second scale. splicing the i-th scale upsampled image, the i-th scale to be restored image, and the i-th scale random noise image to obtain an i-th spliced image; inputting the i-th spliced image into the i-th restoration module to obtain an i-th scale restored image; upsampling the i-th scale restored image to obtain an i+1-th scale upsampled image, where i is an integer greater than or equal to 2; The upsampled image of the Nth scale, the image to be repaired of the Nth size, and the random noise image of the Nth scale are spliced together to obtain an Nth spliced image; the Nth spliced image is input into the Nth repair module to obtain an Nth scale repaired image as the output image of the first generator.

34. An image processing device, wherein: include: A receiving module, configured to receive an input image; A face detection module, configured to perform face detection on the input image to obtain a face image; a first processing module, configured to process the facial image using the method according to any one of claims 1 to 30 to obtain a first restoration training image, wherein the clarity of the first restoration training image is higher than that of the input image; a second processing module, configured to process the input image or the input image with the face image removed to obtain a second restoration training image, wherein the clarity of the second restoration training image is higher than that of the input image; The first restoration training image and the second restoration training image are fused to obtain a fused image, wherein the clarity of the fused image is higher than the clarity of the input image.

35. An electronic device, wherein: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the image processing method according to any one of claims 1 to 30, or implements the steps of the image processing method according to claim 31 or 32 when executed by the processor.

36. A readable storage medium, wherein: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, it implements the steps of the image processing method according to any one of claims 1 to 30, or implements the steps of the image processing method according to claim 31 or 32.

Images