Face image restoration method and device, equipment and storage medium

Abstract

The invention provides a face image restoration method and device, equipment and a storage medium, and the method comprises the steps: detecting a first face image in a to-be-restored image, carrying out the face enhancement operation of the first face image, obtaining an enhanced face image, carrying out the scratch detection and scratch filling operation of the enhanced face image, and obtaining the to-be-restored image. And obtaining a scratch-free second face image, and matching the second face image with the to-be-restored image to obtain a restored image of the to-be-restored image. According to the invention, the restoration effect of the group photo image with the damaged face and the image with the small face area can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to a method, apparatus, device, and storage medium for repairing a face image. Background Art

[0002] Over time, paper old photos may have marks such as scratches or creases, and the scratches or creases may cause the loss of the picture content in the photo. For example, when there is a scratch in the face area of a portrait photo, the integrity of the face in the image will be damaged.

[0003] Although there are existing solutions for image repair in the prior art, most of these solutions divide the damaged area in the image into two types: structural damage and non-structural damage, and then perform repair in two stages: rough repair and fine repair. The rough repair stage mainly removes the non-structural damage in the image, and the fine repair stage mainly solves the structural damage of the image. However, this solution has the problem of distorted repair effect, which is particularly serious for face scratch images with group photos and face scratch images with a small face area. Summary of the Invention

[0004] Embodiments of the present invention provide a method, apparatus, device, and storage medium for repairing a face image, so as to at least solve the problem of distorted repair effect of face images in related technologies.

[0005] According to an embodiment of the present invention, a method for repairing a face image is provided, and the method includes:

[0006] Detect a first face image in the image to be repaired;

[0007] Perform a face enhancement operation on the first face image to obtain an enhanced face image;

[0008] Perform a scratch detection and scratch filling operation on the enhanced face image to obtain a second face image without scratches;

[0009] Match the second face image with the image to be repaired to obtain a repaired image of the image to be repaired.

[0010] Optionally, the performing a face enhancement operation on the first face image to obtain an enhanced face image includes:

[0011] Enhance the face area of the first face image by using a pre-trained face enhancement network to obtain an enhanced face image, where the face enhancement network includes a generator and a discriminator.

[0012] Optionally, the detecting a first face image in the image to be repaired includes:

[0013] Perform downsampling operations on the image to be repaired at multiple multiples to obtain multiple images with different resolutions;

[0014] Input the multiple images with different resolutions into a shared convolutional neural network to obtain an initial face detection result of the image to be repaired;

[0015] Perform non-maximum suppression operations on the initial face detection result to obtain a target face detection result, where the target face detection result includes at least one face detection box;

[0016] Crop out the first face image in the image to be repaired based on the coordinate information of the at least one face detection box.

[0017] Optionally, the cropping out the first face image in the image to be repaired based on the coordinate information of the at least one face detection box includes:

[0018] Obtain the coordinate information of the at least one face detection box;

[0019] Sort the at least one face detection box according to the magnitude of the abscissa value or ordinate value in the coordinate information to obtain a sorting result;

[0020] Crop the face detection boxes in sequence according to the sorting result to obtain the first face image in the image to be repaired.

[0021] Optionally, the enhancing the face region of the first face image by using a pre-trained face enhancement network to obtain an enhanced face image includes:

[0022] A1. Use the generator to generate a predicted face region of the first face image;

[0023] A2. Input the predicted face region and the expected face region into the discriminator to obtain a loss value between the predicted face region and the expected face region;

[0024] A3. Determine whether the loss value is greater than a preset threshold;

[0025] If not, adjust the parameters of the generator through the loss value, and repeat steps A1 to A3 until the loss value is less than or equal to the preset threshold to obtain an enhanced face image.

[0026] Optionally, the determining whether the loss value is greater than a preset threshold further includes:

[0027] If the loss value is less than or equal to the preset threshold, use the predicted face region as the enhanced face image.

[0028] Optionally, the matching of the second face image with the image to be restored to obtain the restored image of the image to be restored includes:

[0029] Performing scratch detection and scratch filling operations on the image to be restored to obtain a scratch-free image to be restored;

[0030] Detecting the overall face image in the scratch-free image to be restored;

[0031] Performing histogram matching between the second face image and the overall face image to obtain the restored image of the image to be restored.

[0032] According to another embodiment of the present invention, there is provided a device for restoring a face image, including:

[0033] A detection module: configured to detect a first face image in the image to be restored;

[0034] A face processing module: configured to perform a face enhancement operation on the first face image to obtain an enhanced face image;

[0035] A scratch processing module: configured to perform scratch detection and scratch filling operations on the enhanced face image to obtain a second scratch-free face image;

[0036] A matching module: configured to match the second face image with the image to be restored to obtain the restored image of the image to be restored.

[0037] According to still another embodiment of the present invention, there is further provided an electronic device, including a memory and a processor, where a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0038] According to still another embodiment of the present invention, there is further provided a storage medium, where a computer program is stored in the storage medium, and the computer program is configured to execute the steps in any one of the above method embodiments when running.

[0039] Through the present invention, when there are face images in a group photo or face images with a small face area in the image to be repaired, by detecting the first face image in the image to be repaired, the faces in the image to be repaired can be determined. Performing a face enhancement operation on the first face image to obtain an enhanced face image can improve the quality and clarity of the face image. Then, performing a scratch detection and scratch filling operation on the enhanced face image to obtain a second face image without scratches. By matching the second face image with the image to be repaired, a repaired image of the image to be repaired can be obtained. Even for group photo images with scratched faces and images with small face areas and scratches, there is a better visual repair effect, avoiding the situation of distorted repaired images. Description of the Drawings

[0040] The drawings described herein are used to provide a further understanding of the present invention, form a part of this application, and the illustrative embodiments and descriptions thereof of the present invention are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0041] Figure 1 is a hardware structure block diagram of the electronic device of the present invention;

[0042] Figure 2 is a flowchart of the method for repairing a face image according to an embodiment of the present invention;

[0043] Figure 3 is a structure block diagram of the device for repairing a face image according to an embodiment of the present invention;

[0044] Figure 4 is a hardware structure block diagram of the electronic device according to another embodiment of the present invention. Detailed Embodiments

[0045] The present invention will be described in detail below with reference to the drawings and in combination with embodiments. It should be noted that, without conflict, the embodiments and features in this application can be combined with each other.

[0046] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above drawings are used to distinguish similar objects and do not necessarily have to be used to describe a specific order or sequence.

[0047] Embodiment 1

[0048] The embodiment of the method for repairing a face image provided in the first embodiment of the present application can be executed in an electronic device, a computer terminal, or a similar computing device. Taking running on an electronic device as an example, Figure 1 is a hardware structure block diagram of the electronic device according to an embodiment of the present invention. As Figure 1 shown, the electronic device 10 may include one or more ( Figure 1Only one processor 102 is shown (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a field programmable gate array FPGA), and a memory 104 for storing data. Optionally, the above-mentioned electronic device may further include a transmission device 106 for communication functions and an input / output device 108. Those of ordinary skill in the art can understand that Figure 1 The structure shown is only illustrative and does not limit the structure of the above-mentioned electronic device. For example, the electronic device 10 may further include more or fewer components than Figure 1 shown in, or have a different configuration from Figure 1 shown.

[0049] The memory 104 can be used to store computer programs. For example, software programs and modules of application software, such as the computer program corresponding to the face image restoration method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implements the above-mentioned method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely disposed relative to the processor 102, and these remote memories can be connected to the electronic device 10 through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0050] The transmission device 106 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wireless network provided by a communication provider of the electronic device 10. In one instance, the transmission device 106 includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other electronic devices through a base station and thus communicate with the Internet. In one instance, the transmission device 106 may be a radio frequency (Radio Frequency, abbreviated as RF) module, which is used to communicate with the Internet wirelessly.

[0051] Figure 1 Only the electronic device with components 102-108 is shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented.

[0052] In an embodiment of the present application, when the processor 102 executes the computer program stored on the memory 104, the following steps are implemented, including:

[0053] Detect a first face image in the image to be restored;

[0054] Perform a face enhancement operation on the first face image to obtain an enhanced face image;

[0055] Perform a scratch detection and scratch filling operation on the enhanced face image to obtain a second face image without scratches;

[0056] Match the second face image with the image to be repaired to obtain a repaired image of the image to be repaired.

[0057] For a detailed introduction to the above steps, please refer to the following Figure 2 Description of the flowchart of the embodiment of the method for repairing a face image.

[0058] In this embodiment, a method for repairing a face image running on the above electronic device is provided. Figure 2 It is a flowchart of the method for repairing a face image according to an embodiment of the present invention. As Figure 2 shown, the process includes the following steps:

[0059] Step S201: Detect the first face image in the image to be repaired;

[0060] Step S202: Perform a face enhancement operation on the first face image to obtain an enhanced face image;

[0061] Step S203: Perform a scratch detection and scratch filling operation on the enhanced face image to obtain a second face image without scratches;

[0062] Step S204: Match the second face image with the image to be repaired to obtain a repaired image of the image to be repaired.

[0063] The image to be repaired can be a group photo of multiple people with scratches, or an old photo with a small face area of a person with scratches. In actual application scenarios, the image to be repaired can also be other types of images with scratches, and the type of the image to be repaired is not specifically limited here. The repair request for the face image can be initiated by the user through the human-computer interaction interface of the electronic device, or the electronic device can actively repair the image to be repaired during the image processing process.

[0064] The image to be repaired can be carried in the request or obtained from a specified storage path. After obtaining the image to be repaired, the first face image in the image to be repaired is detected. The first face image refers to the face region image in the image to be repaired. For example, assuming the image to be repaired is a group photo of three people, the first face image is an image containing three face regions. Specifically, the first face image in the image to be repaired can be detected by combining the scale of the image to be repaired (i.e., the degree of blurriness), the resolution of the image, and the context semantic information of the image using a convolutional neural network model. For example, the YOLO model can be used to detect the first face image, and the detected first face image is cropped.

[0065] Since there may be problems such as unclear face, color distortion, and low resolution in the first face image, it is necessary to perform a face enhancement operation on the first face image. Specifically, the face enhancement algorithm is used to refine the face region of the first face image, thereby obtaining an enhanced face image. After performing the enhancement operation, the quality and clarity of the face image can be improved, so that the scratches existing on the face can be accurately processed subsequently.

[0066] Since there may be scratch regions in the enhanced face image, it is necessary to perform scratch detection and scratch repair operations on the enhanced face image. Specifically, the enhanced face image is input into the scratch detection model, so as to detect the positions of the scratches in the enhanced face image, mark the scratch regions in the image, and output an image with the scratch regions blank and a binary mask recording the scratch positions. The scratch detection model can be the Unet model. The Unet model is an image segmentation model based on a convolutional neural network. The Unet model can extract the scratch features of the enhanced face image through 4 downsampling operations and 4 upsampling operations. Among them, the training process of the scratch detection network includes:

[0067] Obtain scratch old photos from a predetermined database as the training sample set. The training sample set contains image samples of real scratches (i.e., real old photos) and image samples of synthetic scratch simulations (i.e., synthetic old photos), and label the scratch regions so that the image samples and the images to be segmented after labeling correspond one by one.

[0068] After fine-tuning the scratch detection network and training it, the cross-entropy loss is used to minimize the difference between the predicted scratch mask and the actual scratch mask, and the objective function of scratch detection is determined, thereby obtaining the trained scratch detection model.

[0069] After detecting the scratched area of the enhanced face image, a scratch repair operation is performed on the scratched area. The variational auto-encoder (VAE) is used to fill and repair the detected scratched area, and a complete scratch-free face image (denoted as the second face image) is output. Since the VAE has good encoding and decoding capabilities, after training, it can achieve a good repair effect for images with scratches. Therefore, an image with scratches is locally filled in the scratched area after being encoded by the trained VAE, and then a scratch-free image can be output after being decoded by the VAE.

[0070] To reduce the impact of generated data and narrow the spatial distribution difference between synthetic photos and real photos, in this embodiment, scratch repair is regarded as an image conversion problem. The image conversion includes using the variational auto-encoder VAE1, the mapping space for image feature map repair, and the variational auto-encoder VAE2 for image conversion. That is, the variational auto-encoder 1 encodes the scratched image and then performs repair filling in the mapping space, and finally decodes it through the variational auto-encoder 2 to restore it to a scratch-free image. The specific repair steps are as follows:

[0071] First, the images of the variational auto-encoder VAE are respectively mapped to their corresponding mapping spaces. Since both synthetic old photos and real old photos are degraded images and are relatively similar in appearance and vision, some restrictive conditions are added to achieve the purpose of aligned distribution. Two VAEs are used for spatial mapping. The real old photo image {r} and the synthetic photo image {x} share VAE1, which consists of an encoder ER,X and a generator VR,X. The images {r} and {x} are encoded into the mapping spaces ZX and ZY, respectively, and at the same time, the latent encodings both satisfy the Gaussian distribution. Then, the reparameterization is used to achieve differentiable random sampling. After that, {r} and {x} are optimized respectively, and the optimized encoded feature maps are input into the mapping network. Using the synthetic image pair {y,x}, the mapping space is used to guide image repair. The role of the mapping space is to restore the degraded photo to the original photo. When filling the scratched area, the idea of similarity is adopted to fill the scratched area, that is, the target area is filled according to the global image information. Since the pixels in the damaged area have interference, the pixels in the damaged area do not participate in the calculation. By calculating the similarity of any two pixels and the weight of the similarity in turn, the pixel value of the area to be filled is comprehensively calculated, so as to achieve the filling of the scratched area.

[0072] After obtaining the second face image without scratches, perform histogram matching on the second face image and the image to be restored, so as to obtain the restored image of the image to be restored. Specifically, calculate the cumulative distribution functions of the second face image and the image to be restored respectively, match the pixel values of the second face image with the pixel values of the image to be restored according to the cumulative distribution functions, obtain the mapping relationship representing the matching result, and map each pixel of the second face image according to this mapping relationship to obtain the restored image of the image to be restored.

[0073] When there is a group photo face image or a face image with a small face area in the image to be restored, by detecting the first face image in the image to be restored, the face in the image to be restored can be determined. Performing a face enhancement operation on the first face image to obtain an enhanced face image can improve the quality and clarity of the face image. Then, performing a scratch detection and scratch filling operation on the enhanced face image to obtain a second face image without scratches, and obtaining the restored image of the image to be restored by matching the second face image with the image to be restored. Even for a group photo image with scratches on the face and an image with a small face area and scratches, there is a better visual restoration effect, avoiding the situation of distorted restored images.

[0074] Optionally, performing a face enhancement operation on the first face image to obtain an enhanced face image includes:

[0075] Using a pre-trained face enhancement network to enhance the face area of the first face image to obtain an enhanced face image, where the face enhancement network includes a generator and a discriminator.

[0076] The face enhancement network can be an adversarial network with a progressive generator. The adversarial network includes a generator and a discriminator. The generator is used to generate a clear face image according to the first face image, and the discriminator is used to discriminate the image generated by the generator, so that the face image generated by the generator is clearer. Through this way of mutual game, the generator can generate a higher-quality face image, thereby realizing the enhancement operation on the first face image.

[0077] Further, using a pre-trained face enhancement network to enhance the face area of the first face image to obtain an enhanced face image includes:

[0078] A1. Using the generator to generate a predicted face area of the first face image, where the face enhancement network includes a generator and a discriminator;

[0079] A2. Inputting the predicted face area and the expected face area into the discriminator to obtain the loss value between the predicted face area and the expected face area;

[0080] A3. Determine whether the loss value is greater than a preset threshold;

[0081] If not, adjust the parameters of the generator according to the loss value, and repeat steps A1 to A3 until the loss value is less than or equal to the preset threshold, so as to obtain an enhanced face image.

[0082] The expected face region refers to an expected clear face image. The face part in the first face image r (i.e., the face part in the degraded photo r) is denoted as r_f. While the generator generates a predicted face region based on the first face image, r_f is injected into the generator at each scale in a spatially adaptive manner, so as to capture the style and structural information of the degraded face as much as possible. The generator performs upsampling operations and detail generation operations through deconvolution and instance normalization, and obtains the expected face region through multiple layers of iteration. Input the predicted face region and the expected face region into the discriminator, and the discriminator can calculate the loss value between the predicted face region and the expected face region. This loss value represents the gap between the predicted face region generated by the generator and the expected clear face image. Determine whether the loss value is greater than the preset threshold. If not, it means that the predicted face region generated by the generator does not meet the expectations, and the parameters of the generator need to be adjusted according to the loss value, so that the image generated by the generator is closer to the expected clear face image. After the generator adjusts the parameters, repeat the above steps A1 to A3 until the loss value is less than or equal to the preset threshold, so as to obtain an enhanced face image. The enhanced face image has higher clarity compared with the first face image. Among them, for the generated face image G f (z, r_f) and the real high-resolution face image (expected face region) r c The formula for the perceptual loss between them is as follows:

[0083]

[0084] where r_f is the degraded image of r c , z is the downsampled feature map of r_f, E represents the expectation of the distribution function, φ VGG represents the pre-trained VGG network, and n represents the number of samples. Another adversarial loss is introduced during the training process to ensure the synthesis of high-frequency details. The specific formula is as follows:

[0085]

[0086] where Z, respectively represent the distributions of z, r_f, r c real samples, and D f represents the discriminator.

[0087] Furthermore, the determination of whether the loss value is greater than the preset threshold further includes:

[0088] If the loss value is less than or equal to a preset threshold, the predicted face region is used as the enhanced face image.

[0089] If the loss value calculated by the discriminator for the predicted face region and the expected face region is less than or equal to the preset threshold, it indicates that the face image generated by the generator meets the expectation. At this time, the predicted face region generated by the generator is directly used as the enhanced face image.

[0090] Optionally, detecting the first face image in the image to be repaired includes:

[0091] Performing downsampling operations on the image to be repaired at multiple multiples to obtain multiple images with different resolutions;

[0092] Inputting the multiple images with different resolutions into a shared convolutional neural network to obtain an initial face detection result of the image to be repaired;

[0093] Performing non-maximum suppression operations on the initial face detection result to obtain a target face detection result, where the target face detection result includes at least one face detection box;

[0094] Based on the coordinate information of the at least one face detection box, cropping out the first face image in the image to be repaired.

[0095] The pyramid method using image multi - resolution is used to process the image to be inpainted, that is, the image to be inpainted is downsampled by multiple factors to obtain multiple images with different resolutions, thus forming a pyramid - shaped image structure. In order to more accurately detect smaller faces in the image to be inpainted, the additional context information of the face region (i.e., the information around the face region) is used to assist in the detection. At the same time, the fusion of features at different resolution levels is further utilized to improve the detection efficiency of small - target faces in the image to be inpainted, so as to complete the detection of all faces in the image to be inpainted. The multiple images with different resolutions obtained are input into a shared convolutional neural network to obtain the face detection results of the image to be inpainted (denoted as the initial detection results). There are multiple overlapping detection boxes in the initial face detection results, so it is necessary to perform non - maximum suppression on the initial face detection results, that is, select the best detection box from the overlapping detection boxes. Specifically, each detection box has a corresponding score value. The scores of all detection boxes are sorted from large to small. First, the box with the highest score value is selected. Then, the remaining boxes are traversed. If the overlapping area with the box with the highest score value is greater than a certain threshold, the detection box is deleted. Continue to select the box with the highest score from the unprocessed boxes and repeat the traversal process. Finally, the most appropriate target face detection results can be obtained. The target face detection results include at least one face detection box. That is, the number of face detection boxes corresponds to the number of faces in the image to be inpainted.

[0096] The face detection box is a rectangular box. The coordinate information of the face detection box can refer to the coordinate value of any corner of the face detection box. If the image to be inpainted is a group photo of multiple people, there will be multiple face detection boxes. In order to avoid the disorder of face order during subsequent image matching, the face region image (i.e., the first face image) in the image to be inpainted is cropped through the coordinate information of the face detection box, which can prevent the disorder of face order during subsequent matching. For example, assume the group photo is a photo of three people, A, B, and C. Then, according to the size order of the coordinate values of the detection boxes of A, B, and C, the face regions of A, B, and C are cropped.

[0097] Furthermore, cropping the first face image in the image to be inpainted based on the coordinate information of the at least one face detection box includes:

[0098] Obtain the coordinate information of the at least one face detection box;

[0099] Sort the at least one face detection box according to the magnitude of the abscissa value or ordinate value in the coordinate information to obtain a sorting result;

[0100] Crop the face detection box in sequence according to the sorting result to obtain the first face image in the image to be inpainted.

[0101] Obtain the coordinate information of each face detection box. For example, obtain the abscissa value or ordinate value of the upper left corner of each face detection box. Sort each face detection box according to the magnitude of the abscissa value or ordinate value in the coordinate information to obtain a sorting result. Taking the above example, assume that the abscissa value of the upper left corner of the face detection box of A is 100, the abscissa value of the upper left corner of the face detection box of B is 300, and the abscissa value of the upper left corner of the face detection box of C is 200. Then the sorting result is A, C, B. Crop the face detection boxes in sequence according to this sorting result to obtain the first face image in the image to be repaired. Since the first face image is cropped according to the sorting result, it is possible to prevent the disorder of the face order during subsequent image matching.

[0102] Optionally, the matching the second face image with the image to be repaired to obtain the repaired image of the image to be repaired includes:

[0103] Perform scratch detection and scratch filling operations on the image to be repaired to obtain a scratch-free image to be repaired;

[0104] Detect the overall face image in the scratch-free image to be repaired;

[0105] Perform histogram matching between the second face image and the overall face image to obtain the repaired image of the image to be repaired.

[0106] The technical means adopted for performing scratch detection and scratch filling operations on the image to be repaired are the same as those adopted for performing scratch detection and scratch filling operations on the enhanced face image, which will not be elaborated here. After obtaining a scratch-free image to be repaired, detect the overall face image in the scratch-free image to be repaired. The technical means adopted for detecting the overall face image are the same as those adopted for detecting the first face image in the image to be repaired, which will not be elaborated here. Perform histogram matching between the second face image and the overall face image to obtain the repaired image of the image to be repaired. The repaired image is a complete and scratch-free image. Since the second face image is a local image after performing face enhancement, scratch detection, and scratch repair on the face image cropped from the image to be repaired, and the overall face image is the face image directly detected from the image after performing scratch detection and scratch repair on the image to be repaired, by performing histogram matching between the second face image and the overall face image, the visual effect of the image can be improved, making the repaired image clearer and having a good contrast, and improving the repair effect of the image.

[0107] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server, or electronic device, etc.) to execute the methods described in various embodiments of the present invention.

[0108] Embodiment 2

[0109] In this embodiment, a face image restoration device is provided. This device is used to implement the above embodiments and preferred implementation manners, and those that have been described will not be repeated. As used hereinafter, the term "module" can be a combination of software and / or hardware that realizes a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0110] Figure 3 is a structural block diagram of a face image restoration device according to an embodiment of the present invention. As Figure 3 shown, the device includes:

[0111] Detection module 31: used to detect the first face image in the image to be restored;

[0112] Face processing module 32: used to perform face enhancement operations on the first face image to obtain an enhanced face image;

[0113] Scratch processing module 33: used to perform scratch detection and scratch filling operations on the enhanced face image to obtain a second face image without scratches;

[0114] Matching module 34: used to match the second face image with the image to be restored to obtain a restored image of the image to be restored.

[0115] It should be noted that the above-mentioned various modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to this: the above-mentioned modules are all located in the same processor; or, the above-mentioned various modules are located in different processors in any combination form.

[0116] Embodiment 3

[0117] Referring to Figure 4 shown, it is a schematic diagram of another embodiment of the electronic device of the present application.

[0118] The electronic device includes a processor 111, a communication interface 112, a memory 113, and a communication bus 114. Among them, the processor 111, the communication interface 112, and the memory 113 complete communication with each other through the communication bus 114;

[0119] The memory 113 is used to store computer programs, for example, a program for repairing a face image;

[0120] Figure 4 Only the electronic device with components 111 - 114 is shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively.

[0121] In an embodiment of the present application, when the processor 111 executes the program stored on the memory 113, it implements the method for repairing a face image provided by any one of the foregoing method embodiments, including:

[0122] Detect a first face image in the image to be repaired;

[0123] Perform a face enhancement operation on the first face image to obtain an enhanced face image;

[0124] Perform a scratch detection and scratch filling operation on the enhanced face image to obtain a second face image without scratches;

[0125] Match the second face image with the image to be repaired to obtain a repaired image of the image to be repaired.

[0126] For a detailed introduction to the above steps, please refer to the Figure 2 Explanation of the flowchart of the method embodiment for repairing a face image.

[0127] Embodiment 4

[0128] An embodiment of the present invention provides a storage medium, in which a computer program is stored, and the computer program is set to execute the steps in any one of the foregoing method embodiments when running.

[0129] Optionally, in this embodiment, the above storage medium can be set to store a computer program for executing the following steps:

[0130] Detect a first face image in the image to be repaired;

[0131] Perform a face enhancement operation on the first face image to obtain an enhanced face image;

[0132] Perform a scratch detection and scratch filling operation on the enhanced face image to obtain a second face image without scratches;

[0133] Match the second face image with the image to be restored to obtain the restored image of the image to be restored.

[0134] Optionally, the performing a face enhancement operation on the first face image to obtain an enhanced face image includes:

[0135] Enhance the face region of the first face image by using a pre-trained face enhancement network to obtain an enhanced face image, where the face enhancement network includes a generator and a discriminator.

[0136] Optionally, the detecting the first face image in the image to be restored includes:

[0137] Perform downsampling operations on the image to be restored at multiple multiples to obtain multiple images with different resolutions;

[0138] Input the multiple images with different resolutions into a shared convolutional neural network to obtain an initial face detection result of the image to be restored;

[0139] Perform a non-maximum suppression operation on the initial face detection result to obtain a target face detection result, where the target face detection result includes at least one face detection box;

[0140] Crop out the first face image in the image to be restored based on the coordinate information of the at least one face detection box.

[0141] Optionally, the cropping out the first face image in the image to be restored based on the coordinate information of the at least one face detection box includes:

[0142] Obtain the coordinate information of the at least one face detection box;

[0143] Sort the at least one face detection box according to the magnitude of the abscissa value or ordinate value in the coordinate information to obtain a sorting result;

[0144] Crop the face detection boxes in sequence according to the sorting result to obtain the first face image in the image to be restored.

[0145] Optionally, the enhancing the face region of the first face image by using a pre-trained face enhancement network to obtain an enhanced face image includes:

[0146] A1. Use the generator to generate a predicted face region of the first face image, where the face enhancement network includes a generator and a discriminator;

[0147] A2. Input the predicted face region and the expected face region into the discriminator to obtain the loss value between the predicted face region and the expected face region;

[0148] A3. Determine whether the loss value is greater than a preset threshold;

[0149] If not, adjust the parameters of the generator according to the loss value, and repeat steps A1 to A3 until the loss value is less than or equal to the preset threshold to obtain an enhanced face image.

[0150] Optionally, determining whether the loss value is greater than a preset threshold further includes:

[0151] If the loss value is less than or equal to the preset threshold, use the predicted face region as the enhanced face image.

[0152] Optionally, matching the second face image with the image to be repaired to obtain the repaired image of the image to be repaired includes:

[0153] Perform scratch detection and scratch filling operations on the image to be repaired to obtain a scratch-free image to be repaired;

[0154] Detect the overall face image in the scratch-free image to be repaired;

[0155] Perform histogram matching between the second face image and the overall face image to obtain the repaired image of the image to be repaired.

[0156] In this embodiment, the above storage medium may include, but is not limited to: various media such as USB flash drives, read-only memories (ROM), random access memories (RAM), mobile hard disks, magnetic disks, or optical discs that can store computer programs.

[0157] Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present invention can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. Optionally, they can be implemented by program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described herein can be executed in a different order, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module to implement. Thus, the present invention is not limited to any specific combination of hardware and software.

[0158] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the present invention. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for repairing a face image, characterized in that, the method includes: detecting a first face image in the image to be repaired; performing a face enhancement operation on the first face image to obtain an enhanced face image; performing scratch detection and scratch filling operations on the enhanced face image to obtain a second face image without scratches; matching the second face image with the image to be repaired to obtain a repaired image of the image to be repaired.

2. The method according to claim 1, characterized in that, the performing a face enhancement operation on the first face image to obtain an enhanced face image includes: using a pre-trained face enhancement network to enhance the face area of the first face image to obtain an enhanced face image, where the face enhancement network includes a generator and a discriminator.

3. The method according to claim 1, characterized in that, the detecting a first face image in the image to be repaired includes: performing downsampling operations on the image to be repaired at multiple multiples to obtain multiple images with different resolutions; inputting the multiple images with different resolutions into a shared convolutional neural network to obtain an initial face detection result of the image to be repaired; performing a non-maximum suppression operation on the initial face detection result to obtain a target face detection result, where the target face detection result includes at least one face detection box; cropping out the first face image in the image to be repaired based on the coordinate information of the at least one face detection box.

4. The method according to claim 3, characterized in that, the cropping out the first face image in the image to be repaired based on the coordinate information of the at least one face detection box includes: obtaining the coordinate information of the at least one face detection box; sorting the at least one face detection box according to the magnitude of the abscissa value or ordinate value in the coordinate information to obtain a sorting result; successively cropping the face detection boxes according to the sorting result to obtain the first face image in the image to be repaired.

5. The method according to claim 2, characterized in that, the using a pre-trained face enhancement network to enhance the face area of the first face image to obtain an enhanced face image includes: A1. Using the generator to generate a predicted face area of the first face image; A2. Inputting the predicted face area and the expected face area into the discriminator to obtain a loss value between the predicted face area and the expected face area; A3. Judging whether the loss value is greater than a preset threshold; if not, adjusting the parameters of the generator through the loss value, and repeating steps A1 to A3 until the loss value is less than or equal to the preset threshold to obtain an enhanced face image.

6. The method according to claim 5, characterized in that, the judging whether the loss value is greater than a preset threshold further includes: if the loss value is less than or equal to the preset threshold, using the predicted face area as the enhanced face image.

7. The method according to claim 1, characterized in that, Matching the second face image with the image to be restored to obtain a restored image of the image to be restored, including: Performing scratch detection and scratch filling operations on the image to be restored to obtain a scratch-free image to be restored; Detecting an overall face image in the scratch-free image to be restored; Performing histogram matching between the second face image and the overall face image to obtain a restored image of the image to be restored.

8. A device for restoring a face image, Characterized in that, It includes: A detection module: configured to detect a first face image in an image to be restored; A face processing module: configured to perform a face enhancement operation on the first face image to obtain an enhanced face image; A scratch processing module: configured to perform scratch detection and scratch filling operations on the enhanced face image to obtain a second scratch-free face image; A matching module: configured to match the second face image with the image to be restored to obtain a restored image of the image to be restored.

9. An electronic device, including a memory and a processor, Characterized in that, A computer program is stored in the memory, and the processor is configured to run the computer program to execute the method described in any one of claims 1 to 7.

10. A storage medium, Characterized in that, A computer program is stored in the storage medium, wherein the computer program is configured to execute the method described in any one of claims 1 to 7 when running.

Images