A photo restoration method and system

Abstract

This invention discloses a photo restoration method, comprising the following steps: Step S1: Extracting the photo region from the image. Step S2: Extracting damaged, creased, and scratched areas from the photo region. Step S3: Repairing the damaged, creased, and scratched areas in the photo region. Step S4: Detecting the face region in the photo region. Step S5: Removing various degradations in the face region and adding facial details to the face region. The technical effects achieved by this invention are: it can restore damage, scratches, and creases in photos; the facial restoration is realistic and natural, adding facial details while preserving the portrait's identity features.

Description

Technical Field

[0001] This invention relates to a method for restoring photographs. Background Technology

[0002] The shooting environment, shooting quality, and image storage all affect photo quality. For example, inaccurate camera focus or camera shake during shooting can result in blurry photos and low image quality. Similarly, printed photos may suffer damage, fading, stains, or blurring over long-term storage. Furthermore, poor scanning or photography quality can lead to low-quality images and video frames. All of these situations necessitate photo restoration.

[0003] Existing photo restoration methods are often quite simple and cannot solve the various defects listed above. Summary of the Invention

[0004] The technical problem to be solved by this invention is to propose a method for efficient photo restoration through neural networks.

[0005] To address the aforementioned technical problems, this invention proposes a photo restoration method, comprising the following steps: Step S1: Extracting the photo region from the image. Step S2: Extracting damaged, creased, and scratched areas from the photo region. Step S3: Repairing the damaged, creased, and scratched areas in the photo region. Step S4: Detecting and locating the face region in the photo region. Step S5: Removing various degradations in the face region and adding facial details to the face region.

[0006] Further, step S1 is implemented as follows: a neural network for image segmentation is used and trained, and the trained neural network is used to extract photo regions from the image. In step S1, the training method of the neural network is as follows: (1) generating training data; using existing photos to generate images containing photos, called synthetic images, the position of the photo region in each synthetic image is specified, i.e., known; or collecting some images containing photos, the position of the photo region in each image is manually labeled, i.e., known; (2) using the training data to train the neural network so that the neural network can be used to extract photo regions from the image.

[0007] Preferably, the training data includes the following four categories: black and white photos composited onto a black and white background, black and white photos composited onto a color background, color photos composited onto a black and white background, and color photos composited onto a color background; the background refers to the non-photo area in the image.

[0008] Further, step S2 is implemented as follows: a neural network for image segmentation is used and trained, and the trained neural network is used to extract damaged, creased, and scratched areas in the photo area. In step S2, the training method of the neural network is as follows: (1) generating training data; collecting some photos with existing damaged, creased, and scratched areas, where the damaged, creased, and scratched areas in each photo are known; or, generating some photos containing damaged, creased, and scratched areas, where the positions of the damaged, creased, and scratched areas in each photo are specified, i.e., known; (2) training the neural network with the training data so that the neural network can be used to extract damaged, creased, and scratched areas in the photo.

[0009] Further, step S3 is implemented as follows: a neural network for image restoration is used and trained, and the trained neural network is used to repair the damaged, creased, and scratched areas of the photo. In step S3, the training method of the neural network is as follows: (1) generating training data; adding some damaged, creased, and scratched areas at specified locations in some photos, the original pixel content of the photos covered by these damaged, creased, and scratched areas is known; (2) training the neural network with the training data so that the neural network can be used to repair the damaged, creased, and scratched areas of the photo.

[0010] Preferably, step S3 employs a neural network for image inpainting that includes convolution operations, while replacing the convolution operations with fast Fourier convolution operations.

[0011] Further, step S5 is implemented as follows: an end-to-end neural network for image denoising or deblurring is used and trained with training data generated using multiple small-amplitude degradation methods. The trained neural network is then used to remove multiple degradations from the face region in the photo area. The end-to-end neural network includes an encoder and a decoder. A generative adversarial network for image processing is used to add facial details to the degraded face region. The generative adversarial network is used to generate high-definition face images based on prior facial information. The generative adversarial network includes a generator and a discriminator. The generator includes an encoder and a decoder. Feature fusion is performed between the feature layer in the decoder of the end-to-end neural network and the feature layer in the decoder of the generator in the conditional adversarial generative network. In step S5, the training method of the end-to-end neural network is as follows: (1) Training data is generated in a multiple small-amplitude degradation manner; this means that one or more small-amplitude degradations are added to the photo first, called the first degradation; then one or more small-amplitude degradations are added to the photo after the first degradation, called the second degradation; and so on, to obtain the photo after multiple degradations; the content of the original undegraded photo is known, and the pixels of the photo after multiple degradations correspond one-to-one with the pixels of the original undegraded photo; the small-amplitude degradation means that a single degradation does not significantly reduce the image quality; the multiple refers to one to six layers; (2) The end-to-end neural network is trained using the "training data generated in a multiple small-amplitude degradation manner" so that the end-to-end neural network can be used to remove various degradations in the image.

[0012] Optionally, step S6 is added after step S4; step S5 and step S6 may either occur before each other or be performed simultaneously. Step S6: Repair and enlarge the portion of the photo area excluding the face area.

[0013] Preferably, in step S6, an image super-resolution reconstruction or restoration technique based on generative adversarial networks is used.

[0014] Optionally, steps S7 and S8 are added sequentially after step S5. Step S7: Determine whether the photo area needs colorization; if colorization is needed, proceed to step S8; if colorization is not needed, the entire restoration process ends. Photos requiring colorization include: black and white photos, faded color photos, and color photos with color casts; photos not requiring colorization refer to vibrant and accurate color photos. Step S8: Automatically colorize the photo area requiring colorization; the colorization includes: converting black and white photos to color photos, converting faded color photos to vibrant color photos, and converting color photos with color casts to accurate color photos.

[0015] Further, step S7 is implemented as follows: a neural network for classification is used and trained, and the trained neural network is used to determine whether the photo area needs to be colored. In step S7, the training method of the neural network is as follows: (1) Generate training data; collect some original photos, which are bright and accurate color photos; keep some original photos as is, convert some original photos to black and white photos, convert some original photos to faded color photos, and convert some original photos to color-biased color photos; the category to which the converted photos belong is known; there are only two categories, namely "photos that need to be colored" and "photos that do not need to be colored"; (2) use the training data to train the neural network so that the neural network can be used as a binary classifier to distinguish between the two categories.

[0016] Alternatively, in step S7, the training method of the neural network is changed to: (1) generating training data; collecting some original photos, which are bright and accurate color photos; keeping some of the original photos as they are, converting some of the original photos to black and white photos, converting some of the original photos to faded color photos, and converting some of the original photos to color-skewed color photos; the category to which the converted photos belong is known; there are only four categories, namely "black and white photos", "faded color photos", "color-skewed color photos" and "bright and accurate color photos"; the first three categories all indicate that the photos need to be colored, and the last category indicates that the photos do not need to be colored; (2) training the neural network with the training data so that the neural network can be used as a four-classifier to distinguish four categories.

[0017] Preferably, in step S8, the coloring is performed only in the color channels of the photo, without changing the luminance channels.

[0018] Further, step S8 is implemented as follows: a neural network for image style transfer is used and trained, and the trained neural network is used to automatically colorize photos that need to be colorized. In step S8, the training method of the neural network is as follows: (1) the training data generated in step S7 is used, or the amount of training data is increased; the category of the training data is known, and the training data is unpaired; (2) the training data is used to train the neural network so that the neural network can be used to automatically colorize photos that need to be colorized.

[0019] Optionally, step S6 is added after step S4; step S5 and step S6 may be performed either before or simultaneously; step S7 and step S8 are added sequentially after step S5 and step S6.

[0020] This invention also proposes a photo restoration system, including a photo extraction unit, a damage extraction unit, a damage repair unit, a face detection unit, and a face restoration unit. The photo extraction unit is used to extract photo regions from an image. The damage extraction unit is used to extract damaged, creased, and scratched areas from the photo regions. The damage repair unit is used to repair the damaged, creased, and scratched areas in the photo regions. The face detection unit is used to detect and locate face regions in the photo regions. The face restoration unit is used to remove various degradations in the face regions and add facial details to the face regions.

[0021] Optionally, a non-face magnification unit is added. The non-face magnification unit is used to repair and magnify the parts of the photo area other than the face area.

[0022] Optionally, a coloring judgment unit and an automatic coloring unit are added. The coloring judgment unit is used to determine whether a photo area needs coloring; if coloring is needed, the photo area is sent to the automatic coloring unit; if coloring is not needed, the entire restoration process ends. The automatic coloring unit is used to automatically color the photo areas that need coloring.

[0023] The technical effects achieved by this invention are: it can repair damage, scratches, and creases in photos; the facial restoration is realistic and natural, and facial details are added while preserving the identity features of the person; the restoration speed is fast and the efficiency is high. Attached Figure Description

[0024] Figure 1 This is a flowchart of an embodiment of the photo restoration method proposed in this invention.

[0025] Figure 2 This is a flowchart of Embodiment 2 of the photo restoration method proposed in this invention.

[0026] Figure 3 This is a flowchart of Embodiment 3 of the photo restoration method proposed in this invention.

[0027] Figure 4 This is a flowchart of Embodiment 4 of the photo restoration method proposed in this invention.

[0028] Figure 5 This is a schematic diagram of the structure of a first embodiment of the photo restoration system proposed in this invention.

[0029] Figure 6 This is a schematic diagram of the structure of Embodiment 2 of the photo restoration system proposed in this invention.

[0030] Figure 7 This is a schematic diagram of the structure of Embodiment 3 of the photo restoration system proposed in this invention.

[0031] Figure 8This is a schematic diagram of the structure of Embodiment 4 of the photo restoration system proposed in this invention.

[0032] The following are the labels in the attached diagram: 1 is the photo extraction unit, 2 is the damage extraction unit, 3 is the damage repair unit, 4 is the face detection unit, 5 is the face repair unit, 6 is the non-face magnification unit, 7 is the colorization judgment unit, and 8 is the automatic colorization unit. Detailed Implementation

[0033] Please see Figure 1 The first embodiment of the photo restoration method proposed in this invention includes the following steps.

[0034] Step S1: Extract the photo region from the image. The photo region is the set of pixels in the image that are displayed as a photo. It can be regular or irregular in shape, such as a quadrilateral. At least a portion of the image is a photo region, or the entire image may be a photo region. If no photo region is detected and located in the image, the entire image is considered a photo region. Subsequent steps will only detect damaged, creased, and scratched areas within the photo region, and will only detect face areas within the photo region to determine whether colorization is needed and perform automatic colorization.

[0035] An exemplary implementation of step S1 is as follows: A neural network (NN) for image segmentation is employed and trained. The trained neural network is then used to extract regions from the image. Neural network is short for Artificial Neural Network (ANN). Examples of neural networks used in this step include BiseNet, U-Net, and U... 2 The neural network can be any one of the following: -Net, UNet++, DeepLabv3 series, or SegFormer series. Regardless of the specific neural network used, the training data and methods are the same; the differences lie only in model size (space occupied), computational speed, and computational precision (accuracy). Preferably, after detecting and locating the photo region in the image, the neural network used for image segmentation crops the photo region through perspective transformation. The perspective-transformed photo region is rectangular; this is the ROI (region of interest) commonly found in image processing.

[0036] In step S1, the training method of the neural network is as follows: (1) Generate training data. Generate images containing photos using existing photos. These images are called composite images. The location of the photo region in each composite image is specified, i.e., known. Alternatively, collect some images containing photos, where the location of the photo region in each image is manually labeled, i.e., known. The non-photo region in each image is called the background. (2) Train the neural network using the training data so that the neural network can be used to extract photo regions from images. Preferably, the training data includes the following four categories: black and white photos composited onto a black and white background, black and white photos composited onto a color background, color photos composited onto a black and white background, and color photos composited onto a color background; the combination of these four types of training data helps to improve the accuracy of the neural network in extracting photo regions from images.

[0037] Step S2: Extract the damaged, creased, and scratched areas from the photo area. The damaged, creased, and scratched areas are the collection of pixels in the photo area that appear as damaged, creased, or scratched, and can be regular or irregular in shape.

[0038] An exemplary implementation of step S2 is as follows: A neural network for image segmentation is employed and trained. The trained neural network is then used to extract damaged, creased, and scratched areas from the photograph area. The neural network used in this step is, for example, U-Net or U... 2 Any one of the following: -Net, UNet++, DeepLabv3 series, or SegFormer series.

[0039] In step S2, the training method of the neural network is as follows: (1) Generate training data. Collect some photos with existing damaged, creased, and scratched areas (any one or more), where the damaged, creased, and scratched areas in each photo are known, for example, by manual annotation. Alternatively, generate some photos containing damaged, creased, and scratched areas (any one or more), where the location of the damaged, creased, and scratched areas in each photo is specified, i.e., known. (2) Train the neural network using the training data so that the neural network can be used to extract damaged, creased, and scratched areas from photos.

[0040] Step S3: Repair damaged, creased, and scratched areas in the photo. Specifically, this step uses the edges of the damaged, creased, and scratched areas to infer the pixel content of the damaged, creased, and scratched areas based on the information of the surrounding pixels. Then, the damaged, creased, and scratched areas are automatically filled with pixels, while maintaining coordination between the filled area and the surrounding pixel area to achieve the purpose of image repair.

[0041] An exemplary implementation of step S3 is as follows: A neural network for image inpainting is trained, and the trained neural network is used to repair damaged, creased, or scratched areas in the photo; or an existing image inpainting method is used. The neural network used in this step is, for example, DeepFillv2, HiFill, or an image inpainting method such as the cv2.inpaint() function in OpenCV. Preferably, this step uses a neural network for image inpainting that includes convolution operations, but replaces the convolution operations with Fast Fourier Convolution (FFC) operations, for example, changing the convolutional layers of the HiFill backbone network to FFC. Fast Fourier Convolution is a channel-wise convolution based on the Fast Fourier Transform (FFT), which can expand the receptive field of the neural network, improve the inpainting effect, and make the filling effect more realistic and natural.

[0042] In step S3, the training method of the neural network is as follows: (1) Generate training data. Add some damaged, creased, or scratched areas (any one or more) at designated locations in some photos. The original pixel content of the photos covered by these damaged, creased, or scratched areas is known. (2) Train the neural network using the training data so that the neural network can be used to repair the damaged, creased, or scratched areas of the photos, so that the pixel content of the repaired damaged, creased, or scratched areas is restored to the original pixel content.

[0043] Step S4: Detect and locate face regions in the photo area. This step, for example, uses the Retinaface face detection algorithm to detect and locate all face regions in the photo area. The resulting face regions are, for example, rectangular regions containing faces.

[0044] Step S5: Remove various degradations in the face region. After completion, there may be a loss of facial details in the face region, but the facial identity features of the original photo will still be retained. Then, facial details are added to the face region. The degradations refer to defects in the image, including noise, blur, JPEG compression, downsampling, pixel jitter, artifacts, fading, etc.

[0045] An exemplary implementation of step S5 is as follows: An end-to-end neural network for image denoising or deblurring is employed and trained using training data generated with multiple small-amplitude degradation methods. The trained neural network is then used as a degradation removal network to remove various degradations from the face region in the photo (referred to as coarse face restoration). Examples of end-to-end neural networks used in this step include MPRNet and pixel2pixel (also known as pixel2pix). The end-to-end neural network includes an encoder and a decoder. The degradation removal network collects high-quality face images and synthesizes low-quality degradation data, which is used as training data for the encoder and decoder to learn the mapping relationship from blurry low-quality images to clear high-quality images. Subsequently, a generative adversarial network (GAN) for image processing is used to add facial details to the degraded face region (referred to as fine face restoration). Examples of GANs used in this step include CycleGAn, StyleGAN, StyleGAN2, and BEGAN. These GANs are used to generate high-definition face images based on rich prior facial information. The generative adversarial network (GAN) includes a generator and a discriminator. The discriminator constrains the generator to generate high-resolution face images and learns prior facial information. The generator includes an encoder and a decoder. Adding facial details involves fusing features from the decoder in the end-to-end neural network with features from the decoder in the generator within the GAN. The former feature layer contains facial identity features from the original photo but may lose some facial details; the latter feature layer is used to refine and add more facial details (texture). Using only an end-to-end neural network may result in lost facial details, such as partially missing facial contours that are difficult to recover. Using only a GAN may result in lost facial identity features. This invention fuses the features of both, thus balancing facial details and facial identity features.

[0046] In step S5, the training method for the end-to-end neural network is as follows: (1) Training data is generated using multiple small-amplitude degradation. First, high-quality (clear, noise-free, compression artifact-free, and fade-free) photos containing human faces are collected. One or more small-amplitude degradations are added to the photos, which is called the first degradation. Then, one or more small-amplitude degradations are added to the photos after the first degradation, which is called the second degradation. And so on, to obtain photos after multiple degradations. Multiple degradations simulate the real degradation process in nature and obtain low-quality (blurry, noisy, compression artifact-free, and faded) photos containing human faces that are consistent with the distribution of real old photos. The content of the original undegraded photos is known. The pixels of the photos after multiple degradations correspond one-to-one with those of the original undegraded photos, serving as paired training datasets. The small-amplitude degradation refers to a single degradation that does not significantly reduce the image quality. For example, for JPEG compression, it means that the single JPEG compression quality factor is ≥0.65 and <1, preferably ≥0.90 and <1. The term "multiple" refers to a range from one to six levels, with three levels being preferred. (2) The end-to-end neural network is trained using the "training data generated in a multiple small-amplitude degradation manner" so that the end-to-end neural network can be used to remove various degradations in the photo. Existing end-to-end neural networks can usually only be used to remove one type of degradation, such as for removing image noise. This invention generates training data in a multiple small-amplitude degradation manner, which realistically simulates the degradation process of the photo; the end-to-end neural network trained using this training data can be used to remove multiple degradations simultaneously, which is a major technical innovation of this invention.

[0047] The above embodiment 1 achieves crack repair of the photo (step S3) and image quality improvement of the face area (step S5). After step S5 is completed, the face area is enlarged, still retaining the facial identity features in the original photo, and adding a large amount of facial details from prior facial information.

[0048] Please see Figure 2 The second embodiment of the photo restoration method proposed in this invention includes steps S1 to S5, which are the same as those in the first embodiment and will not be described again. The second embodiment also adds step S6. The order of steps S5 and S6 is not strictly limited; either can be performed first, or simultaneously. However, both steps S5 and S6 follow step S4.

[0049] Step S6: Use image super-resolution reconstruction or restoration techniques to repair and enlarge the areas of the photo other than the face area (i.e., non-face areas), for example, by magnifying them by 2x or 4x. The image super-resolution reconstruction or restoration techniques may be based on generative adversarial networks or employ the RealSR algorithm. The magnification in this step is preferably lossless, but it can also be lossy magnification that does not affect the visual quality (the image does not become noticeably blurry).

[0050] The above-described embodiment two adds the repair and magnification of non-face areas in the photo area based on embodiment one, thereby realizing the repair and magnification of the entire photo (not just the face area in the photo).

[0051] Please see Figure 3 The third embodiment of the photo restoration method proposed in this invention includes steps S1 to S5, which are the same as those in the first embodiment and will not be described again. The third embodiment also adds steps S7 and S8 sequentially after step S5.

[0052] Step S7: Determine if the photo area needs colorization. If colorization is needed, proceed to Step S8. If colorization is not needed, the entire restoration process is complete. Photos that need colorization generally fall into three categories: black and white photos, faded color photos, and color photos with color casts. There is only one type of photo that does not need colorization: vibrant and accurate color photos.

[0053] An exemplary implementation of step S7 is as follows: A neural network for classification is employed and trained. The trained neural network is then used to determine whether a region of the photo needs colorization. The neural network used in this step is, for example, ResNet-50, which is typically used to distinguish the types of objects in an image.

[0054] In step S7, the training method for the neural network is as follows: (1) Generate training data. Collect some original photos, which are bright and accurate color photos. Keep some of the original photos as is, convert some of the original photos to black and white photos, convert some of the original photos to faded color photos, and convert some of the original photos to color photos with color cast. The category to which the converted photos belong is known. There are two categories: "photos that need colorization" and "photos that do not need colorization". (2) Train the neural network using the training data so that the neural network can act as a binary classifier and be used to distinguish between the two categories.

[0055] The above-mentioned neural network training method one is only an example. The binary classifier can also be changed to a three-classifier, a four-classifier, etc. In the current application scenario, changing it to a four-classifier is particularly suitable. In step S7, the neural network training method two is as follows: (1) Generate training data. Collect some original photos. The original photos are bright and accurate color photos. Keep some of the original photos as they are, convert some of the original photos to black and white photos, convert some of the original photos to faded color photos, and convert some of the original photos to color-skewed color photos. The category to which the converted photos belong is known. There are four categories, namely "black and white photos", "faded color photos", "color-skewed color photos" and "bright and accurate color photos"; the first three categories all indicate that the photos need to be colored, and the last category indicates that the photos do not need to be colored. (2) Use the training data to train the neural network so that the neural network can be used as a four-classifier to distinguish four categories.

[0056] Step S8: Automatically colorize the areas of the photo that need colorization. The colorization includes: converting a black and white photo to a color photo, converting a faded color photo to a vibrant color photo, and converting a color-skewed color photo to a color photo with accurate colors. Preferably, the colorization only fills the color channels of the photo, without changing the luminance channels.

[0057] An exemplary implementation of step S8 is as follows: A neural network for image style transfer is employed and trained. The trained neural network is then used to automatically colorize the photograph requiring colorization. For example, CycleGAN is used as the neural network in this step.

[0058] In step S8, the training method of the neural network is as follows: (1) The training data generated in step S7 is used, and the amount of training data can be further increased on this basis. The categories of the training data are known, and the training data are unpaired. (2) The neural network is trained using the training data so that the neural network can be used to automatically colorize photos that need to be colorized.

[0059] In steps S7 and S8, the same training data is reused. Only one training data is needed to train two neural networks with different functions, which simplifies the training process and shortens the training time. This is a major technical innovation of the present invention.

[0060] The above-described embodiment three, based on embodiment one, adds the ability to determine whether a photo area needs to be colored and to automatically color it, thereby improving the overall restoration effect of the photo.

[0061] Please see Figure 4The fourth embodiment of the photo restoration method proposed in this invention includes steps S1 to S8. The order of steps S5 and S6 is not strictly limited; either can be performed first, or simultaneously. However, steps S5 and S6 are both after step S4 and before step S7. Each step has been described in embodiments one, two, and three, and the content is the same, so it will not be repeated here.

[0062] Please see Figure 5 The first embodiment of the photo restoration system proposed in this invention includes a photo extraction unit 1, a damage extraction unit 2, a damage repair unit 3, a face detection unit 4, and a face repair unit 5, which corresponds to the first embodiment of the photo restoration method proposed in this invention.

[0063] The photo extraction unit 1 is used to extract photo regions from an image.

[0064] The damage extraction unit 2 is used to extract damaged, creased, and scratched areas from the photo area.

[0065] The damage repair unit 3 is used to repair damaged, creased, and scratched areas in the photo area.

[0066] The face detection unit 4 is used to detect face regions in the photo area.

[0067] The face restoration unit 5 is used to remove various degradations in the face region and add face details in the face region.

[0068] Please see Figure 6 The second embodiment of the photo restoration system proposed in this invention adds a non-face magnification unit 6 based on the first embodiment, which corresponds to the second embodiment of the photo restoration method proposed in this invention.

[0069] The non-face magnification unit 6 is used to repair and magnify the parts of the photo area other than the face area (i.e., the non-face area).

[0070] Please see Figure 7 The third embodiment of the photo restoration system proposed in this invention adds a color judgment unit 7 and an automatic coloring unit 8 to the first embodiment, which corresponds to the third embodiment of the photo restoration method proposed in this invention.

[0071] The coloring determination unit 7 is used to determine whether a photo area needs coloring. If coloring is required, the photo area is sent to the automatic coloring unit 8. If coloring is not required, the entire restoration process ends.

[0072] The automatic coloring unit 8 is used to automatically color the areas of the photo that need to be colored.

[0073] Please see Figure 8The fourth embodiment of the photo restoration method proposed in this invention includes all the units in embodiments two and three, and corresponds to the fourth embodiment of the photo restoration method proposed in this invention. Each unit has been described in embodiments one, two, and three respectively, and the content is the same, so it will not be repeated here.

[0074] Compared with existing photo restoration technologies, this invention achieves the following beneficial effects: First, this invention can repair damage, scratches, and creases in photos. Second, this invention provides realistic and natural facial restoration results, enhancing facial details while preserving facial identity features. Third, embodiments three and four of this invention can automatically determine whether a photo needs colorization and automatically colorize photos that require colorization. Fourth, each step of this invention is preferably implemented using a neural network, resulting in fast overall restoration speed; a single image can be restored in just 3 seconds.

[0075] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of photo retouching, characterized by, The method comprises the following steps; Step S1: extracting a photo region in a picture by using a neural network for image segmentation; Step S2: extracting a damage, crease and scratch region in the photo region by using a neural network for image segmentation; Step S3: repairing the damage, crease and scratch region in the photo region by using a neural network for image inpainting; Step S4: detecting and locating a face region in the photo region by using a face detection algorithm; Step S5: removing various degradations in the face region by using an end-to-end neural network for image denoising or image deblurring; the end-to-end neural network comprises an encoder and a decoder, wherein a feature map layer in the decoder is referred to as a first feature map layer; then, a generative adversarial network for image processing is used, the generative adversarial network comprises a generator and a discriminator, wherein a feature map layer in a decoder in the generator is referred to as a second feature map layer; the first feature map layer and the second feature map layer are fused to increase face details in the face region after removing the degradations; The first feature map layer comprises face identity features in the original photo; The second feature map layer is used to improve and increase more face details; After step S5, steps S7 and S8 are sequentially added; Step S7: determining whether the photo region needs coloring by using a neural network for classification; if the photo region needs coloring, step S8 is entered; if the photo region does not need coloring, the whole repairing process is ended; Step S8: automatically coloring the photo region needing coloring by using a neural network for image style conversion; The neural networks in steps S7 and S8 reuse the same training data.

2. The photo retouching method according to claim 1, characterized in that, The implementation of step S1 is as follows: a neural network for image segmentation is used and trained, and the trained neural network is used to extract the photo region in the picture; In step S1, the training method of the neural network is as follows: (1) generating training data; generating a picture containing a photo by using an existing photo, referred to as a synthetic picture, wherein the position of the photo region in each synthetic picture is specified and known; or collecting some pictures containing photos, wherein the position of the photo region in each picture is manually annotated and known; (2) training the neural network by using the training data, so that the neural network can be used to extract the photo region in the picture.

3. The photo retouching method according to claim 2, characterized in that, The training data comprises the following four types: black-and-white photos synthesized into a black-and-white background, black-and-white photos synthesized into a color background, color photos synthesized into a black-and-white background, and color photos synthesized into a color background; the black-and-white background and the color background refer to a non-photo region in the picture.

4. The photo retouching method of claim 1, wherein The implementation of step S2 is as follows: a neural network for image segmentation is used and trained, and the trained neural network is used to extract the damage, crease and scratch region in the photo region; In the step S2, the neural network is trained in the following way: (1) generating training data; collecting some photos with known damage, crease, and scratch regions; or generating some photos with specified damage, crease, and scratch regions; (2) training the neural network with the training data so that the neural network can be used to extract damage, crease, and scratch regions in photos.

5. The photo retouching method of claim 1, wherein, In the step S3, the neural network is trained in the following way: (1) generating training data; adding some damage, crease, and scratch regions to specified positions of some photos, and the original pixel content of the photos covered by the damage, crease, and scratch regions is known; (2) training the neural network with the training data so that the neural network can be used to repair damage, crease, and scratch regions in photos. In the step S3, the neural network for image repair containing convolution operation is used, and the convolution operation is replaced by fast Fourier convolution operation.

6. The photo retouching method of claim 5, wherein, In the step S5, the end-to-end neural network for image denoising or image deblurring is trained with training data generated in multiple small-amplitude degradation ways, and the trained neural network is used to remove multiple degradations in the face region of the photo region; the end-to-end neural network contains an encoder and a decoder; a generative adversarial network for image processing is used to add face details to the face region after removing the degradations, and the generative adversarial network is used to generate high-definition face images based on face prior information; the generative adversarial network contains a generator and a discriminator, and the generator contains an encoder and a decoder; the feature map layer in the decoder of the end-to-end neural network is fused with the feature map layer in the decoder of the generator in the conditional adversarial generative network; 7. The photo retouching method of claim 1, wherein, In the step S5, the end-to-end neural network is trained in the following way: (1) generating training data in multiple small-amplitude degradation ways; this means that first, one or more small-amplitude degradations are added to the photo, which is called the first-order degradation; then, one or more small-amplitude degradations are added to the photo after the first-order degradation, which is called the second-order degradation; in this way, multiple degraded photos are obtained; the original non-degraded photo content is known, and the pixel points of the multiple degraded photos correspond one-to-one to the original non-degraded photo; the small-amplitude degradation means that a single degradation does not significantly reduce the image quality; (2) training the end-to-end neural network with the training data generated in the multiple small-amplitude degradation ways so that the end-to-end neural network can be used to remove various degradations in the photo. After the step S4, the step S6 is added; the step S5 and the step S6 or any preceding step are performed simultaneously or sequentially.

8. The photo retouching method of claim 1, wherein, in In the step S6, the parts of the photo region except the face region are repaired and enlarged. ​ 9. The photo retouching method of claim 8, wherein, The step S6 adopts an image super-resolution reconstruction or restoration technology based on a generative adversarial network.

10. The photo retouching method of claim 1, wherein, The step S7 includes black-and-white photos, faded color photos, and color-biased color photos that need to be colored; and color photos with bright and accurate colors that do not need to be colored. The step S8 includes converting black-and-white photos into color photos, converting faded color photos into bright color photos, and converting color-biased color photos into color photos with accurate colors.

11. The photo retouching method of claim 10, wherein, The step S7 is implemented as follows: a neural network for classification is adopted and trained, and the trained neural network is used to determine whether a photo region needs to be colored. The step S7 is implemented as follows: a neural network for classification is adopted and trained, and the trained neural network is used to determine whether a photo region needs to be colored.

12. The photo retouching method of claim 11, wherein, The step S7 is implemented as follows: a neural network for classification is adopted and trained, and the trained neural network is used to determine whether a photo region needs to be colored.

13. The photo retouching method of claim 10, wherein, The step S8 only fills the color channel of the photo, without changing the brightness channel of the photo.

14. The photo retouching method according to claim 11 or 12, characterized in that, The step S8 is implemented as follows: a neural network for image style conversion is adopted and trained, and the trained neural network is used to automatically color the photos that need to be colored. The step S8 is implemented as follows: a neural network for image style conversion is adopted and trained, and the trained neural network is used to automatically color the photos that need to be colored.

15. The photo retouching method of claim 8 or 10, wherein, in the step of The step S8 is implemented as follows: a neural network for image style conversion is adopted and trained, and the trained neural network is used to automatically color the photos that need to be colored. The step S4 is followed by the step S6; the step S5 and the step S6 are performed simultaneously or in any order; and the step S7 and the step S8 are sequentially added after the step S5 and the step S6.

16. A photo retouching system characterized by, The system comprises a photo extraction unit, a damage extraction unit, a damage repair unit, a face detection unit, a face repair unit, a coloring judgment unit and an automatic coloring unit. The photo extraction unit extracts photo regions in a picture using a neural network for image segmentation. The damage extraction unit extracts damage, crease and scratch regions in the photo regions using a neural network for image segmentation. The damage repair unit repairs the damage, crease and scratch regions in the photo regions using a neural network for image repair. The face detection unit detects and locates face regions in the photo regions using a face detection algorithm. The face repair unit removes various degradations in the face regions using an end-to-end neural network for image denoising or image deblurring; the end-to-end neural network comprises an encoder and a decoder, wherein the feature map layer in the decoder is referred to as a first feature map layer; then a generative adversarial network for image processing is used, which comprises a generator and a discriminator, wherein the feature map layer in the decoder of the generator is referred to as a second feature map layer; the first feature map layer and the second feature map layer are fused to increase face details in the face regions after removing the degradations. The first feature map layer contains face identity features in the original photo. The second feature map layer is used to improve and increase more face details. The coloring judgment unit judges whether the photo regions need coloring using a neural network for classification; if coloring is needed, the photo regions are sent to the automatic coloring unit; if coloring is not needed, the whole repair process ends. The automatic coloring unit automatically colors the photo regions that need coloring using a neural network for image style conversion. The neural networks in the coloring judgment unit and the automatic coloring unit reuse the same training data.

17. The photo retouching system of claim 16, wherein, A non-face enlargement unit is added. The non-face enlargement unit repairs and enlarges the parts in the photo regions other than the face regions.

Images