Face repairing method and system

Abstract

The application discloses a face repairing method, comprising the following steps. Step S1: extracting a photo region in a picture. Step S2: detecting and positioning a face region in the photo region. Step S3: segmenting the face region to obtain a facial feature region and a skin region. Step S4: removing various degenerations in the facial feature region and the skin region, and adding face details in the facial feature region and the skin region. The application improves the low-quality face repairing efficiency and effect.

Description

Technical Field

[0001] This invention relates to a method for restoring low-quality facial areas in photographs. Background Technology

[0002] With the widespread use of smartphones, a large number of photos containing faces are taken and saved. Due to limitations in smartphone image quality and shooting environment, some photos suffer from low-quality issues such as blurriness, out-of-focus images, and compression of facial areas, requiring face restoration to improve image quality. Existing face restoration methods typically employ standalone end-to-end neural networks or generative adversarial networks (GANs), which struggle to restore complex, degraded faces in real-world scenes. Summary of the Invention

[0003] The technical problem to be solved by this invention is to address the low image quality issues such as blurriness, out-of-focus, and compression in photos containing human faces during the shooting process, and to propose a method for repairing low-quality human face areas.

[0004] To address the aforementioned technical problems, this invention discloses a face restoration method, comprising the following steps: Step S1: Extracting the photo region from the image. Step S2: Detecting and locating the face region within the photo region. Step S3: Segmenting the face region to obtain the facial features region and the skin region. Step S4: Removing various degradations in the facial features region and the skin region, and adding facial details to the facial features region and the skin region. The above method divides face restoration (step S4) into two stages: coarse restoration and fine restoration, thus improving the effectiveness of face restoration.

[0005] Further, step S1 is implemented as follows: a neural network for image segmentation is used and trained, and the trained neural network is used to extract photo regions from the image. In step S1, the training method of the neural network is as follows: (1) generating training data; using existing photos to generate images containing photos, called synthetic images, the position of the photo region in each synthetic image is specified, i.e., known; or collecting some images containing photos, the position of the photo region in each image is manually labeled, i.e., known; (2) using the training data to train the neural network so that the neural network can be used to extract photo regions from the image.

[0006] Preferably, the training data includes the following four categories: black and white photos composited onto a black and white background, black and white photos composited onto a color background, color photos composited onto a black and white background, and color photos composited onto a color background; the background refers to the non-photo area in the image.

[0007] Furthermore, in step S2, the Retinaface face detection algorithm is used to detect and locate all face regions in the photo.

[0008] Furthermore, in step S3, a face segmentation algorithm, Face Parsing, is used, or a neural network for image segmentation is used and trained, and the trained neural network is used to segment the face region.

[0009] Further, step S4 is implemented as follows: an end-to-end neural network for image denoising or deblurring is used and trained with training data generated using multiple small-amplitude degradation methods. The trained neural network is then used to remove multiple degradations from the face region in the photo area. The end-to-end neural network includes an encoder and a decoder. A generative adversarial network for image processing is used to add facial details to the degraded face region. The generative adversarial network is used to generate high-definition face images based on prior facial information. The generative adversarial network includes a generator and a discriminator. The generator includes an encoder and a decoder. Feature fusion is performed between the feature layer in the decoder of the end-to-end neural network and the feature layer in the decoder of the generator in the conditional adversarial generative network. In step S4, the training method of the end-to-end neural network is as follows: (1) Training data is generated in a multiple small-amplitude degradation manner; this means that one or more small-amplitude degradations are added to the photo first, called the first degradation; then one or more small-amplitude degradations are added to the photo after the first degradation, called the second degradation; and so on, to obtain the photo after multiple degradations; the content of the original undegraded photo is known, and the pixels of the photo after multiple degradations correspond one-to-one with the pixels of the original undegraded photo; the small-amplitude degradation means that a single degradation does not significantly reduce the image quality; the multiple refers to one to six layers; (2) The end-to-end neural network is trained using the "training data generated in a multiple small-amplitude degradation manner" so that the end-to-end neural network can be used to remove various degradations in the image.

[0010] Furthermore, in step S4, the end-to-end neural network is either MPRNet or pixel2pixel; the generative adversarial network is either CycleGAn, StyleGAN, StyleGAN2, or BEGAN.

[0011] Optionally, step S5 may be added after step S4. Step S5: Perform skin resurfacing and / or whitening on the skin area.

[0012] Preferably, a bilateral filter is used to calculate the blurred result of the pixels in the skin area to achieve a skin smoothing effect.

[0013] This invention also discloses a face restoration system, including a photo extraction unit, a face detection unit, a face segmentation unit, and a face restoration unit. The photo extraction unit is used to extract photo regions from an image. The face detection unit is used to detect and locate face regions within the photo regions. The face segmentation unit is used to segment the face regions to obtain facial feature regions and skin regions. The face restoration unit is used to remove various degradations in the facial feature regions and skin regions, and to add facial details to the facial feature regions and skin regions.

[0014] Optionally, the face restoration system further includes a skin beautification unit. The skin beautification unit is used to perform skin smoothing and / or whitening operations on skin areas.

[0015] The technical effect achieved by this invention is to improve the efficiency and effect of low-quality face restoration, and to provide support for subsequent image editing. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating an embodiment of the face restoration method proposed in this invention.

[0017] Figure 2 This is a flowchart illustrating Embodiment 2 of the face restoration method proposed in this invention.

[0018] Figure 3 This is a schematic diagram of the structure of a first embodiment of the face restoration system proposed in this invention.

[0019] Figure 4 This is a schematic diagram of the structure of Embodiment 2 of the face restoration system proposed in this invention.

[0020] The following are the annotations in the figure: 1 is the photo extraction unit, 2 is the face detection unit, 3 is the face segmentation unit, 4 is the face restoration unit, and 5 is the skin beautification unit. Detailed Implementation

[0021] Please see Figure 1 The first embodiment of the face restoration method proposed in this invention includes the following steps.

[0022] Step S1: Extract the photo region from the image. The photo region is the set of pixels in the image that appear as a photo, and can be regular or irregular in shape, such as a quadrilateral. At least a portion of the image is a photo region, or the entire image may be a photo region. If no photo region is detected and located in the image, the entire image is considered a photo region. Subsequent face detection and facial feature / skin region segmentation are performed only within the photo region, which helps improve the accuracy of subsequent face detection and facial feature / skin region segmentation. If subsequent face detection and facial feature / skin region segmentation are performed on the entire image, and the area of ​​the photo region is too small compared to the entire image area, it will inevitably lead to inaccurate face detection and decreased accuracy of facial feature / skin region segmentation.

[0023] An exemplary implementation of step S1 is as follows: A neural network (NN) for image segmentation is employed and trained. The trained neural network is then used to extract regions from the image. Neural network is short for Artificial Neural Network (ANN). Examples of neural networks used in this step include BiseNet, U-Net, and U... 2 The neural network can be any one of the following: -Net, UNet++, DeepLabv3 series, or SegFormer series. Regardless of the specific neural network used, the training data and methods are the same; the differences lie only in model size (space occupied), computational speed, and computational precision (accuracy). Preferably, after detecting and locating the photo region in the image, the neural network used for image segmentation crops the photo region through perspective transformation. The perspective-transformed photo region is rectangular; this is the ROI (region of interest) commonly found in image processing.

[0024] In step S1, the training method of the neural network is as follows: (1) Generate training data. Generate images containing photos using existing photos. These images are called composite images. The location of the photo region in each composite image is specified, i.e., known. Alternatively, collect some images containing photos, where the location of the photo region in each image is manually labeled, i.e., known. The non-photo region in each image is called the background. (2) Train the neural network using the training data so that the neural network can be used to extract photo regions from images. Preferably, the training data includes the following four categories: black and white photos composited onto a black and white background, black and white photos composited onto a color background, color photos composited onto a black and white background, and color photos composited onto a color background; the combination of these four types of training data helps to improve the accuracy of the neural network in extracting photo regions from images.

[0025] Step S2: Detect and locate face regions within the photo area. This step, for example, uses the Retinaface face detection algorithm to detect and locate all face regions in the photo. The resulting face regions are, for example, rectangular regions containing faces. If no face regions are detected and located within the photo area, the entire method ends.

[0026] Step S3: Segment the face region to obtain the facial features region and the skin region. The facial features region is the set of pixels displayed as eyes, eyebrows, nose, lips, teeth, and ears in the face region, and optionally also includes the set of pixels displayed as hair in the face region. The skin region is the set of pixels displayed as facial skin in the face region, and optionally also includes the set of pixels displayed as neck in the face region. This step can, for example, use the face segmentation algorithm Face Parsing, or it can use a neural network for image segmentation and train it to segment the face region. Examples of neural networks used in this step include BiseNet, U-Net, and U... 2 Choose any one of the following: -Net, UNet++, DeepLabv3 series, or SegFormer series. This step uses a neural network for face segmentation, which has higher accuracy compared to traditional manual segmentation.

[0027] The reason for further segmenting the face region into the facial features region and the skin region is twofold: firstly, to limit the scope of face restoration to the facial features region and the skin region; and secondly, to allow for beautification of the skin region while minimizing beautification of the facial features region, so as to preserve as much detail as possible in the facial features region.

[0028] Step S4: Remove various degradations in the facial features and skin areas. After completion, facial details may be missing in the facial features and skin areas, but the facial identity features of the original photo will still be retained. Then, facial details are added to the facial features and skin areas. The degradations refer to defects in the image, including noise, blur, JPEG compression, downsampling, pixel jitter, artifacts, fading, etc.

[0029] An exemplary implementation of step S4 is as follows: An end-to-end neural network for image denoising or deblurring is employed and trained using training data generated with multiple small-amplitude degradation methods. The trained neural network is then used as a degradation removal network to remove various degradations from the facial features and skin regions (referred to as coarse face restoration). Examples of end-to-end neural networks used in this step include MPRNet and pixel2pixel (also known as pixel2pix). The end-to-end neural network includes an encoder and a decoder. The degradation removal network collects high-quality face images and synthesizes low-quality degradation data, which is used as training data for the encoder and decoder to learn the mapping relationship from blurry low-quality images to clear high-quality images. Subsequently, a generative adversarial network (GAN) for image processing is used to add facial details to the degraded facial features and skin regions (referred to as fine face restoration). Examples of GANs used in this step include CycleGAn, StyleGAN, StyleGAN2, and BEGAN. These GANs are used to generate high-definition face images based on rich prior facial information. The generative adversarial network (GAN) includes a generator and a discriminator. The discriminator constrains the generator to generate high-resolution facial images and learns prior facial information. The generator includes an encoder and a decoder. Adding facial details involves fusing features from the decoder in the end-to-end neural network with features from the decoder in the generator within the GAN. The former feature layer contains facial identity features from the original facial features and skin regions, but may lose some facial details. The latter feature layer is used to refine and add more facial details (texture). Using only an end-to-end neural network may result in lost facial details, such as partially missing facial contours that are difficult to recover. Using only a GAN may result in lost facial identity features. This invention fuses the features of both, thus balancing facial details and facial identity features.

[0030] In step S4, the training method for the end-to-end neural network is as follows: (1) Training data is generated using multiple small-amplitude degradation. First, high-quality (clear, noise-free, compression artifact-free, and fade-free) photos containing human faces are collected. Small-amplitude degradation or one or more degradations are added to the photos, which is called the first degradation. Then, small-amplitude degradation or one or more degradations are added to the photos after the first degradation, which is called the second degradation. And so on, to obtain photos after multiple degradation. Multiple degradation simulates the real degradation process in nature and obtains low-quality (blurry, noisy, compression artifact-free, and faded) photos containing human faces that are consistent with the distribution of real old photos. The content of the original undegraded photos is known. The pixels of the photos after multiple degradation correspond one-to-one with the pixels of the original undegraded photos, which serve as paired training datasets. The small-amplitude degradation refers to a single degradation that does not significantly reduce the image quality. For example, for JPEG compression, it means that the single JPEG compression quality factor is ≥0.65 and <1, preferably it means that the single JPEG compression quality factor is ≥0.90 and <1. The term "multiple" refers to a range from one to six levels, with three levels being preferred. (2) The end-to-end neural network is trained using the "training data generated in a multiple small-amplitude degradation manner" so that the end-to-end neural network can be used to remove various degradations in the photo. Existing end-to-end neural networks can usually only be used to remove one type of degradation, such as for removing image noise. This invention generates training data in a multiple small-amplitude degradation manner, which realistically simulates the degradation process of the photo; the end-to-end neural network trained using this training data can be used to remove multiple degradations simultaneously, which is a major technical innovation of this invention.

[0031] The above embodiment 1 achieves image quality improvement for the face region. After step S4 is completed, the facial features in the facial features and skin regions are preserved, and a large amount of facial details from prior facial information are added.

[0032] Please see Figure 2 The second embodiment of the face restoration method proposed in this invention fully includes steps S1 to S4 of the first embodiment, and adds step S5 after step S4.

[0033] Step S5: Perform skin smoothing and / or whitening operations on the skin area. This step, for example, uses a bilateral filter to calculate the blurred result of the pixels in the skin area to achieve a skin smoothing effect. Bilateral filtering uses a weighted averaging method, where the intensity of a pixel is represented by a weighted average of the brightness values ​​of surrounding pixels, which can preserve edge information in the image while blurring.

[0034] The above-described embodiment two adds beautification to the skin area based on embodiment one, but does not beautify the facial features area, thereby preserving the facial feature information of the facial features area.

[0035] Please see Figure 3 The first embodiment of the face restoration system proposed in this invention includes a photo extraction unit 1, a face detection unit 2, a face segmentation unit 3, and a face restoration unit 4, which corresponds to the first embodiment of the face restoration method proposed in this invention.

[0036] The photo extraction unit 1 is used to extract photo regions from an image.

[0037] The face detection unit 2 is used to detect and locate the face region in the photo area.

[0038] The face segmentation unit 3 is used to segment the face region to obtain the facial features region and the skin region.

[0039] The face restoration unit 4 is used to remove various degenerations in the facial features and skin areas, and to add facial details in the facial features and skin areas.

[0040] Please see Figure 4 The second embodiment of the face restoration system proposed in this invention adds a skin beautification unit 5 based on the first embodiment, which corresponds to the second embodiment of the face restoration method proposed in this invention.

[0041] The skin beautification unit 5 is used to perform skin resurfacing and / or whitening operations on skin areas.

[0042] Compared with existing face restoration technologies, this invention achieves the following beneficial effects. First, the face restoration effect of this invention is realistic and natural, enhancing facial details while preserving facial identity features. Second, each step of this invention is preferably implemented using a neural network, resulting in fast overall restoration speed and high efficiency.

[0043] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A face repairing method, characterized in that, The method comprises the following steps: Step S1: extracting a photo region in a picture by using a neural network for image segmentation; Step S2: detecting and locating a face region in the photo region by using a face detection algorithm; Step S3: segmenting the face region by using a face segmentation algorithm or by using a neural network for image segmentation to obtain a feature region and a skin region; Step S4: removing various degradations in the feature region and the skin region by using an end-to-end neural network for image denoising or image deblurring; the end-to-end neural network comprises an encoder and a decoder, and a feature map layer in the decoder is referred to as a first feature map layer; Then, a generative adversarial network for image processing is used, the generative adversarial network comprises a generator and a discriminator, a feature map layer in a decoder in the generator is referred to as a second feature map layer; the first feature map layer and the second feature map layer are fused to add face details to the feature region and the skin region after removing the degradations; The first feature map layer contains face identity features in the original photo; The second feature map layer is used to add more face details; Step S5: performing skin smoothing and / or whitening on the skin region by using a bilateral filter.

2. The face repairing method of claim 1, wherein, The step S1 is implemented in the following manner: a neural network for image segmentation is used and trained, and the trained neural network is used to extract a photo region in a picture; In the step S1, the neural network is trained in the following manner: (1) generating training data; generating a picture containing a photo by using an existing photo, the picture is referred to as a synthetic picture, and the position of the photo region in each synthetic picture is specified and known; or collecting some pictures containing photos, the position of the photo region in each picture is manually annotated and known; (2) training the neural network by using the training data, so that the neural network can be used to extract a photo region in a picture.

3. The face repairing method of claim 2, wherein, The training data comprises the following four types: black-and-white photos are synthesized into a black-and-white background, black-and-white photos are synthesized into a color background, color photos are synthesized into a black-and-white background, and color photos are synthesized into a color background; the black-and-white background and the color background refer to a non-photo region in a picture.

4. The face repairing method of claim 1, wherein, In the step S2, a face detection algorithm Retinaface is used to detect and locate all face regions in a photo.

5. The face repairing method of claim 1, wherein, In the step S3, a face segmentation algorithm Face Parsing is used, or a neural network for image segmentation is used and trained, and the trained neural network is used to segment a face region. In the step S3, a face segmentation algorithm Face Parsing is used, or a neural network for image segmentation is used and trained, and the trained neural network is used to segment a face region.

6. The face repairing method of claim 1, wherein, The implementation of the step S4 is as follows: a kind of end-to-end neural network for image denoising or image deblurring is trained using training data generated in multiple small amplitude degradation ways, and the trained neural network is used to remove multiple degradations in the face region in the photo area;The end-to-end neural network contains an encoder and a decoder;A generative adversarial network for image processing is used to add face details in the face region after removing degradation, which is used to generate high-definition face images based on face prior information;The generative adversarial network contains a generator and a discriminator, and the encoder and the decoder in the generator contain feature fusion of the feature map layer in the decoder in the end-to-end neural network and the feature map layer in the decoder in the generator in the conditional adversarial generative network; In the step S4, the training method of the end-to-end neural network is as follows: (1) generate training data in multiple small amplitude degradation ways; this means that first add one or more small amplitude degradations in the photo, called the first degradation; then continue to add one or more small amplitude degradations in the photo after the first degradation, called the second degradation; in this way, the photo after multiple degradations is obtained; the original un-degraded photo content is known, and the pixel points of the photo after multiple degradations correspond one-to-one with the original un-degraded photo; the small amplitude degradation means that single degradation does not significantly reduce image quality; (2) use the training data generated in multiple small amplitude degradation ways to train the end-to-end neural network, so that the end-to-end neural network can be used to remove various degradations in the picture.

7. The face repairing method of claim 6, wherein, In the step S4, the end-to-end neural network is one of MPRNet or pixel2pixel; the generative adversarial network is one of CycleGAn, StyleGAN, StyleGAN2, BEGAN.

8. The face repairing method of claim 1, wherein, In the step S5, a bilateral filter is used to calculate the blurred result of the pixels in the skin area to achieve the skin smoothing effect.

9. A face restoration system, characterized by, It includes a photo extraction unit, a face detection unit, a face segmentation unit, a face repair unit, and a skin beautification unit. The photo extraction unit is used to extract a photo area in the picture using a neural network for image segmentation. The face detection unit is used to detect and locate the face area in the photo area using a face detection algorithm. The face segmentation unit is used to segment the face area using a face segmentation algorithm or a neural network for image segmentation to obtain the feature area and the skin area. The face repair unit is used to remove multiple degradations in the feature area and the skin area using an end-to-end neural network for image denoising or image deblurring; the end-to-end neural network contains an encoder and a decoder, and the feature map layer in the decoder is called the first feature map layer. Subsequently, a generative adversarial network for image processing is used, the generative adversarial network comprises a generator and a discriminator, a feature map layer in a decoder in the generator is referred to as a second feature map layer; the first feature map layer is fused with the second feature map layer, and face details are added in the removed degenerated facial feature region and skin region; The first feature map layer comprises a face identity feature in the original photo; The second feature map layer is used to perfect and add more face details; The skin beautifying unit is configured to perform skin smoothing and / or whitening operation on the skin region by using a bilateral filter.

Images