Face restoration method based on multi-scale face analysis graph

Abstract

The application discloses a face restoration method based on a multi-scale face analysis graph, and belongs to the technical field of computer vision, machine learning and the like. First, the application uses an encoder-decoder structure to build a basic network, and adds a skip connection with a channel attention module to the basic network, so that the decoder can fully utilize the effective information contained in the extracted feature map to restore the face. Meanwhile, the face analysis graph is taken as a style image to be fused into the generated face image. Finally, the application adds a loss function that is beneficial to saving identity information during training, and uses the feature layer extracted by the face recognition network to constrain the identity of the restored face, and adds an adversarial loss and a multi-scale discriminator, so that the network can further generate a face image with more real details.

Description

Technical Field

[0001] This invention belongs to the technical fields of computer vision and machine learning, and in particular relates to a face restoration method based on deep learning. Background Technology

[0002] Face restoration is a crucial problem in computer vision. Face restoration technology is a method that enables the recovery of high-quality face images from low-quality ones, laying the foundation for advanced applications such as face recognition and expression recognition. Currently, face restoration technology is applied in commonly used image processing software, the restoration of old photos and films, zoom compensation in digital cameras, and intelligent surveillance systems.

[0003] Based on whether or not GAN prior information is used, most current face restoration methods can be divided into two categories: The first category utilizes a pre-trained StyleGAN model as a GAN prior and uses it as the main network design. The advantage of this type of method is superior visual quality; it can obtain very high-resolution output results even from extremely low-quality input face images, making it a popular approach in recent years. However, the core idea of ​​these GAN-based methods is to encode the degraded face image into the latent space of the pre-trained GAN. This requires designing complex modules to modify the encapsulated GAN prior or fine-tuning the GAN prior through additional training. Therefore, this results in a large number of model parameters, placing high demands on the GPU and system memory of the training device. Furthermore, due to the low dimensionality of the latent space of the pre-trained GAN, its spatial representation ability is poor, failing to fully capture the facial structure of the degraded face image, resulting in an unnatural facial structure in the restored image.

[0004] The second category is face restoration networks that use other prior information or do not use prior information. The advantage of this type of method is that the number of network parameters is small and it is easy to train, but there are two common problems: (1) The high-definition output image cannot preserve the identity information of the original input image. The ultimate goal of face restoration is to perform deep vision tasks such as face detection and recognition, so the identity information of the face in the image is crucial for both humans and network models. However, most existing methods of this type only focus on the structural design of the generative network, aiming to obtain images with better visual quality, without considering the need to fully extract and effectively utilize the information in the low-quality input image, resulting in the problem of inconsistency between the identity of the input image and the restored image. (2) Facial structural information cannot be effectively utilized. The structural information used in the network, such as face parsing maps and face key point maps, is often extracted from low-quality input images. The information contained in the low-definition image itself is relatively coarse. If a simple facial structure estimation network is used directly, the estimated prior image will not be accurate enough, thus directly affecting the restoration result. Furthermore, current methods for utilizing facial structural information often employ direct feature stitching and attention mechanisms, which fail to fully leverage global and local information, thus limiting the assistance that facial structural information can provide for face reconstruction.

[0005] Based on the above analysis, we hope to design a face restoration network that meets the following three requirements: (1) The network model has few parameters and is easy to train, i.e., it does not use GAN priors. (2) It can effectively utilize the pixel and structural information in the input low-resolution image, i.e., fully extract the pixel information in the low-resolution face and make reasonable use of the face prior information. (3) It can design a loss function that helps to preserve identity information and maintain image fidelity. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a face reconstruction network that fully utilizes facial structure and identity information, thereby improving the fidelity of the reconstructed face. First, this invention uses the Encoder-Decoder structure commonly used for image restoration to build the basic network. The encoder progressively extracts features from the low-resolution input face image, while the decoder progressively upsamples to the same resolution as the input image. Second, skip connections with channel attention modules are added to the basic network, allowing the decoder to fully utilize the effective information contained in the extracted feature maps for face reconstruction. Simultaneously, a face parsing map is selected as the structural prior. To efficiently integrate the prior information into the network, this invention borrows the style transfer method AdaIN to incorporate the face parsing map as a style image into the generated face image. Finally, this invention incorporates a loss function that preserves identity information during training, constraining the reconstructed face identity at the feature level extracted by the face recognition network. Adversarial loss and a multi-scale discriminator are also added, enabling the network to generate face images with more realistic details.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A face restoration method based on an encoder-decoder structure incorporating multi-scale face parsing images is characterized by: adjusting the original low-resolution face image to a face image with a resolution of 512×512 and then inputting it into the face restoration network to obtain a restored high-resolution face image with a resolution of 512×512.

[0009] The face restoration network includes an initialization layer, a backbone network, and an RGB conversion layer connected in sequence.

[0010] The initialization layer is used to adjust a face image with a resolution of 512×512 to obtain a feature map F0 of 512×512×32.

[0011] The backbone network adopts an encoder-decoder structure with added skip connections;

[0012] The encoder-decoder structure includes a fourth downsampling block, a third downsampling block, a second downsampling block, a first downsampling block, a first upsampling block, a second upsampling block, a third upsampling block, and a fourth upsampling block connected in sequence. The feature map F0 is gradually reduced in resolution to 32×32 through four downsampling blocks, and then gradually increased in resolution through four upsampling blocks to generate a feature map of the same size as the feature map F0.

[0013] The skip connection method is as follows: the feature map output by the downsampling block Feature map with the same resolution as the upsampling block output Feature fusion is performed through vector concatenation, where the feature map output by the first downsampling block... Feature fusion is performed by concatenating the feature vector with its own vector; the feature map is then obtained by dimensionality reduction after feature fusion. As input to the next upsampling block;

[0014] The RGB conversion layer is used to process feature maps of size 512×512×32. The conversion yields a high-resolution face image with a resolution of 512×512.

[0015] Preferably, based on the backbone network, feature map F′ i Up The feature map F obtained after style modulation branching i Up As the input feature map for the next upsampling block, the structural information in the face parsing map is lightweightly integrated into the face reconstruction network;

[0016] Specifically, the style modulation branch includes a convolutional layer, an activation layer, and a two-branch structure connected in sequence; the two-branch structure consists of two identical convolutional layers, and the two branches respectively output style parameter a. i With b i And it is integrated into the face reconstruction network through the following formula:

[0017]

[0018] Where μ represents the mean and σ represents the variance.

[0019] Preferably, during the skip connection process, a channel attention module is inserted to select features that are helpful for face reconstruction.

[0020] Preferably, a multi-scale discriminator is used to stabilize the training process during the training of the face reconstruction network.

[0021] The beneficial effects of this invention are as follows:

[0022] (1) A face restoration network based on encoder-decoder structure was built, and a skip connection with added channel attention mechanism was added, so that the network can fully extract and utilize the effective information in low-resolution faces.

[0023] (2) In the basic network for face restoration, a style modulation branch that can effectively utilize the face parsing image is added to construct a high-performance face restoration network.

[0024] (3) A loss function was designed that is conducive to preserving texture and identity details. Attached Figure Description

[0025] Figure 1Schematic diagram of backbone network and hop connection network structure.

[0026] Figure 2 .exist Figure 1 A schematic diagram of adding a style modulation branch to the network structure.

[0027] Figure 3 A schematic diagram of a face restoration method based on an encoder-decoder structure incorporating multi-scale face parsing images.

[0028] Figure 4 Schematic diagram of channel attention structure.

[0029] Figure 5 A detailed schematic diagram of the face analysis image integrated into the module.

[0030] Figure 6 Qualitative comparison with state-of-the-art face restoration methods on the Helen dataset.

[0031] Figure 7 Quantitative comparison with state-of-the-art face restoration methods on the Helen dataset. Detailed Implementation

[0032] The following provides a detailed description of each of the details involved in the technical solution of this invention.

[0033] (1) Constructing an encoder-decoder face reconstruction network with added skip connections.

[0034] The backbone network of this invention is an encoder-decoder structure with added skip connections.

[0035] The encoder-decoder structure consists of four downsampling blocks and four upsampling blocks. First, the low-resolution face image is resized to 512×512 pixels and then input into the backbone network. The resolution is gradually reduced to 32×32 pixels by the four downsampling blocks, and then the resolution is gradually increased by the four upsampling blocks, finally generating a face image of the same size as the input image.

[0036] Considering that different network layers focus on different features, such as shallow networks focusing on texture features and deep networks focusing on global features, and that both global and texture features are very important for face restoration tasks; and that the loss of some edge features is unavoidable due to downsampling operations, in order to make use of more comprehensive information from low-resolution images, this invention adds skip connections to the backbone network, and performs vector concatenation of feature maps of the same resolution in the encoder and decoder to achieve feature fusion.

[0037] (2) Add a channel attention module to the above face reconstruction network.

[0038] Considering that not all features extracted by the encoder provide effective information for face restoration—for example, the input low-resolution image may contain noise, artifacts, and other invalid information—introducing all of this information can lead to poor restoration, such as patchy areas, in the output restored image. To address this issue, this invention adds a Channel Attention Module (CAM) to the feature map extracted from each downsampling block in the encoder. This CAM learns which features are helpful for face restoration and then concatenates them with the feature map generated from the corresponding upsampling block in the decoder. The working mechanism of the Channel Attention Module is to model the importance of each feature channel and enhance or suppress different channels based on the learned information, providing a function similar to a "filter." This embodiment uses the Channel Attention Module SENe t (Sequeeze and Excitation Net) as an example for illustration.

[0039] like Figure 4 As shown, the feature map F output by the downsampling block i Down The input is fed into a two-branch structure. In the branch that obtains the weights, a pooling layer is first used to compress the spatial dimension of the feature map, that is, to aggregate the various two-dimensional feature maps into a constant. This operation is equivalent to pooling the global receptive field, keeping the number of feature channels unchanged. Then, two fully connected layers are used to establish the correlation between channels, generating weights for each feature channel. Next, an activation layer normalizes the weights. Finally, the weights are compared with the feature map F. i Down Multiplying them alters the relative importance of different channels.

[0040] (3) Add a style modulation branch to the above face restoration network.

[0041] To enable face parsing maps to provide more information for face reconstruction in face restoration networks while remaining lightweight, this invention employs the classic Adaptive Instance Normalization (AdaIN) method from style transfer. Specifically, the face parsing map is used as input, and the style modulation branch includes sequentially connected convolutional layers, activation layers, and a two-branch structure. The two-branch structure consists of two identical convolutional layers, with each branch outputting a style parameter 'a'. i With b i And it is integrated into the face reconstruction network through the following formula:

[0042]

[0043] Where μ represents the mean and σ represents the variance.

[0044] AdaIN, based on a feedforward neural network, offers fast generation speed and supports arbitrary style transfer. Style GAN uses AdaIN to integrate the style of real faces into generated faces, achieving the goal of generating realistic synthetic faces. AdaIN effectively integrates styled face images into synthetic face images. In its implementation, unlike AdaIN which uses the mean and standard deviation of the face resolution image as style parameters, this invention learns these two style parameters through simple convolutional layers. This is because the task is not simply style transfer, but rather aims to provide additional prior knowledge from the face resolution image for face restoration—information beneficial to the final task. Therefore, simple convolutional layers adaptively learn these two style parameters. Each face resolution image input used by the face restoration network can be extracted from the output of the previous upsampled block. This is because higher-resolution face input yields more accurate face resolution images, resulting in higher-quality generated face images.

[0045] (3) Train the model and verify its effectiveness through experiments.

[0046] The constraints on face reconstruction networks can be mainly divided into the following five points:

[0047] (a) Loss of texture detail

[0048] This invention uses the difference between the Gram matrices of different semantic regions in the generated image and the high-resolution image as the texture loss. Specifically, it uses VGG19 to extract features and utilizes ReLU3_1, ReLU4_1, and ReLU5_1 features to calculate the loss. The m-th layer feature in VGG19 is represented as φ. m The analytical masking representation of the 19 regions is M. n So L ss That is:

[0049]

[0050] in, To generate a face image, I H For a real, high-resolution human face image, G(·) is used to calculate the feature φ. m Semantic region masking M n Gram matrix:

[0051]

[0052] Where ε is a constant added to avoid division by zero.

[0053] (b) Reconstruction losses

[0054] Reconstruction loss, also known as generation loss, measures the difference between the output image of the generative network and the real image. The reconstruction loss used in this invention is a combination of pixel and feature space mean square error (MSE), designed to constrain the network's output. As close as possible to the real image I H .

[0055]

[0056] in, The intermediate face image generated for the i-th upsampled block. For true high-definition facial images of the corresponding resolution, L rec The second term is the multi-scale feature matching loss, which matches... and Discriminator features.

[0057] (c) Loss of identity

[0058] To prevent situations where the output image has good qualitative and quantitative indicators but fails to maintain the same identity as the original input image, this invention introduces an identity loss mechanism. This mechanism constrains the distance between the high-dimensional features of the generated image and the real image, thereby improving identity similarity. Specifically, it uses a pre-trained face recognition model, Arcface, to extract the high-dimensional features of the generated image and uses Euclidean distance to measure the difference.

[0059]

[0060] Where φ(·) represents the pre-trained face recognition model Arcface.

[0061] (d) Combating Losses

[0062] To achieve high fidelity in face images generated by the EDSM network, this invention designs a discriminator and a corresponding loss function to extend it into EDSM-GAN. Specifically, the loss function is a non-saturating loss function, which aims to maximize the probability that the generated image is judged as real, thereby providing a larger gradient for the generator in the early stages of GAN training.

[0063]

[0064]

[0065] Among them, L GAN_D For the optimization goal of the discriminator, L GAN_G Let I represent the optimization objective of the generator, and D(·) represent the discriminant. The intermediate face image generated for the i-th upsampled block. This is a true high-definition human face image with the corresponding resolution.

[0066] (e) Training Objectives

[0067] The EDSM-GAN output is obtained by minimizing L... G and L GAN_D To achieve the training objective:

[0068] L G =λ ss L ss +λ rec L rec +λ id L id +λ adv L GAN_G

[0069] Where, λ ss , λ rec , λ id , λ adv These are the weights for texture detail loss, reconstruction loss, identity loss, and adversarial loss, respectively.

[0070] In addition to the constraints mentioned above, to generate more realistic details and improve the subjective performance of the restoration results, this embodiment also incorporates a discriminator. This aims to address a common problem in GAN networks that generate high-resolution images. In the early stages of training, due to the generator's insufficient capabilities and the presence of many difficult-to-imitate details in high-resolution images, the discriminator can easily distinguish between real and fake images, thus failing to provide the generator with reliable feedback. This can easily amplify gradient problems during GAN training, leading to training instability and even model collapse. To circumvent this issue, this invention employs a multi-scale discriminator to stabilize training.

[0071] The multi-scale discriminator consists of multiple sub-discriminators. Each sub-discriminator receives the intermediate face reconstruction image of its corresponding upsampled block and the ground truth value at the corresponding resolution obtained by downsampling the high-resolution image. Each sub-discriminator can only obtain limited information from images at its corresponding resolution; therefore, different sub-discriminators are used to address different levels of discrimination tasks. For example, distinguishing between real and fake generated images at lower resolutions is more difficult than at higher resolutions, making the low-resolution sub-discriminators crucial for stabilizing the early training of the GAN. As training progresses, their role gradually shifts upwards to higher-resolution sub-discriminators, ultimately enabling all discriminators to possess excellent discrimination capabilities and provide effective feedback to the face reconstruction network.

[0072] To more comprehensively demonstrate the visual appeal of the face restoration model, this invention selected three sets of low-resolution images with degradation levels ranging from slight to severe as input. Figure 6This comparison focuses on the full-face reconstruction results on the Helen dataset. It can be seen that when the input image degradation is mild, all face restoration algorithms can achieve visually appealing results. However, when the input image degradation is severe, the performance of all face restoration algorithms declines to varying degrees. Specifically, the method proposed by Wan et al. has no restoration effect on low-resolution input images, and GFPGAN and GCFSR perform poorly, retaining color blocks from the low-resolution image in the restored image. GPEN and the EDSM-GAN proposed in this invention both restore relatively good facial structure and texture details.

[0073] Figure 7 The results show a comparison of image quality metrics for different face restoration algorithms on the Helen dataset. It can be seen that the network proposed in this invention achieves the best results across all four image quality metrics, verifying that the proposed method has excellent restoration performance.

Claims

1. A face restoration method based on a multi-scale face parsing map integrated encoder-decoder structure, characterized in that: The original low-definition face image is adjusted to a face image with a resolution of 512*512, and then input into the face restoration network to obtain a high-definition face image with a resolution of 512*512 after restoration; The face restoration network comprises an initialization layer, a backbone network and an RGB conversion layer connected in sequence; The initialization layer is configured to adjust the face image with a resolution of 512*512 to obtain a feature map F0 of 512*512*32; The backbone network adopts an encoder-decoder structure with a jump connection; The encoder-decoder structure comprises, in sequence, a fourth downsampling block, a third downsampling block, a second downsampling block, a first downsampling block, a first upsampling block, a second upsampling block, a third upsampling block, and a fourth upsampling block. The feature map F0 is gradually reduced in resolution to 32x32 by the four downsampling blocks, and then gradually increased in resolution by the four upsampling blocks to generate a feature map of the same size as the feature map F0 The jump connection mode is that the feature map output by the downsampling block The feature map output by the upsampling block has the same resolution The feature fusion is performed by vector splicing, wherein the feature map output by the first downsampling block The feature fusion is performed by self vector splicing; and the feature fusion is dimensionally reduced to obtain a feature map The feature map The feature map obtained after the style modulation branch The feature map The feature map is input into the next upsampling block, so that the structural information in the face parsing map is lightweightly integrated into the face restoration network The style modulation branch comprises a convolution layer, an activation layer and a two-branch structure connected in sequence; the two-branch structure is composed of two identical convolution layers, and the two branches respectively output style parameters a i and b i , and are integrated into the face restoration network by the following formula: Wherein, μ represents the mean, and σ represents the variance; The RGB conversion layer is used to process feature maps of size 512×512×32. The conversion yields a high-resolution face image with a resolution of 512×512.

2. The encoder-decoder structure-based face restoration method according to claim 1, wherein: In the process of the jump connection, a channel attention module is inserted to select features conducive to face restoration.

3. The encoder-decoder structure based face restoration method according to claim 2, wherein: In the process of training the face restoration network, a multi-scale discriminator is used to stabilize the training.

Images