A face repairing method based on a CAR encoder and face generation prior knowledge

Abstract

The application discloses a face repairing method based on a CAR encoder and face generation prior knowledge, and belongs to the face repairing field.The method comprises the following steps: (1) obtaining a high-definition face image FFHQ; (2) performing multiple manual degenerations on the high-definition face image FFHQ; and (3) pairing the degenerated image with the high-definition image to train a model, so that the face repairing model can enhance the detail features and facial features of a face to be repaired.The repairing model is composed of an encoder and a generator, the encoder is a CAR encoder based on a Transformer self-attention mechanism, and the generator is a pre-trained StyleGAN2 generator, namely face generation prior knowledge.The application balances the relationship between the fidelity and the detail repairing degree of the face repairing model, significantly improves the peak signal-to-noise ratio (PSNR) while enhancing the visual effect.

Description

Technical Field

[0001] This invention belongs to the field of low-light image enhancement technology, specifically relating to a face restoration method based on a CAR encoder and prior knowledge of face generation. Background Technology

[0002] In real-world traffic surveillance scenarios, faces in traffic video surveillance often suffer severe degradation due to factors such as weather and time. This degradation is characterized by low visibility and contrast, severe color distortion, and unpredictable image noise. These low-quality facial images have low recognizability, and the information they contain is often inaccurate or missing. Therefore, facial restoration can help users better understand the facial identity information of traffic drivers, thereby facilitating relevant investigations.

[0003] Current face restoration methods mainly target two types of problems: single degradation and multiple degradation. Methods for single degradation suffer from poor generalization, while methods for multiple degradation suffer from poor restoration detail, image detail distortion, and poor robustness. Considering that the face is a structural object to be restored, current algorithm developers have introduced prior knowledge such as landmark prior information, heatmap prior information, and generation prior information to assist in face image restoration. This invention addresses the traffic monitoring scenario, which falls under the multiple degradation problem. It employs face generation prior knowledge and a Transformer self-attention mechanism to effectively solve the problem of poor face detail restoration in multiple degradation problems. Summary of the Invention

[0004] This invention addresses the problems of existing technologies by providing a face restoration method based on a CAR encoder and prior knowledge of face generation. First, data acquisition is performed, and the training set FFHQ undergoes image degradation processing. Then, a CAR encoder and a StyleGAN2 generator are constructed to build a face restoration network that relies on prior knowledge of face generation. After training this network, low-quality face images are further tested to obtain restored face images. The CAR encoder outputs two types of features: affine features α... i and β i And latent code features. In the pre-trained StyleGAN2 generator, affine features α i and β i The intermediate features of the StyleGAN2 generator were processed using channel separation and affine transformation. The latent code features are controlled by the StyleGAN2 generator, and the latent code features are applied to the modulation convolution in the StyleGAN2 generator. Through the effective features provided by the CAR encoder, this invention effectively improves the visual quality of low-quality traffic monitoring driver face images.

[0005] To achieve the above objectives, the technical solution adopted by this invention is: a face restoration network based on a CAR encoder and prior information for face generation, comprising a CAR encoder and a pre-trained StyleGAN2 generator, the steps of which are as follows:

[0006] Step S1: Training set acquisition and processing;

[0007] Acquire a high-resolution face image FFHQ, and perform multiple manual degradations on the high-resolution face image FFHQ to obtain the degraded image FFHQ-M;

[0008] Step S2: Construct and train the CAR encoder and the pre-trained StyleGAN2 generator;

[0009] The CAR encoder and the pre-trained StyleGAN2 generator constitute the face restoration model. The training mode of generative adversarial network is adopted, and the degraded image FFHQ-M is paired with the high-definition face image FFHQ for face restoration model training.

[0010] Step S3: Output the result;

[0011] Low-quality face images are input into the CAR encoder to obtain corresponding affine features and latent codes, and then the images are processed by the StyleGAN2 generator to generate face restoration images.

[0012] As an improvement of the present invention, the CAR encoder includes three CAR downsampling layers, three CAR upsampling layers, and one CAR feature refinement layer. Each CAR downsampling / upsampling layer is composed of several CAR modules, one convolutional layer (Conv), and one pixelunshuffle downsampling / upsampling module connected sequentially. Specifically, the number of CAR modules in CAR downsampling / upsampling layer-1, CAR downsampling / upsampling layer-2, CAR downsampling / upsampling layer-3, and the intermediate CAR layer corresponds to [2, 3, 3, 4], the number of attention heads corresponds to [1, 2, 4, 8], and the corresponding number of channels is [16, 32, 64, 128]. The feature refinement layer is configured in the same way as CAR downsampling / upsampling layer-1.

[0013] The CAR downsampling layer is composed of several CAR modules, one convolutional layer Convd3, and one pixelunshuffle downsampling module connected in sequence; the CAR upsampling layer is composed of a concatenation operation, a convolutional layer Convp1, several CAR modules, one convolutional layer Convd4, and one pixelunshuffle upsampling module connected in sequence; the CAR feature refinement layer is composed of several CAR modules connected in sequence.

[0014] As an improvement of the present invention, the affine network consists of a 4-layer affine transformation (AFT) module and a 3-layer dual downsampling (DDS) module, which is used to connect the CAR encoder and the StyleGAN2 generator.

[0015] The affine transformation (AFT) module is composed of a convolutional layer Convd10, a LeakyReLU activation layer, and a convolutional layer Convd11 connected in sequence.

[0016] The dual downsampling DDS module consists of two branches: one branch is formed by connecting the Convd13 convolutional layer and the Pixelunshuffle downsampling module in sequence, and the other branch is formed by connecting the Bilinear bilinear downsampling module and the Convd12 convolutional layer in sequence. The two branches are connected in parallel to form the dual downsampling module.

[0017] As an improvement of the present invention, the CAR module is composed of three modules connected in sequence: a multi-head transposition self-attention module MDTA, a gated feedforward network GDFN, and a channel height and width axis self-attention module CHWA.

[0018] The StyleGAN2 generator is divided into 7 levels, with each level corresponding to a pixel value of 512. 2 256 2 128 2 64 2 32 2 16 2 and 8 2 The size is intermediate. Each level consists of one ToRGB module and two cascaded StyleConv modules connected in parallel.

[0019] As an improvement of this invention, the CAR encoder introduces a Transformer self-attention network to construct a CAR module, thereby enhancing the network's ability to capture long-distance feature interactions and compensating for the inherent defects of convolutional networks. The CAR module is composed of a multi-head transposition self-attention module MDTA, a gated feedforward network GDFN, and channel height and width axis attention modules CHWA connected sequentially.

[0020] As an improvement of the present invention, the channel height and width axis self-attention module CHWA is composed of an average pooling layer Avgpool, a nonlinear module LRLS, a broadcast operation and a residual operation cascaded together. The nonlinear module LRLS is composed of a linear layer Linear, a function activation layer ReLU, a linear layer Linear and a function activation layer Sigmoid cascaded together.

[0021] As an improvement of this invention, the channel height and width axis attention module CHWA comprises two sequentially connected sub-modules: the width attention module WAM and the height attention module HAM. Based on the channel attention mechanism, this invention expands the channel-dimensional attention map into a channel-dimensional + width-dimensional attention map and a channel-dimensional + height-dimensional attention map, further enhancing the attention module's ability to learn spatial features.

[0022] As an improvement to the present invention, the dual downsampling module is combined with the affine transformation module to generate the 7-level affine transformation parameters α required for the StyleGAN2 generator. i and β i These represent scaling (scale) and shift (shift), respectively. The affine transformation (AFT) module consists of a convolutional layer Convd10, a function activation layer LeakyReLU, and a convolutional layer Convd11 connected sequentially. One branch of the dual downsampling (DDS) module is composed of a bilinear downsampling module Bilinear and a convolutional layer Convd12 cascaded sequentially, while the other branch is composed of a convolutional layer Convd13 and a downsampling module Pixelunshuffle cascaded sequentially. The two branches are connected in parallel and finally element-wise added to obtain the final output.

[0023] As an improvement to this invention, we propose a supervised loss function. This promotes the convergence of the CAR encoder and accelerates the network training speed. The supervised loss function is:

[0024]

[0025] Let y represent the supervision loss, and y represent the high-resolution image. u λ represents the face restoration image from the CAR encoder. l1 This represents the weights of the loss function.

[0026] As an improvement to this invention, four loss functions are used to train our face restoration model. The four loss functions include (1) supervised loss for supervising intermediate images; (2) adversarial loss to enhance image realism; (3) perceptual loss; and (4) identity loss to ensure identity consistency. The total loss formula is as follows:

[0027]

[0028] in Indicates monitoring losses, Indicates adversarial loss. Indicates perceived loss. This indicates a loss of identity.

[0029] Compared to existing technologies, the advantages of this invention are as follows: This invention innovatively combines a Transformer self-attention network with prior information from face generation, producing face restoration images with high detail restoration and high realism. Based on a channel attention mechanism, this invention proposes a channel height and width axis attention module (CHWA) to further enhance the network's feature representation capabilities. This invention proposes a dual downsampling module to generate affine features, thereby achieving an organic combination of the CAR encoder and the StyleGAN2 generator. The face restoration method designed in this invention can effectively restore low-quality face images, and the enhanced images have good visual effects and a high peak signal-to-noise ratio (PSNR). Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating the face restoration method based on CAR encoder and prior knowledge of face generation in this invention.

[0031] Figure 2 This is a schematic diagram of the CAR encoder and StyleGAN2 generator network in this invention;

[0032] Figure 3 This is a schematic diagram of the CHWA module for channel height and width axes in this invention;

[0033] Figure 4 This is a schematic diagram of the affine network in this invention;

[0034] Figure 5 This is a schematic diagram of the Affine Transformation (AFT) module in this invention;

[0035] Figure 6 This is a schematic diagram of the dual downsampling DDS module in this invention;

[0036] Figure 7 This is a schematic diagram of the StyleGAN2 generator in this invention; Detailed Implementation

[0037] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0038] Example: This example is a face restoration method based on a CAR encoder and prior knowledge of face generation. Its workflow is as follows: Figure 1 As shown, the specific steps include:

[0039] Step S1: Acquisition and processing of the training set;

[0040] High-resolution FFHQ face images were acquired as high-resolution training images and subjected to multiple manual degradation processes to obtain low-quality FFHQ-M face images.

[0041] The image size input in this example is (3, 512, 512), which means there are 3 channels, and the width and height are 512 and 512 respectively.

[0042] The formula (1) for multiple manual degradation is shown below:

[0043]

[0044] First, use a Gaussian blur kernel k σ A high-quality image y is convolved, and then downsampled using a scaling factor r. Afterwards, additive white Gaussian noise n is added. δ The values ​​are added to the image, and finally compressed using JPEG compression with a quality factor of q. For each image, we randomly sample r, δ, and q from {0.2:10}, {1:8}, {0:15}, and {60:100}, respectively.

[0045] We also added color dithering to the image to degrade its colors. Furthermore, we randomly sampled contrast (c), saturation (s), illuminance (i), and hue (h) from {0.5:1.5}, {0.5:1.5}, {0.5:1.5}, and {-0.1:0.1}, and performed corresponding image processing on the input image to finally obtain FFHQ-M. (The contrast (c), saturation (s), and illuminance (i) of the input image are set to 1 by default. The hue (h) of the input image is set to 0 by default.)

[0046] Step S2: Construct and train the CAR encoder and the pre-trained StyleGAN2 generator;

[0047] The CAR encoder and the pre-trained StyleGAN2 generator constitute a face restoration model. The training mode of generative adversarial network is adopted. The degraded image FFHQ-M is paired with the FFHQ high-definition face image to train the face restoration model. This model can enhance the detailed features of the face to be restored.

[0048] The model as a whole Figure 2 As shown, the input image size is (3, 512, 512). It first passes through the convolutional layer Convd1, resulting in an output size of (16, 512, 512). The convolutional kernel size of Convd1 is 3×3, with 3 input channels, 16 output channels, a stride of 1, and padding of 1.

[0049] The CAR encoder is as follows Figure 2As shown, the architecture includes 3 CAR downsampling layers, 3 CAR upsampling layers, and 1 CAR feature refinement layer. Each CAR downsampling layer consists of several CAR modules, one convolutional layer (Convd3), and one pixelunshuffle downsampling module connected sequentially. Each CAR upsampling layer consists of a concatenation operation, a convolutional layer (Convp1), several CAR modules, one convolutional layer (Convd4), and one pixelunshuffle upsampling module connected sequentially. In the CAR downsampling layers, the kernel size of the convolutional layer (Convd3) is 3×3, with C input channels, C / 2 output channels, a stride of 1, and padding of 1. The feature refinement layer maintains the same structure as the CAR upsampling layer, but it does not contain the convolutional layer (Convd4) or the pixelunshuffle upsampling module. In the CAR upsampling layer, the convolutional layer Convp1 has a kernel size of 1×1, 2×C input channels, C output channels, and a stride of 1; the convolutional layer Convd4 has a kernel size of 3×3, C input channels, 2×C output channels, a stride of 1, and padding of 1. The convolutional layer Convd1 has a kernel size of 3×3, 3 input channels, 16 output channels, a stride of 1, and padding of 1.

[0050] In the CAR encoder, the input dimensions of CAR downsampling layer-1 are (16, 512, 512), and the output dimensions are (32, 256, 256). The input dimensions of CAR downsampling layer-2 are (32, 256, 256), and the output dimensions are (64, 128, 128). The input dimensions of CAR downsampling layer-3 are (64, 128, 128), and the output dimensions are (128, 64, 64). The input dimensions of the CAR intermediate layer are (128, 64, 64), and the output dimensions are (128, 64, 64). The input dimensions of CAR upsampling layer-1 are (128, 64, 64), and the output dimensions are (64, 128, 128). The input dimensions of CAR upsampling layer-2 are (64, 128, 128), and the output dimensions are (32, 256, 256). The CAR upsampling layer-3 has an input size of (32, 256, 256) and an output size of (16, 512, 512). The feature refinement layer has an input size of (16, 512, 512) and an output size of (16, 512, 512).

[0051] The encoder output is finally passed through the Convd2 convolutional layer to obtain the encoder face restoration image, thus providing a supervised object for the supervised loss and accelerating the convergence speed of the network. The Convd2 convolutional layer has a kernel size of 3×3, 16 input channels, 3 output channels, and a stride of 1. Its input size is (16, 512, 512), and its output size is (3, 512, 512).

[0052] Specifically, the number of CAR modules in CAR downsampling / upsampling layer-1, CAR downsampling / upsampling layer-2, CAR downsampling / upsampling layer-3, and the CAR intermediate layer are [2, 3, 3, 4], the number of attention heads are [1, 2, 4, 8], and the number of channels are [16, 32, 64, 128]. The feature refinement layer settings are consistent with those of CAR downsampling / upsampling layer-1.

[0053] The CAR module consists of three modules connected in sequence: the Multi-Head Transposition Self-Attention Module (MDTA), the Gated Feedforward Network (GDFN), and the Channel Height and Width Axis Attention Module (CHWA).

[0054] The channel height and width axis attention module CHWA, as shown in... Figure 3 As shown, it contains two sequentially connected sub-modules: the width attention module (WAM) and the height attention module (HAM).

[0055] Taking the Width Attention (WAM) module as an example, given an input feature X of size (C, H, W), this feature is first passed through an average pooling layer (Avgpool1) to compress the width dimension of each channel, resulting in a feature of size (C, H). Then features After passing through a nonlinear module LRLS, a feature of size (C, H) is obtained. The nonlinear module LRLS consists of a cascaded linear layer, a ReLU activation layer, another linear layer, and a Sigmoid activation layer. Finally, features are broadcast. Processed as a feature of size (C, H, W) This feature can be multiplied element-wise with the input feature X, thereby weighting the input feature X and obtaining the feature output WAM(X).

[0056] Taking the High Attention Module (HAM) as an example, given an input feature X of size (C, H, W), this feature is first passed through an average pooling layer (Avgpool2) to compress the height dimension of each channel, resulting in a feature of size (C, W). Then features After passing through a nonlinear module LRLS, a feature of size (C, W) is obtained. The nonlinear module LRLS consists of a cascaded linear layer, a ReLU activation layer, another linear layer, and a Sigmoid activation layer. Finally, features are broadcast. Processed as a feature of size (C, H, W) This feature can be element-wise multiplied with the input feature X, thereby weighting the input feature X to obtain the feature output HAM(X). The calculation process of the channel height and width axis attention module CHWA is shown in formulas (4), (5) and (6):

[0057] CHWA(X)=HAM(WAM(X))+X, (4)

[0058] WAM(X)=LRLS(Avgpool1(X))⊙X, (5)

[0059] HAM(X)=LRLS(Avgpool2(X))⊙X, (6)

[0060] LRLS stands for Linear-ReLU-Linear-Sigmoid.

[0061] Furthermore, the affine network consists of a 4-layer affine transform (AFT) module and a 3-layer dual downsampling (DDS) module, used to connect the CAR encoder and the StyleGAN2 generator, such as... Figure 4 , Figure 5 and Figure 6 As shown.

[0062] The affine transform (AFT) module consists of a convolutional layer Convd10, a LeakyReLU activation layer, and another convolutional layer Convd11 connected sequentially. Its input comes from the features of the CAR upsampling layer, and its output is the affine feature α. i and β i The convolutional layers Convd10 and Convd11 have a kernel size of 3×3, C input channels, C output channels, a stride of 1, and padding of 1; the LeakyReLU activation layer has a negative slope coefficient of 0.2.

[0063] The dual downsampling DDS module has two branches: one branch consists of a bilinear downsampling module (Bilinear) and a convolutional layer (Convd12) cascaded sequentially; the other branch consists of a convolutional layer (Convd13) and a downsampling module (Pixelunshuffle) cascaded sequentially. These two branches are connected in parallel and then element-wise added to obtain the final output. The input is the affine feature α. i-1 and βi-1 The convolutional kernels of Convd12 and Convd13 are 3×3, with C input channels, C output channels, a stride of 1, and padding of 1. The calculation processes of the affine transformation (AFT) module and the dual downsampling (DDS) module are shown in formulas (7) and (8):

[0064]

[0065] α i ,β i =DDS(α) i-1 ,β i-1 ), i∈{5,6,7}, (8)

[0066] In the affine network, AFT module-1 inputs features (feature refinement layer features). The dimensions are (16, 512, 512), and the outputs are α. 1 and β 1 All dimensions are (16, 512, 512); AFT module - 2 input features (CAR upsampling layer - 2 features) The dimensions are (32, 256, 256), and the outputs are α. 2 and β 2 All dimensions are (32, 256, 256); AFT module - 3 input features (CAR upsampling layer - 3 features) The dimensions are (64, 128, 128), and the outputs are α. 3 and β 3 All dimensions are (64, 128, 128); AFT module - 4 input features (CAR upsampling layer - 4 features) The dimensions are (128, 64, 64), and the outputs are α. 4 and β 4 All dimensions are (128, 64, 64); DDS module -5 input feature α 4 and β 4 The dimensions are (128, 64, 64), and the outputs are α. 5 and β 5 All dimensions are (256, 32, 32); DDS module -6 input feature α 5 and β 5 The dimensions are (256, 32, 32), and the outputs are α. 6 and β 6 All dimensions are (256, 16, 16); DDS module -7 input feature α 6 and β 6 The dimensions are (256, 16, 16), and the outputs are α. 7 and β7 All dimensions are (256, 8, 8);

[0067] The StyleGAN2 generator is as follows: Figure 7 As shown, it is divided into 7 layers, with each layer corresponding to a pixel value of 512. 2 256 2 128 2 64 2 32 2 16 2 8 2 The intermediate features of the size are processed by each image generation layer, SFTlayer. Each image generation layer, SFTlayer, consists of a ToRGB module, two cascaded StyleConv modules, and an affine feature processing module. The size of the features processed by the network is changed step by step through the scale transformation layer.

[0068] In this embodiment, the image generation layer SFTlayer-7 has two inputs. The first is feature F7, with a size of (512, 4, 4), which outputs feature F6 with a size of (512, 8, 8) after passing through two cascaded StyleConv modules and an affine feature processing module. The second is feature I7, with a size of (3, 4, 4), which outputs feature I6 with a size of (3, 8, 8) after passing through a ToRGB module. Similarly, the image generation layer SFTlayer-6 has two inputs. The first is feature F6, with a size of (512, 8, 8), which outputs feature F5 with a size of (512, 16, 16) after passing through two cascaded StyleConv modules and an affine feature processing module. The second is feature I6, with a size of (3, 8, 8), which outputs feature I5 ​​with a size of (3, 16, 16) after passing through a ToRGB module. The image generation layer SFTlayer-5 has two inputs. The first is feature F5, with a size of (512, 16, 16), which outputs feature F4, with a size of (512, 32, 32), after passing through two cascaded StyleConv modules and an affine feature processing module. The second is feature I5, with a size of (3, 16, 16), which outputs feature I4, with a size of (3, 32, 32), after passing through a ToRGB module. Similarly, the image generation layer SFTlayer-4 has two inputs. The first is feature F4, with a size of (512, 32, 32), which outputs feature F3, with a size of (256, 64, 64), after passing through two cascaded StyleConv modules and an affine feature processing module. The second is feature I4, with a size of (3, 32, 32), which outputs feature I3, with a size of (3, 64, 64), after passing through a ToRGB module. The image generation layer SFTlayer-3 has two inputs. The first is feature F3, with a size of (256, 64, 64), which outputs feature F2 with a size of (128, 128, 128) after passing through two cascaded StyleConv modules and an affine feature processing module. The second is feature I3, with a size of (3, 64, 64), which outputs feature I2 with a size of (3, 128, 128) after passing through a ToRGB module. Similarly, the image generation layer SFTlayer-2 has two inputs. The first is feature F2, with a size of (128, 128, 128), which outputs feature F1 with a size of (64, 256, 256) after passing through two cascaded StyleConv modules and an affine feature processing module. The second is feature I2, with a size of (3, 128, 128), which outputs feature I1 with a size of (3, 256, 256) after passing through a ToRGB module. The image generation layer SFTlayer-1 has two inputs. The first is feature F1, with a corresponding size of (64, 256, 256), which is processed by two cascaded StyleConv modules and an affine feature processing module.The second feature is I1, with a size of (3, 256, 256). After passing through the ToRGB module, the repaired image is output with a size of (3, 256, 256).

[0069] In the image generation layer SFTlayer, the input size of the style convolutional layer StyleConv1 is (C1, H, W), and the output size is (C2, 2×H, 2×W). The input size of the style convolutional layer StyleConv2 is (C2, 2×H, 2×W), and the output size is (C2, 2×H, 2×W). The input size of the image conversion module ToRGB is (3, H, W), and the output size is (3, 2×H, 2×W). C1 and C2 are determined according to the specific SFTlayer layer hierarchy.

[0070] We employ the Channel Segmentation Spatial Feature Transform (CS-SFT module) to perform the transformation between the two StyleConv1 and StyleConv2 in the image generation layer SFTlayer. Perform an affine transformation on the features. Half of them. The feature will be determined by the affine parameter α i and β i The modulation occurs on one half, while the other half remains unchanged. The calculation process of the CS-SFT module is shown in formula (9):

[0071]

[0072] and for The two features are divided equally according to the channel dimension. This represents GAN features without affine transformation. This represents the GAN features obtained through affine transformation. Concat[,] represents the feature concatenation operation.

[0073] We use four loss functions to train our face restoration model, as follows: (1) supervised loss for supervising intermediate images; (2) adversarial loss to enhance image realism; (3) perceptual loss; and (4) identity loss to ensure identity consistency.

[0074] (1) Monitoring loss:

[0075]

[0076] y represents the supervision loss, and y represents the high-resolution image. u λ represents the face restoration image from the CAR encoder. l1 This represents the weights of the loss function.

[0077] (2) Adversarial losses:

[0078]

[0079] Let λ represent the adversarial loss, D represent the discriminator, and λ represent the λ-value. adv The weights represent the adversarial loss, and softplus represents the softplus activation function. This represents the generated image from the repaired network.

[0080] (3) Perceived loss:

[0081]

[0082] Let φ represent the perceptual loss, and φ represent the pre-trained VGG-19 network. Represents a face restoration image, λ per This represents the perceived loss weight.

[0083] (4) Loss of identity:

[0084]

[0085] Indicates identity loss, λ id Let η represent the identity loss weights, and let η represent the pre-trained Arcface network.

[0086] The total loss formula is as follows:

[0087]

[0088] The weights for each loss are λ. l1 =0.1, λ adv =0.1, λ per =1,

[0089] Step S3: Output the result;

[0090] A low-quality face image is input into a CAR encoder to obtain corresponding affine features and latent codes. This result is then processed by a StyleGAN2 generator to produce a face-restored image. This face restoration method can remove noise, restore details, and enhance and restore colors in the input image. Figure 1 As shown.

[0091] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A face restoration method based on a CAR encoder and prior knowledge of face generation, characterized in that, The method includes the following steps: Step S1: Training set acquisition and processing; Acquire a high-resolution face image FFHQ, and perform multiple manual degradations on the high-resolution face image FFHQ to obtain the degraded image FFHQ-M; Step S2: Construct and train the CAR encoder and the pre-trained StyleGAN2 generator; the CAR encoder and the pre-trained StyleGAN2 generator constitute a face restoration model. The training mode of generative adversarial network is adopted, and the degraded image FFHQ-M is paired with the high-definition face image FFHQ to train the face restoration model. Step S3: The affine network consists of 4 layers of affine transform (AFT) modules and 3 layers of dual downsampling (DDS) modules, used to connect the CAR encoder and the StyleGAN2 generator. One branch of the dual downsampling DDS module is formed by cascading the bilinear downsampling module Bilinear and the convolutional layer Convd12, and the other branch is formed by cascading the convolutional layer Convd13 and the downsampling module Pixelunshuffle. The two branches are connected in parallel and finally element-wise addition is performed to obtain the final output.

2. The face restoration method based on a CAR encoder and prior knowledge of face generation according to claim 1, characterized in that, The CAR encoder is divided into 1-4 levels, including 3 CAR downsampling layers, 3 CAR upsampling layers, and 1 CAR feature refinement layer. The CAR downsampling layer is composed of several CAR modules, one Convd3 convolutional layer, and one Pixelunshuffle downsampling module connected in sequence; The CAR upsampling layer is composed of a splicing operation, a convolutional layer Convp1, several CAR modules, a convolutional layer Convd4, and a pixelunshuffle upsampling module connected in sequence. The CAR feature refinement layer is composed of several CAR modules connected sequentially. The number of CAR modules in CAR downsampling / upsampling layer-1, CAR downsampling / upsampling layer-2, CAR downsampling / upsampling layer-3 and CAR intermediate layer are [2, 3, 3, 4], the number of attention heads are [1, 2, 4, 8], and the number of channels is [16, 32, 64, 128]. The feature refinement layer is set in the same way as CAR downsampling / upsampling layer-1.

3. The face restoration method based on a CAR encoder and prior knowledge of face generation according to claim 1, characterized in that, The affine network consists of a 4-layer affine transform (AFT) module and a 3-layer dual downsampling (DDS) module, which are used to connect the CAR encoder and the StyleGAN2 generator. The affine transformation (AFT) module is composed of a convolutional layer Convd10, a LeakyReLU activation layer, and a convolutional layer Convd11 connected in sequence. The dual downsampling DDS module consists of one branch consisting of a convolutional layer Convd13 and a Pixelunshuffle downsampling module connected in sequence, and another branch consisting of a bilinear downsampling module and a convolutional layer Convd12 connected in sequence. The two branches are connected in parallel to form the dual downsampling module. The StyleGAN2 generator is divided into 1 to 7 levels, each level consisting of a ToRGB module and two cascaded StyleConv modules connected in parallel.

4. The face restoration method based on a CAR encoder and prior knowledge of face generation according to claim 2, characterized in that, The CAR module consists of three modules connected in sequence: Multi-Head Transposition Self-Attention Module (MDTA), Gated Feedforward Network (GDFN), and Channel Height and Width Axis Attention Module (CHWA).

5. The face restoration method based on a CAR encoder and prior knowledge of face generation according to claim 4, characterized in that, The channel height and width axis attention module CHWA is composed of an average pooling layer (Avgpool), a nonlinear module (LRLS), a broadcast operation, and a residual operation cascaded together. The nonlinear module (LRLS) is composed of a linear layer (Linear), a function activation layer (ReLU), a linear layer (Linear), and a function activation layer (Sigmoid) cascaded together.

6. The face restoration method based on a CAR encoder and prior knowledge of face generation according to claim 1, characterized in that, Our face restoration model is trained using four loss functions: (1) supervised loss for supervising intermediate images; (2) adversarial loss to enhance image realism; (3) perceptual loss; and (4) identity loss to ensure identity consistency. The total loss formula is as follows: ,in Indicates monitoring losses, Indicates adversarial loss. Indicates perceived loss. This indicates a loss of identity.

Images