A face image restoration method based on feature fusion and reuse

Abstract

This invention discloses a face image restoration method based on feature fusion and reuse, belonging to the field of high-resolution image restoration technology, including the following steps: Step S1: Preprocessing the input image to obtain the incomplete face image to be restored; Step S2: Establishing a face image restoration model based on feature fusion and reuse, and inputting the damaged edge image from S1 into the image restoration model for training; Step S3: Through multiple iterations of training until the network finally converges, obtaining the face image restoration weight model; Step S4: Inputting the damaged face image into the trained face image restoration model to obtain the restored face image. The encoder-decoder structure is constructed using a skip connection approach. The improved aggregation context transition module enhances contextual reasoning and feature enhancement of the image, and a feature reuse module is constructed to fully utilize the detailed features of the image structure and texture, resulting in better restoration results.

Description

Technical Field

[0001] This invention relates to the field of image restoration technology, and in particular to a face image restoration method based on feature fusion and reuse. Background Technology

[0002] Image inpainting methods aim to restore damaged pixels in an image by filling in missing pixels based on known information, ensuring the inpainted image remains visually and semantically consistent with the original. Currently, image inpainting methods can be categorized into pixel-based and semantic-based methods. Pixel-based methods utilize the principle of constructing a new pixel within the damaged area, ensuring consistency with surrounding pixels. These methods are primarily divided into diffusion-based and block-matching-based methods. While effective for small missing or occluded areas, pixel-based methods become less effective and prone to distortion when dealing with larger areas. Semantic-based methods, also known as deep learning-based methods, predict semantic features of the missing area and establish a mapping between the damaged and original images. This allows for structural and detail restoration of the missing area. These methods are mainly divided into convolutional neural network-based and generative adversarial network-based methods.

[0003] Generative Adversarial Networks (GANs) are network models proposed by Goodfellow et al. in 2014 based on game theory. A GAN consists of a generator network and a discriminator network. The generator network aims to learn the distribution of real data as closely as possible, while the discriminator network aims to correctly determine whether the input data comes from real data or the generator network. Both are continuously optimized until their performance is optimal. In image inpainting, GANs are primarily used to generate restored images that closely resemble real images. For example, a low-resolution, blurry image can be restored to a high-resolution image with rich details. The application of GANs in image inpainting has significantly improved the consistency and similarity between restored and real images.

[0004] Face restoration, as an important branch of image restoration, plays a crucial role in practical applications. Compared to ordinary image restoration, faces have stronger semantics and more complex texture details, requiring not only consideration of the rationality of the facial structure but also preservation of personal information during the restoration process.

[0005] Image inpainting has made significant progress, from early traditional methods to current deep learning-based approaches. Traditional methods are only suitable for repairing single, simple images with small missing regions and lack semantic consistency. Therefore, deep learning-based methods have become the mainstream.

[0006] Pathak first proposed Context Encoders, which uses an encoder-decoder network to extract features and output reconstruction results. It is also the first GAN-based inpainting method. Iizuka et al. introduced a local-global dual discriminator on the basis of Context Encoders and proposed the GLCIC network using dilated convolution. Yu et al. proposed the deepfill network, which borrows or copies feature information from known background patches to generate missing foreground patches through a context attention mechanism. Nazeri et al. designed EdgeConnect using a two-stage model. First, the edge generator generates an imaginary edge map of the irregular missing area as a prior result. Then, based on this imaginary edge map, an image inpainting network is used to fill in the missing area.

[0007] However, these methods may produce distorted structures and blurred textures in high-resolution images (e.g., 512×512). The main challenges arise from inferring pixel content over long distances in the image and synthesizing fine-grained textures for large missing regions. To overcome these two challenges, Zeng et al. proposed an enhanced GAN-based model for high-resolution image generation.

[0008] In 2021, Zeng proposed the paper "Aggregated Contextual Transformations for High-Resolution Image Inpainting" (AOT-GAN). AOT-GAN is a high-resolution inpainting network structure that obtains high-fine-grained image textures by acquiring contextual image information through multiple layers of dilated convolutional branches and then utilizing different discriminator discrimination strategies. The algorithm mainly consists of an encoder-decoder network in the generator for feature extraction and a discriminator with improved discrimination strategies. To enhance contextual reasoning capabilities, the network constructs the AOT-GAN generator by stacking multiple layers of proposed AOT blocks. AOT blocks aggregate contextual transformations from various receptive domains, allowing the capture of rich long-range image contextual information and a rich receptive field for contextual reasoning. To improve texture synthesis, the network trains the AOT-GAN discriminator mask-prediction. This training objective forces the discriminator to distinguish between realistic and synthetic detail appearance patches, thereby promoting the generator to synthesize sharp textures. The network structure is as follows: Figure 10 As shown.

[0009] The existing technology still has the following shortcomings:

[0010] Image inpainting models suffer from problems such as loss of detail information, insufficient utilization of feature information, inability to correctly distinguish effective pixels in the image to be repaired, and insufficient attention to key areas of the image when repairing high-resolution images. These problems can lead to unclear structure, inconsistent texture, and distortion in the repaired images.

[0011] Existing research still suffers from issues such as low image accuracy or blurred texture when repairing large, irregularly structured missing regions. This is mainly due to the insufficient utilization of the image's contextual information and the lack of integration of deep and shallow features, such as semantic and spatial information. This results in insufficient connectivity from local features to overall consistency. Summary of the Invention

[0012] This invention proposes a face image restoration method based on feature fusion and reuse. It adopts a skip connection approach to construct the encoder-decoder structure, improves the aggregation context transformation module to enhance the contextual reasoning and feature enhancement of the image, and constructs a feature reuse module to make full use of the detailed features of the image structure and texture, so as to improve the restoration results.

[0013] A face image restoration method based on feature fusion and reuse includes the following steps:

[0014] Step S1: Preprocess the input image to obtain the incomplete face image to be repaired;

[0015] Step S2: Establish a face image restoration model based on feature fusion and reuse, and input the damaged edge image in S1 into the image restoration model for training;

[0016] The face image restoration model based on feature fusion and reuse includes a generator and a discriminator. The generator includes an encoder-decoder feature fusion module and a feature reuse module. In the encoder-decoder feature fusion module, the convolutional layers are connected by skip links.

[0017] The context aggregation transformation module EAOT-Block is formed by combining the activation function GELU and the channel attention module CA. The context aggregation transformation module EAOT-Block is constructed in the form of feature reuse and deployed in the encoding and decoding structure of the feature reuse module.

[0018] Step S3: Through multiple iterations of training, until the network finally converges, the face image restoration weight model is obtained;

[0019] Step S4: Input the damaged face image into the trained face image restoration model to obtain the restored face image.

[0020] Preferably, in step S2, the specific method for connecting the convolutional layers using skip links is as follows:

[0021] The first convolutional layer passes the feature information obtained to the next convolutional layer, and at the same time, it also passes the same feature information to the last layer of the decoding structure. Similarly, the feature information obtained from the second convolutional layer is passed to the penultimate layer of the decoding structure while passing it to the next convolutional layer.

[0022] Preferably, in step S2, the context aggregation conversion module EAOT-Block includes the following steps:

[0023] (i) Splitting: The feature map x1 in the input channel is reduced in dimensionality into four 64-channel sub-feature maps using four 3×3 gated convolutions;

[0024] (ii) Transformation: Each gated convolution kernel has a different dilation rate. Different image region information is obtained through different dilation rates. Feature information is enhanced by adding the activation function GELU and the channel attention module CA.

[0025] (iii) Aggregation: The context transformation features from different receptive fields are finally aggregated by concatenating the channel dimensions and standard convolution to obtain the fused feature x2; the feature map x1 is subjected to 3×3 standard convolution and Sigmoid operation to form a threshold g, and then the transformed fused feature is gated and weighted with the original feature to obtain the final output feature.

[0026] Preferably, the weighted calculation formula in (iii) is:

[0027] x1×g+x2×(1-g);

[0028] In the formula, g is the threshold, x1 is the feature map, and x2 is the fused feature.

[0029] Preferably, in step S2, the context aggregation transformation module EAOT-Block is constructed in a feature reuse manner by the following method: each convolutional layer takes all the preceding convolutional layers as its additional input, thereby realizing the interconnection of all convolutional layers and thus achieving feature reuse.

[0030] Preferably, in step S2, the discriminator compares the image generated by the generator with the original image, feeding back the image to the generator using soft labels and assigning weights to prevent the discriminator from completely classifying the generated portion as fake; around the boundary of the missing region, the discriminator is trained using a soft patch-level mask, and the adversarial loss of the discriminator is:

[0031]

[0032] The adversarial loss of the generator is denoted as:

[0033]

[0034] in,

[0035]

[0036] In the formula, σ is the combination function of downsampling and Gaussian filtering, and m is the binary inner drawing mask. For pixel-level multiplication, D is the discriminator, G is the generator, L is the loss value, x is the real image, and E is the expected value.

[0037] Preferably, in step S3, the face image restoration weight model is:

[0038]

[0039] in,

[0040]

[0041]

[0042] L sty =E i [‖φ i (x) T φ i (x)-φ i (z) T φ(z)‖1;

[0043] In the formula, G is the generator, and λ adv λ is the generator weight. rec For L1 loss weights, λ per To perceive the loss weights, λ sty For style loss weights, L rec For L1 loss, L per To perceive loss, L sty Let D be the style loss, M be the discriminator, and L be the mask image. Let L be the loss value, where |||1 is the L1 norm, and φ is the loss value. i For the activation map of the i-th pooling layer in Vgg19, φ i ()-φ i T ()φ i () represents the activation map φ i Gram matrix, N i For φ i The number of elements in the middle.

[0044] The beneficial effects of this invention are:

[0045] (1) This invention fuses deep features and shallow features, which helps the network extract feature information at different scales. After fusing the feature information of different layers, the decoding can take into account the advantages of the semantic representation ability of the high-level network and the geometric detail representation ability of the low-level network, thereby improving the overall texture and detail accuracy of the generated image.

[0046] (2) The context aggregation transformation module EAOT-Block of the present invention is used to capture long-distance context information, adopts CA attention mechanism to enhance the network’s focus and improve the accuracy of repair from the channel perspective, and uses Gelu activation function to prevent gradient vanishing, making training more efficient and faster to converge.

[0047] (3) The present invention uses the feature reuse method to construct the EAOT-Block connection module, which reduces the number of parameters compared with the Resnet connection method in the prior art, while enhancing the reuse of features, which is conducive to preserving feature information, making the network easier to train, and solving the problem of insufficient use of feature information. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the process of the present invention;

[0049] Figure 2 This is a network structure diagram of the present invention;

[0050] Figure 3 This is a structural diagram of the EAOT-Block aggregation context feature enhancement module in the generator of this invention;

[0051] Figure 4 This is a detailed connection structure diagram of the EAOT-Block structure of the present invention;

[0052] Figure 5 This is a structural diagram of the feature reuse module of the present invention;

[0053] Figure 6 This is a structural diagram of the generator of the present invention;

[0054] Figure 7 This is a detailed connection structure diagram of the generator of the present invention;

[0055] Figure 8 This is a schematic diagram illustrating the qualitative comparison of the effects of this invention with other methods;

[0056] Figure 9 This is a schematic diagram of the connection method of the present invention;

[0057] Figure 10 This is a schematic diagram of the AOT-GAN network structure;

[0058] Figure 11 This is a schematic diagram of the discrimination strategy of the AOT-GAN discriminator. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided with reference to the accompanying drawings and embodiments.

[0060] like Figures 1-9 As shown, a face image restoration method based on feature fusion and reuse includes the following steps:

[0061] Step S1: Preprocess the input image to obtain the incomplete face image to be repaired;

[0062] The input image is preprocessed to obtain the incomplete face image to be repaired. First, the image size is adjusted to 512×512 by center cropping and padding. Then, a binarized mask is obtained from the irregular mask dataset provided by NVIDIA to perform damage processing on the image, resulting in the incomplete face image. This damaged image is then converted to grayscale to obtain a damaged grayscale image. Finally, the damaged grayscale image is used to extract the face contour information using the Canny edge detection algorithm to obtain the damaged edge image.

[0063] Step S2: Establish a face image restoration model based on feature fusion and reuse, and input the damaged edge image in S1 into the image restoration model for training;

[0064] Face image inpainting models based on feature fusion and reuse, such as... Figure 2 As shown, the model is based on a generative adversarial network structure and consists of a generator and a discriminator. The generator consists of two parts: an encoder-decoder feature fusion module and a feature reuse module.

[0065] The encoder-decoder uses standard convolution. During the encoding stage, the encoder encodes features of the received damaged image. Higher layers of the network have relatively larger receptive fields and stronger semantic information representation capabilities, but lower feature map resolution and weaker geometric information representation capabilities (lacking spatial geometric feature details). Lower layers of the network have smaller receptive fields and stronger geometric detail representation capabilities, although they have higher resolution, but weaker semantic information representation capabilities.

[0066] This invention utilizes skip links to connect convolutional layers, such as... Figures 6-7 As shown, the encoding structure consists of three convolutions with progressively increasing channel numbers to extract features, while the decoder consists of three convolutions with progressively decreasing channel numbers to generate an image. The first convolution layer passes the obtained feature information to the next convolution layer and also passes the same feature information to the last layer of the decoding structure. Similarly, the feature information obtained from the second convolution layer is passed to the penultimate layer of the decoding structure while passing it to the next convolution layer.

[0067] like Figure 3 As shown, the context aggregation transformation module EAOT-Block is formed by combining the activation function GELU and the channel attention module CA. This module weights the obtained feature information during the context aggregation process, making it easier to preserve the image's positional information. The network considers attention in both the channel and spatial dimensions simultaneously and can learn adaptive channel weights to pay more attention to useful feature information.

[0068] The context aggregation and transformation module EAOT-Block adopts a splitting, transformation, and aggregation approach:

[0069] (i) Splitting: The feature map x1 in the input channel is reduced in dimensionality into four 64-channel sub-feature maps using four 3×3 gated convolutions;

[0070] (ii) Transformation: Each gated convolution kernel has a different dilation rate. Different image region information is obtained through different dilation rates. Feature information is enhanced by adding the activation function GELU and the channel attention module CA.

[0071] (iii) Aggregation: The context transformation features from different receptive fields are finally aggregated by concatenating the channel dimensions and standard convolution to obtain the fused feature x2; the feature map x1 is subjected to 3×3 standard convolution and Sigmoid operation to form a threshold g, and then the transformed fused feature is gated and weighted with the original feature to obtain the final output feature.

[0072] GELU (Gaussian error linear units) is a high-performance neural network activation function. Because the non-linear transformation of GELU is a predictable stochastic regularization method, it has two advantages:

[0073] 1. The derivative of the GELU function is continuous, which makes it easier to propagate gradients when training deep neural networks. This avoids the problem of discontinuous derivatives of the ReLU function in special cases, thereby reducing the gradient vanishing problem during training and resulting in a smoother derivative.

[0074] 2. The GELU function introduces a transformation similar to the sigmoid function in the nonlinear transformation of the activation function. This allows the output of the GELU function to fall within a wider range, which helps to accelerate the convergence speed of the model.

[0075] like Figures 4-5As shown, four enhanced context aggregation transformation modules EAOT-Block are constructed in a feature reuse manner and deployed in the middle of the encoding and decoding structure (feature reuse module). Feature reuse is achieved through the connection of features on the channel. Each layer accepts all the layers before it as its additional input and connects all the layers to each other. That is, the feature information output by the first EAOT-Block module is not only passed to the next EAOT-Block module, but also to all EAOT-block modules.

[0076] like Figure 9 As shown, in the context aggregation and transformation module EAOT-Block, each layer is concatenated with all preceding layers along the channel dimension and used as input to the next layer. Furthermore, DenseNet directly concatenates feature maps from different layers, enabling feature reuse, improving efficiency, and utilizing features across multiple layers to increase repair accuracy. Because the input and output feature maps of the context aggregation and transformation module EAOT-Block are of the same size, it is well-suited for this feature reuse connection method, offering two advantages:

[0077] 1. DenseNet uses dense connections to improve gradient backpropagation, making the network easier to train;

[0078] 2. Because DenseNet achieves short-circuit connections by concatting features, it enables feature reuse and uses a smaller growth rate. Each layer has a smaller feature map, fewer parameters, and is more computationally efficient, which is somewhat counterintuitive.

[0079] Each layer consists of a structure of BN (batch norm) + GELU + 1×1 convolution. BN is a type of Batch Norm, which is a neural network layer added as part of a linear or convolutional layer to help stabilize the network during training. The reason for using 1×1 convolutions is to reduce the number of features, thereby improving computational efficiency.

[0080] The discriminator compares the image generated by the generator with the original image, feeding back the comparison to the generator using a soft-labeling method. This prevents the discriminator from completely classifying the generated portion as fake, instead assigning weights to it. This discriminative approach improves training performance, thereby encouraging the generator to synthesize more fine-grained realistic textures. Figure 11 As shown (given that pixel values ​​are 0 and missing region values ​​are 1), the discriminator operation mode can be represented as:

[0081] Around the boundaries of the missing regions, the discriminator is trained using a soft-patch level mask. This soft mask is obtained through Gaussian filtering. The adversarial loss of the discriminator is:

[0082]

[0083] The adversarial loss of the generator is denoted as:

[0084]

[0085] in,

[0086]

[0087] In the formula, σ is the combination function of downsampling and Gaussian filtering, and m is the binary inner drawing mask. For pixel-level multiplication, D is the discriminator, G is the generator, L is the loss value, x is the real image, and E is the expected value.

[0088] Step S3: Train the network through 600,000 iterations until it finally converges to obtain the face image restoration weight model.

[0089] The dataset used in this invention is CelebaHQ, comprising 27,000 training images and 3,000 test images. The entire model is implemented in PyTorch, and the experimental equipment is an NVIDIA V100. During model training, the batch size is set to 4, using 1×10T... -4 Initial training was performed using a learning rate of 2×10, which was then fine-tuned to 2×10 based on experimental results. -5 .

[0090] Loss Function: The purpose of selecting the optimization objective in the generated image is to ensure pixel reconstruction accuracy and the visual fidelity of the image rendered by the recognizer. To this end, four optimization objectives were carefully selected: L1 loss, style loss, perceptual loss, and SM-PatchAOT, an adversarial loss for AOT-GAN, following the practices of most existing deep image inpainting models. First, L1 loss is used to ensure pixel-level reconstruction accuracy:

[0091]

[0092] The goal of perceptual loss is to minimize the L1 distance between the inset and the activation map of the ground truth image.

[0093]

[0094] Style loss is defined as the L1 distance between the Cranskiy matrix of the depth features of the drawn image and the ground truth image:

[0095] L sty =E i||φ i (x) T φ i (x)-φ i (z) T φ(z)‖1;

[0096] Finally, the adversarial loss of SM-PatchGAN, described in the equation, is included, improving the visual fidelity of the in-line image. The entire AOT-GAN is trained through joint optimization of these four objectives, resulting in the following overall optimization objective:

[0097]

[0098] In the formula, G is the generator, and λ adv =0.01, λ rec =1,λ per =0.1, λ sty =250, L rec For L1 loss, L per To perceive loss, L sty Let D be the style loss, M be the discriminator, and L be the mask image. Let L be the loss value, where ||.|1 is the L1 norm, and φ is the style loss. i For the activation map of the i-th pooling layer in Vgg19, φ i ()-φ i T ()φ i () represents the activation map φ i Gram matrix, N i For φ i The number of elements in the middle.

[0099] Step S4: Input the damaged face image into the trained face image restoration model to obtain the restored face image. Using the CelebaHQ dataset test set, the algorithm is compared with EdgeConnect, HiFill, and AOT-GAN algorithms under different mask region ratios.

[0100] The quantitative analysis is as follows: This invention is validated based on the CelebaHQ dataset, where the mask uses different proportions of damaged images from 0% to 40% to represent the size of the damaged area (public mask dataset provided by NVIDIA). The generated results are compared using three evaluation metrics: PSNR, SSIM, and MAE, as shown in Table 1: (↑ indicates a larger value is better, ↓ indicates a smaller value is better, and bold indicates the optimal result).

[0101] Table 1 Comparison of objective evaluation metrics for experimental results on the CelebA-HQ dataset

[0102]

[0103] This invention achieves optimal results in all three metrics compared to other methods. Specifically, the closer the SSIM value is to 1, the better; the higher the PSNR, the better; and the closer the MAE is to 0, the better.

[0104] like Figure 8 As shown, column a represents the original face image, column b represents the damaged face image to be repaired, and subsequent columns represent the repair results of each comparative model. EdgeConnect exhibits distortion and severe loss of detail in the repair of facial structures when restoring high-resolution face images. HiFill produces overly smoothed image content, resulting in inconsistent facial colors and blurred textures at high resolutions. AOT-GAN also suffers from blurred textures, distorted structures, and inconsistent colors. The repaired face structure and texture of this invention are more consistent, and the colors are more uniform, resulting in better performance in high-resolution face image restoration.

[0105] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A face image restoration method based on feature fusion and reuse, characterized in that, Includes the following steps: Step S1: Preprocess the input image to obtain the incomplete face image to be repaired; Step S2: Establish a face image restoration model based on feature fusion and reuse, and input the damaged edge image in S1 into the image restoration model for training; The face image restoration model based on feature fusion and reuse includes a generator and a discriminator. The generator includes an encoder-decoder feature fusion module and a feature reuse module. In the encoder-decoder feature fusion module, the convolutional layers are connected by skip links. The context aggregation transformation module EAOT-Block is formed by combining the activation function GELU and the channel attention module CA. The context aggregation transformation module EAOT-Block is constructed in the form of feature reuse and deployed in the encoding and decoding structure of the feature reuse module. Step S3: Through multiple iterations of training, until the network finally converges, the face image restoration weight model is obtained; Step S4: Input the damaged face image into the trained face image restoration model to obtain the restored face image.

2. The face image restoration method based on feature fusion and reuse according to claim 1, characterized in that, In step S2, the specific method for connecting the convolutional layers using skip connections is as follows: The first convolutional layer passes the feature information obtained to the next convolutional layer, and at the same time, it also passes the same feature information to the last layer of the decoding structure. Similarly, the feature information obtained from the second convolutional layer is passed to the penultimate layer of the decoding structure while passing it to the next convolutional layer.

3. The face image restoration method based on feature fusion and reuse according to claim 1, characterized in that, In step S2, the context aggregation transformation module EAOT-Block includes the following steps: (i) Splitting: The feature map x1 in the input channel is reduced in dimensionality into four 64-channel sub-feature maps using four 3×3 gated convolutions; (ii) Transformation: Each gated convolution kernel has a different dilation rate. Different image region information is obtained through different dilation rates. Feature information is enhanced by adding the activation function GELU and the channel attention module CA. (iii) Aggregation: The context transformation features from different receptive fields are finally aggregated by concatenating the channel dimensions and standard convolution to obtain the fused feature x2; the feature map x1 is subjected to 3×3 standard convolution and Sigmoid operation to form a threshold g, and then the transformed fused feature is gated and weighted with the original feature to obtain the final output feature.

4. The face image restoration method based on feature fusion and reuse according to claim 1, characterized in that, The weighted calculation formula in (iii) is: x1×g+x2×(1-g); In the formula, g is the threshold, x1 is the feature map, and x2 is the fused feature.

5. A face image restoration method based on feature fusion and reuse according to claim 1, characterized in that, In step S2, the context aggregation transformation module EAOT-Block is constructed in a feature reuse manner by the following method: each convolutional layer takes the output of all previous convolutional layers as its additional input, thereby connecting all convolutional layers to each other and thus achieving feature reuse.

6. The face image restoration method based on feature fusion and reuse according to claim 1, characterized in that, In step S2, the discriminator compares the image generated by the generator with the original image, feeding back the results to the generator using soft labels and assigning weights to prevent the discriminator from completely classifying the generated portion as fake. Around the boundaries of the missing region, the discriminator is trained using a soft-patch level mask. The adversarial loss of the discriminator is: The adversarial loss of the generator is denoted as: in, In the formula, σ is the combination function of downsampling and Gaussian filtering, and m is the binary inner drawing mask. For pixel-level multiplication, D is the discriminator, G is the generator, L is the loss value, x is the real image, and E is the expected value.

7. A face image restoration method based on feature fusion and reuse according to claim 1, characterized in that, In step S3, the face image restoration weight model is: in, L sty =E i [||φ i (x) T f i (x)-φ i (z) T φ(z)||1; In the formula, G is the generator, and λ adv λ is the generator weight. rec For L1 loss weights, λ per To perceive the loss weights, λ sty For style loss weights, L rec For L1 loss, L per To perceive loss, L sty Let D be the style loss, M be the discriminator, and L be the mask image. Let L be the loss value, where |||1 is the L1 norm, and φ is the loss value. i For the activation map of the i-th pooling layer in Vgg19, φ i ()-φ i T ()φ i () represents the activation map φ i Gram matrix, N i For φ i The number of elements in the middle.

Images