Face local saliency region completion method based on super-resolution

By adopting a super-resolution-based facial local saliency area completion method in the face image processing, combined with the pyramid attention module and encoder structure, the difficulty in effectively restoring saliency areas in the face image processing in the prior art is solved. and low-resolution images have significantly improved the image fidelity and detail recovery capabilities.

CN119067888APending Publication Date: 2024-12-03INNER MONGOLIA UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411259046.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-10
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The prior art cannot effectively recover information of the obstructed portion in the face image processing, especially in salience areas (such as eyes, nose, mouth) and low-resolution images, where the generated image texture details are too smooth and the model processing The effect is poor when large-area is missing, resulting in insufficient fit between the generated image and the real image.

Method used

Using the super-resolution-based local saliency area completion method, the problem of face saliency area occlusion and super-resolution reconstruction is solved simultaneously through a single model, and the face completion generator is trained using the pre-processed public data set. and super-resolution reconstruction generators, combined with pyramid attention modules and encoder structures, optimize the loss function to improve image fidelity and detail recovery capabilities.

Benefits of technology

It significantly improves the texture details and fidelity of the face image, reduces the domain difference between the generated image and the real world image, can better restore the significant texture information of facial features, and improves the visual effect of the image. and the expressive ability of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067888A_ABST
    Figure CN119067888A_ABST
Patent Text Reader

Abstract

A face local saliency region completion method based on super-resolution comprises the following steps: (1) preprocessing a public data set and generating a shielding training set which is used for pre-training a face completion generator in a face saliency region completion network model; (2) training a face analysis network by adopting a public data set, wherein the face analysis network is used for auxiliary training of a face super-resolution reconstruction generator; then fixing network parameters of the pre-trained face completion generator, training a super-resolution reconstruction generator and a discriminator in the face saliency region completion network model in sequence, and finally performing overall training on the face saliency region completion network model; according to the method, the problems of face saliency region shielding and super-resolution image reconstruction are solved at the same time through a single model, the image is recovered from saliency region shielding and large-scale down-sampling, the texture details and fidelity of face image recovery are improved, the domain difference between the generated image and the real world image is reduced, and the image quality is improved. A super-resolution (SR) model can be easily adapted to real LR data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a technology in the field of image processing, in particular to a method for completing local salient areas of a face based on super-resolution. Background Art

[0002] The facial texture information obtained by existing facial image processing technology is not clear enough and does not have the function of preserving identity information. Therefore, it cannot accurately restore the information of the occluded part of the occluded facial image; in addition, when restoring the occluded low-resolution facial image, it cannot effectively restore the salient area of ​​the face, and the texture details of the generated image are too smooth; most of the existing deep learning-based image completion methods ignore the establishment of long-distance information interaction between pixels, resulting in poor results when these models process large-area missing and insufficient overall fit of the generated image. The existing technology that retains image detail information by fusing features of different scales ignores the difference in the contribution of each scale, resulting in insufficient model accuracy and considerable artifact errors in the generated results. Summary of the invention

[0003] In view of the poor fidelity of facial images generated by the prior art, the inability to generate textures similar to the ground truth images in areas with rich texture details such as salient facial regions (eyes, nose, mouth), and the defect that low-resolution occluded facial images cannot be restored, the present invention proposes a super-resolution-based facial local salient region completion method, which solves the problems of occlusion of facial salient regions and super-resolution image reconstruction through a single model, realizes image recovery from occlusion of salient regions and large-scale downsampling, improves the texture details and fidelity of facial image restoration, reduces the domain difference between generated images and real-world images, and can easily adapt super-resolution (SR) models to real LR data.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a method for completing local salient regions of a face based on super-resolution. In an offline stage, 1) a public data set is preprocessed and an occlusion training set is generated, which is used for pre-training a face completion generator in a face salient region completion network model; 2) a face parsing network is trained using the public data set for auxiliary training of a face super-resolution reconstruction generator; then, the network parameters of the pre-trained face completion generator are fixed, and the super-resolution reconstruction generator and the discriminator in the face salient region completion network model are trained in turn, and finally, the face salient region completion network model is trained as a whole; in an online stage, the trained face salient region completion network model is used to perform face completion processing on an image received in real time.

[0006] The public datasets include but are not limited to the Helen dataset and the CelebA dataset.

[0007] The preprocessing refers to: cropping and aligning the face images in the data set according to the positions of the eyes, nose and mouth in the face images provided in the public data set.

[0008] The occlusion training set is prepared by processing the high-definition face images in the public training set into 32×32 resolution face images with occlusion blocks of different sizes, specifically: firstly performing a bicubic downsampling operation on the high-definition face images to obtain 32×32 resolution face images, and then performing occlusion operations on the high-definition face images with different sizes and positions to obtain low-resolution face images with occlusion blocks of different sizes relative to the facial area; and randomly flipping and rotating the low-resolution face images with occlusion blocks for data enhancement to obtain low-resolution face images with occlusion of facial salient areas.

[0001] The pre-training refers to: inputting the pre-processed public data set and the occlusion training set into the face completion generator, and obtaining the pre-training loss function after weighting by style loss, perceptual loss, pixel loss, and smoothness loss, specifically: in: is the weighting parameter, i = 1, 2, 3, ..., 6, is the style loss, is the perceived loss, is the pixel loss, is the smoothing loss.

[0002] Preferably, the weighted parameter

[0009] The super-resolution-based face salient region completion network model includes: a face completion generator, a super-resolution reconstruction generator and a discriminator, wherein: the face completion generator performs completion processing based on an input low-resolution face image with occlusion and generates an unobstructed low-resolution face image, the super-resolution reconstruction generator performs super-resolution reconstruction processing based on the completed low-resolution face image to obtain a high-resolution unobstructed face image, and the discriminator performs discrimination processing based on the generated high-resolution face image and the real high-resolution face image to obtain a discrimination result for guiding the training of the generator.

[0010] The face completion generator comprises: a spectral normalization module, a convolutional layer, an encoder, a pyramid attention module, a decoder, a residual block, a convolutional layer and a Relu activation function, wherein: the spectral normalization module performs spectral normalization processing according to an input low-resolution occluded face image to obtain a normalized result; the convolutional layer performs convolution processing according to the normalized image to extract features and obtain a feature map; the encoder performs encoding processing according to the feature map extracted by the convolutional layer, further compresses the feature map, and obtains an encoded feature map; the pyramid attention module extracts significant key feature information of the face according to the feature map output by the encoder to obtain an enhanced feature map; the decoder performs decoding processing according to the enhanced feature map output by the pyramid attention module, restores the original size of the feature map, and obtains a decoded feature map; the residual block performs residual processing according to the decoded feature map to further improve the representation capability of the features; the convolutional layer and the Relu activation function perform convolution and activation processing according to the feature map output by the residual block to obtain the final completion result.

[0011] The super-resolution reconstruction generator includes: a spectral normalization module, a convolution layer, an average pooling layer, an encoder, a residual network, and an upsampling module, wherein: the spectral normalization module performs spectral normalization processing according to the completed low-resolution face image to obtain a normalized image; the convolution layer performs convolution processing according to the normalized image to extract features and obtain a feature map; the average pooling layer performs downsampling processing according to the feature map extracted by the convolution layer to obtain a low-dimensional feature map; the encoder performs encoding processing according to the low-dimensional feature map to further compress the feature map to obtain encoded deep features; the residual network performs residual processing according to the deep feature information to further improve the representation ability of the features. Skip connections are used in the residual network to retain the input feature information by directly adding the input to the output to obtain an enhanced feature map; the upsampling module performs upsampling processing according to the feature map output by the residual network to obtain a high-resolution image.

[0012] The super-resolution reconstruction generator uses the completed low-resolution face image as input, and obtains the loss function after weighting the adversarial loss, face prior loss, perceptual loss, pixel loss, smoothness loss and identity loss. It learns to generate more realistic images by minimizing the generative adversarial loss, so that the discriminator cannot accurately distinguish between the generated image and the real image. Specifically: in: is the weighting parameter, i = 1, 2, 3, ..., 6, is the face prior loss, To generate adversarial loss, is the perceived loss, is the pixel loss, To smooth the loss, For identity loss.

[0013] The weighted parameter is preferably

[0014] The generated adversarial loss is specifically: Wherein: pdata(x) represents the distribution of real face images, pz(z) represents the face image after occlusion, x represents face image data with pixel values ​​between 0 and 255, sampled from the real face image distribution pdata(x), z represents random noise, sampled from pz(z), noise is the input of generator G, used to generate data samples, G(z) represents the data sample generated by generator G with noise z as input, G represents generator, and D represents discriminator.

[0015] The auxiliary training refers to: using the Helen data set to pre-train a face parsing network, using the trained face parsing network to generate a face parsing graph of the CelebA real-world face image, and inputting it into the face super-resolution reconstruction generator for auxiliary training.

[0016] The overall training refers to: combining the generator and the discriminator for alternating training, using generative adversarial loss, perceptual loss, face prior loss and identity loss loss functions, and performing back propagation and parameter updating.

[0017] The style loss Where: K n is the normalization factor 1 / (Cn·Hn·Wn) of the nth layer VGG-19; Cn, Hn, Wn are the number of channels, height and width of the feature map respectively. and They represent the unobstructed ground truth low-resolution face image and the unobstructed ground truth high-resolution face image respectively. LR Represents the generated low-resolution face image, which is obtained by processing the original image. HR I′ represents the generated high-resolution face image, which is obtained by processing the generated low-resolution face image. LR Represents the intermediate low-resolution face image generated during the processing.

[0018] The pixel loss Among them: ||·|| is the L1 norm, and They represent the unobstructed ground truth low-resolution face image and the unobstructed ground truth high-resolution face image respectively. LR Represents the generated low-resolution face image, which is obtained by processing the original image. HRRepresents a generated high-resolution face image, which is obtained by processing the generated low-resolution face image.

[0019] The perceived loss Where: φ is VGG-19 trained on the ImageNet network, φ i,j represents the feature map of the jth convolutional layer before the i-th Max pooling layer; W i,j and H i,j Represents the dimension of the feature map. n is the index of the image in the batch, which ranges from 0 to N-1, where N is the number of images in the batch.

[0020] The smoothing loss Where: W and H are the width and height of the face image restored by the generator G, i ranges from 0 to H-1, and j ranges from 0 to W-1.

[0021] The face prior loss Where: I landmark_p ,I landmark_gt ,I parsing_p and I parsing_gt They respectively represent the face landmark heat map predicted by the generated face image, the face landmark heat map predicted by the real-world face image, the face parsing map predicted by the generated face image, and the face parsing map predicted by the real-world face image. ||·||2 represents the L2 norm. The values ​​of α and β need to be determined through hyperparameter tuning, and the optimal values ​​are continuously adjusted during the iterative training process.

[0022] Preferably, α and β are between 0.1 and 10.

[0023] The loss of identity Among them: CNN E (I SR )and The model is CNN E Extracted Image I SR and The identity feature of , and ||·||2 represents the L2 norm. Technical Effects

[0024] The present invention uses a context encoder in a face completion generator and adds a pyramid attention mechanism therein; uses an encoder and a residual network in a face super-resolution reconstruction generator, wherein the added encoder structure can encode high-level features of the face to enhance semantic information; uses a residual structure in the encoder and adds a maximum average pooling layer, and better learns residual information while retaining features; compared with the prior art, the present invention can better utilize context features to extract multi-scale information from significant areas such as facial features, improve the completion and restoration effect of facial features, and help the face super-resolution reconstruction part extract more useful information; and help the subsequent residual network extract more significant texture information of facial features. The nonlinear expression ability of the network is enhanced, the expression ability of the network is improved, and the deep feature extraction can better help the generator generate finer facial features details, so that the restored image has a good visual effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 It is a flow chart of the present invention;

[0004] Figure 2 is a schematic diagram of an embodiment;

[0005] Figure 3 It is a schematic diagram of the model of the present invention;

[0006] Figure 4 It is a schematic diagram of the discriminator of the present invention;

[0007] The real faces in the picture have been processed for privacy protection;

[0008] Figure 5 It is a schematic diagram of the effect of the embodiment;

[0009] In the figure: (a) is a low-resolution image (32x32) with obstruction; (b) is a high-resolution salient area image after completion and reconstruction; (c) is a high-resolution salient area image of the real world. DETAILED DESCRIPTION

[0010] like Figure 1 As shown, this embodiment relates to a method for completing local salient areas of a face based on super-resolution, including:

[0011] Step 1) Train the face parsing network using public datasets, specifically: download the CelebA and Helen datasets and preprocess them, divide 2000 images in the Helen dataset into a training set and 300 images as a test set; train the face parsing network on the Helen dataset;

[0012] The face parsing network decomposes the face image into different facial regions, such as eyes, nose, and mouth. The segmented face parsing map output by the network is used as auxiliary supervision to train the face super-resolution reconstruction generator, which specifically includes:

[0013] a: Input the face image Im in the Helen training set into the face parsing network FP, generate a face feature map through multiple convolutional layers and pooling layers, and fuse it through fusion modules such as residual networks to obtain a weighted fused feature map. The fused feature map is mapped to the facial structure information Pm of the face image with the same resolution as the original image through a convolutional layer.

[0014] b: The trained face parsing network is applied to predict the face parsing graph for each face image in the CelebA dataset and to assist in the training of the face super-resolution generator.

[0015] Step 2) Preprocess the public dataset and generate an occlusion training set for pre-training the face completion generator in the face salient region completion network model, specifically including:

[0016] 2.1 Get the CelebA dataset and divide it into training set and test set.

[0017] 2.2 Obtain the WiderFace dataset, extract low-resolution face images with occlusion, and perform preprocessing.

[0018] 2.3 Process the test set of the CelebA dataset to obtain a low-resolution image set (32×32), and perform different degrees of occlusion processing on the eyes, nose, and mouth of the low-resolution image set; use the CelebA test set and the WiderFace dataset as the test set for the entire network model.

[0019] 2.4 Process the training set of the data set to obtain a low-resolution image set (32×32) and a high-resolution image set (128×128), and perform occlusion processing of different sizes and positions on the eyes, nose, and mouth parts of the low-resolution image set.

[0020] 2.5 Input the occluded low-resolution image and high-resolution image into the face completion generator to complete the pre-training of the face completion module, including:

[0021] a: Prepare occluded low-resolution images and corresponding high-resolution face images, define the face completion generator model and style loss, perceptual loss, pixel loss, and smoothness loss functions, and use the Adam optimizer to accelerate the convergence of the model.

[0022] b: Perform a training cycle, define batches, number of training rounds, perform forward propagation, calculate losses, backpropagation and parameter updates. The goal is to enable the face completion generator to generate unobstructed low-resolution face images.

[0023] Step 3) fix the network parameters of the pre-trained face completion generator, use the occluded low-resolution training set obtained in step 2 to train the super-resolution reconstruction generator in the face salient area completion network model, and use the trained face parsing network for auxiliary training, specifically including:

[0024] 3.1 Fix the parameters of the face completion generator: Freeze the parameters of the pre-trained face completion generator so that they remain unchanged during the second stage of training.

[0025] 3.2 Generate an unobstructed low-resolution training set: Use a fixed face completion generator to process the occluded low-resolution images to generate an unobstructed low-resolution training set.

[0026] 3.3 Construct a super-resolution reconstruction generator and a loss function including pixel loss, perceptual loss, face parsing loss, generative adversarial loss, and identity loss.

[0027] 3.4 Use the trained face parsing network for auxiliary training: Use the segmentation map output by the face parsing network as an auxiliary supervision signal, input the segmented face parsing map into the face super-resolution reconstruction generator and calculate the face parsing loss function to achieve auxiliary training.

[0028] 3.5 Perform training loop: Perform training of the super-resolution reconstruction generator, including forward propagation, loss calculation, back propagation, and parameter update.

[0029] Step 4) The occluded low-resolution training set obtained in step 2 is used to train the entire super-resolution-based face salient region completion network model, specifically including:

[0030] 4.1 Prepare occluded low-resolution images and corresponding high-resolution face images as input for training the entire network model.

[0031] 4.2 Perform adversarial training on the overall network model, alternately train the discriminator and generator, and generate more realistic high-resolution face images by generating adversarial loss, perceptual loss, parsing loss, and identity loss.

[0032] Step 5) In the online stage, the trained face salient region completion network model is used to restore the real-time input image.

[0033] After specific practical experiments, the experiment was conducted using an NVIDIA 2080Ti graphics card under the specific environment setting of Python 3.8. This embodiment includes obtaining low-resolution face images from the WiderFace dataset and the CelebA dataset, and performing occlusion operations and other processing on the low-resolution face images; the processed low-resolution face images are transmitted to the trained super-resolution-based face saliency region completion model network for reconstruction. In the training phase, a face parsing network is first trained on the Helen dataset. The images in the CelebA dataset are processed to obtain high-resolution images (32×32) and occluded low-resolution images (128×128), and the two images are input into the designed generative adversarial network for training, and the pre-trained face parsing network is used for auxiliary training. The trained network is used to complete the occluded low-resolution face images in the test phase to generate restored face images by super-resolution reconstruction.

[0034] This experiment uses peak signal-to-noise ratio PSNR and structural similarity index SSIM as evaluation indicators. When trained to 200 rounds, the model of the present invention reaches the best standard, with PSNR reaching 27.02 and SSIM reaching 0.793.

[0035] Compared with the existing technology, the upper Siwen encoder structure and pyramid attention module added to the face completion generator of this method effectively improve the completion ability of the salient areas of low-resolution face images. The encoder structure and residual network used in the face super-resolution reconstruction generator can better reconstruct low-resolution face images in super-resolution, so that the entire network model effectively improves the fidelity of the generated face image when performing super-resolution-based face salient area completion. By using the super-resolution-based face salient area completion network to solve the problem of occluded low-resolution face restoration, a small network model deployed on a mobile device can be obtained, which can effectively provide assistance in the field of face recognition. Figure 5 As shown, the test results of the present invention on the deep learning model obtained through training are shown, and the test results only show the significant areas (eyes, nose and mouth).

[0036] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A method for completing local salient regions of a face based on super-resolution, characterized in that: In the offline stage, ① the public data set is preprocessed and an occlusion training set is generated to pre-train the face completion generator in the face salient region completion network model; ② the face parsing network is trained using the public data set for auxiliary training of the face super-resolution reconstruction generator; then the network parameters of the pre-trained face completion generator are fixed, and the super-resolution reconstruction generator and discriminator in the face salient region completion network model are trained in turn, and finally the face salient region completion network model is trained as a whole; in the online stage, the trained face salient region completion network model is used to complete the face of the real-time received image; The super-resolution-based face salient region completion network model comprises: a face completion generator, a super-resolution reconstruction generator and a discriminator, wherein: the face completion generator performs completion processing based on an input low-resolution face image with occlusion and generates an unobstructed low-resolution face image, the super-resolution reconstruction generator performs super-resolution reconstruction processing based on the completed low-resolution face image to obtain a high-resolution unobstructed face image, and the discriminator performs discrimination processing based on the generated high-resolution face image and the real high-resolution face image to obtain a discrimination result for guiding the training of the generator; The super-resolution reconstruction generator uses the completed low-resolution face image as input, and obtains the loss function after weighting the adversarial loss, face prior loss, perceptual loss, pixel loss, smoothness loss and identity loss. It learns to generate more realistic images by minimizing the generative adversarial loss, so that the discriminator cannot accurately distinguish between the generated image and the real image. Specifically: in: is the weighting parameter, i = 1, 2, 3, ..., 6, is the face prior loss, To generate adversarial loss, is the perceived loss, is the pixel loss, To smooth the loss, For identity loss.

2. The method for completing local salient regions of a face based on super-resolution according to claim 1, characterized in that: The preprocessing refers to: cropping and aligning the face images in the data set according to the positions of the eyes, nose and mouth in the face images provided in the public data set.

3. The method for completing local salient regions of a face based on super-resolution according to claim 1, characterized in that: The occlusion training set is prepared by processing the high-definition face images in the public training set into 32×32 resolution face images with occlusion blocks of different sizes, specifically: firstly performing a bicubic downsampling operation on the high-definition face images to obtain 32×32 resolution face images, and then performing occlusion operations on the high-definition face images with different sizes and positions to obtain low-resolution face images with occlusion blocks of different sizes relative to the facial area; and randomly flipping and rotating the low-resolution face images with occlusion blocks for data enhancement to obtain low-resolution face images with occlusion of facial salient areas.

4. The method for completing local salient regions of a face based on super-resolution according to claim 1, characterized in that: The pre-training refers to: inputting the pre-processed public data set and the occlusion training set into the face completion generator, and obtaining the pre-training loss function after weighting by style loss, perceptual loss, pixel loss, and smoothness loss, specifically: in: is the weighting parameter, i = 1, 2, 3, ..., 6, is the style loss, is the perceived loss, is the pixel loss, is the smoothing loss.

5. The method for completing local salient regions of a face based on super-resolution according to claim 1, characterized in that: The super-resolution reconstruction generator includes: a spectral normalization module, a convolution layer, an average pooling layer, an encoder, a residual network, and an upsampling module, wherein: the spectral normalization module performs spectral normalization processing according to the completed low-resolution face image to obtain a normalized image; the convolution layer performs convolution processing according to the normalized image to extract features and obtain a feature map; the average pooling layer performs downsampling processing according to the feature map extracted by the convolution layer to obtain a low-dimensional feature map; the encoder performs encoding processing according to the low-dimensional feature map to further compress the feature map to obtain encoded deep features; the residual network performs residual processing according to the deep feature information to further improve the representation ability of the features. Skip connections are used in the residual network to retain the input feature information by directly adding the input to the output to obtain an enhanced feature map; the upsampling module performs upsampling processing according to the feature map output by the residual network to obtain a high-resolution image.

6. The method for completing local salient regions of a face based on super-resolution according to claim 1, characterized in that: The generated adversarial loss is specifically: Wherein: pdata(x) represents the distribution of real face images, pz(z) represents the face image after occlusion, x represents face image data with pixel values ​​between 0 and 255, sampled from the real face image distribution pdata(x), z represents random noise, sampled from pz(z), noise is the input of generator G, used to generate data samples, G(z) represents the data sample generated by generator G with noise z as input, G represents generator, and D represents discriminator.

7. The method for completing local salient regions of a face based on super-resolution according to claim 1, characterized in that: The auxiliary training refers to: using the Helen dataset to pre-train a face parsing network, using the trained face parsing network to generate face parsing graphs of CelebA real-world face images and inputting them into a face super-resolution reconstruction generator for auxiliary training; The overall training refers to: combining the generator and the discriminator for alternating training, using generative adversarial loss, perceptual loss, face prior loss and identity loss loss functions, and performing back propagation and parameter updating.

8. The method for completing local salient areas of a face based on super-resolution according to claim 1 or 7, characterized in that: The face prior loss Where: I landmark_p ,I landmark_gt ,I parsing_p and I parsing_gt They represent the face landmark heat map predicted by the generated face image, the face landmark heat map predicted by the real-world face image, the face parsing map predicted by the generated face image, and the face parsing map predicted by the real-world face image, respectively. ∥·∥2 represents the L2 norm. The values ​​of α and β need to be determined by hyperparameter tuning, and the optimal values ​​are continuously adjusted during the iterative training process. The loss of identity Among them: CNN E (I SR )and The model is CNN E Extracted Image I SR and The identity feature of , and ||·||2 represents the L2 norm.

9. The method for completing local salient regions of a face based on super-resolution according to claim 4 or 7, characterized in that: The style loss Where: K n is the normalization factor 1 / (Cn·Hn·Wn) of the nth layer VGG-19; Cn, Hn, Wn are the number of channels, height and width of the feature map respectively, and denote the unobstructed ground truth low-resolution face image and the unobstructed ground truth high-resolution face image, respectively, and I LR Represents the generated low-resolution face image, which is obtained by processing the original image. HR represents the generated high-resolution face image, which is obtained by processing the generated low-resolution face image, I′ LR represents the intermediate low-resolution face image generated during the processing; The pixel loss Among them: ||·|| is the L1 norm, and denote the unobstructed ground truth low-resolution face image and the unobstructed ground truth high-resolution face image, respectively, and I LR Represents the generated low-resolution face image, which is obtained by processing the original image. HR represents a generated high-resolution face image, which is obtained by processing the generated low-resolution face image; The perceived loss Where: φ is VGG-19 trained on the ImageNet network, φ i,j represents the feature map of the jth convolutional layer before the i-th Max pooling layer; W i,j and H i,j represents the dimension of the feature map, n is the index of the image in the batch, ranging from 0 to N-1, where N is the number of images in the batch; The smoothing loss Where: W and H are the width and height of the face image restored by the generator G, i ranges from 0 to H-1, and j ranges from 0 to W-1.