Face image super-resolution method based on heat map prior guidance

Abstract

The application discloses a face image super-resolution method based on a heat map prior guide, which is composed of a super-resolution main branch and a heat map prior estimation branch on a network model structure. The super-resolution main branch receives a low-resolution input image, and outputs a high-resolution image after super-resolution reconstruction; the prior estimation branch takes the output of the super-resolution main branch as input, estimates key points, calculates a heat map, and then feeds the heat map as structural prior information to the super-resolution main branch to guide the super-resolution reconstruction. In the training stage, the two branches are mutually cooperative and are optimized together. In order to maximize the guiding role of the heat map prior, the application utilizes the heat map information from two levels. Finally, a discriminative network is introduced to supervise the training of the reconstruction network, and then a higher-quality SR result is reconstructed.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of face image super-resolution, and relates to a face image super-resolution method based on heat map prior guidance. BACKGROUND

[0002] Existing face-related tasks, such as face recognition, face alignment, expression recognition and three-dimensional face reconstruction, are all implemented based on clear high-resolution (HR) face data sets. When facing low-resolution (LR) face images, the effect decreases obviously. Therefore, face super-resolution (FSR) technology is particularly important in the fields of computer vision and biometric recognition. In addition, due to the limitations of traditional digital imaging devices, the obtained face images often undergo a series of degradation processes such as optical blurring and undersampling, and it is difficult to obtain a relatively clear image in visual sense. As an effective image restoration method, image super-resolution technology can effectively overcome the problem of low image resolution caused by the physical resolution limit of imaging devices and optical blurring.

[0003] Existing face image super-resolution methods based on deep learning can be mainly divided into five categories according to the different ways of utilizing face-specific information in the reconstruction process: general face super-resolution methods, prior-guided face super-resolution methods, attribute-constrained face super-resolution methods, identity-consistent face super-resolution methods and reference image-guided face super-resolution methods. Among them, the super-resolution method based on prior guidance has attracted widespread attention due to its superior reconstruction performance. The core idea is to establish the mapping relationship between low-resolution images and high-resolution images by mining the structured prior information of face images. A large number of studies have shown that the structured prior information of face images can effectively assist the super-resolution reconstruction process. However, the existing face super-resolution method based on prior guidance still has artifacts and distorted textures in the reconstruction results, especially in the important facial feature regions. The reason is that the prior information is not fully utilized and efficient. In addition, in the network model structure, the existing prior-guided super-resolution method is usually composed of a super-resolution backbone network and a prior estimation network, and the correlation between the two is not high enough, which is one of the reasons why the network model is limited. SUMMARY

[0004] The purpose of the present application is to provide a face image super-resolution method based on heat map prior guidance, which solves the problem of existing face super-resolution technology that the result image is too smooth and has artifacts and distorted textures.

[0005] The technical solution adopted by the present application is a face image super-resolution method based on heat map prior guidance, which specifically includes the following steps:

[0006] Step 1, pre-process the training data set to obtain the input image of the training set;

[0007] Step 2, perform the first super-resolution reconstruction on the input image obtained in step 1;

[0008] Step 3, perform the first prior estimation according to the result of the super-resolution reconstruction in step 2 to estimate the heat map prior information;

[0009] Step 4, perform the second super-resolution reconstruction;

[0010] Step 5, perform the second prior estimation according to the result of the super-resolution reconstruction in step 4 to estimate the heat map prior information;

[0011] Step 6, perform the third super-resolution reconstruction;

[0012] Step 7, perform the third prior estimation according to the result of the super-resolution reconstruction in step 6 to estimate the heat map prior information;

[0013] Step 8, perform the fourth super-resolution reconstruction to output a high-resolution image ;

[0014] Step 9, perform error gradient back propagation to optimize network parameters.

[0015] The features of the present application are:

[0016] The specific process of step 1 is:

[0017] Select A and B face images from Helen and CelebA face data sets as target images of the training set ; use bicubic interpolation algorithm to downsample high-resolution images in the training set to output low-resolution images as input images of the training set .

[0018] The specific process of step 2 is:

[0019] Step 2.1, use the pre-up sampling module to up sample the LR image by 2 times, and the process is represented as:

[0020] (1);

[0021] Wherein, represents pre-up sampling, represents the output feature of the pre-up sampling module;

[0022] Step 2.2, the is input into the reconstruction subject network, which is a super-resolution reconstruction network based on the feedback network, and the process is represented as:

[0023] (2);

[0024] wherein, represents the reconstruction subject, which takes the low-resolution feature as input and outputs the reconstructed feature map ;

[0025] Step 2.3, input into the post-upsampling module, which is represented as:

[0026] (3);

[0027] wherein, represents the post-upsampling module, and the output result is the reconstructed residual image. After adding the residual image to the interpolated up-sampling result of the input low-resolution image, the high-resolution image is obtained, which is represented as:

[0028] (4);

[0029] wherein, () represents the global interpolation up-sampling operation.

[0030] The specific process of step 3 is:

[0031] input the high-resolution image obtained in step 2 into the heatmap prior estimation branch, which is used to extract the coordinates of the 68 key points of the face, and then calculate the heat distribution map of the key points. The process can be represented as:

[0032] (5)

[0033] wherein, represents the heatmap extraction, and the 68 key heatmap coordinates obtained.

[0034] The specific process of step 4 is:

[0035] Step 4.1, the LR image is up-sampled by 2 times using the pre-up-sampling module, and the process is represented as:

[0036] (6);

[0037] wherein, represents the pre-up-sampling, and the output feature of the pre-up-sampling module;

[0038] Step 4.2, input into the reconstruction subject network, in the reconstruction subject network, the attention module estimates the five facial probability map of the branch The input is input into the reconstruction subject network, in the reconstruction subject network, the attention module estimates the five facial probability map of the branch The input is input into the reconstruction subject network, in the reconstruction subject network, the attention module estimates the five facial probability map of the branch The process is represented as:

[0039] (7);

[0040] Wherein, The reconstruction subject is output to obtain the reconstructed feature map ;

[0041] Step 4.3, input into the post-up sampling module, represented as:

[0042] (8);

[0043] Wherein, The post-up sampling module is represented as:

[0044] (9);

[0045] Wherein, The global interpolation up-sampling operation is represented as:

[0046] The specific process of step 9 is:

[0047] Step 9.1, calculate the pixel loss, alignment loss, perceptual loss, adversarial loss and total loss between the output result and the high-resolution target image;

[0048] Step 9.2, calculate the error gradient direction, update the parameters through the Adam optimizer, apply the updated network model to the next group of training samples, and repeat the process until the total loss of the network on the validation set is within the floating range of .

[0049] In step 9.1, the pixel loss is represented as:

[0050] (10);

[0051] Wherein, The nth target high-resolution image is represented as: The nth super-resolution reconstruction result is represented as: ​

[0052] The alignment loss is used to measure the difference between the face key point position of the reconstruction output result and the target position, and the expression is:

[0053] (11);

[0054] The perceptual feature is extracted by using the pre-trained LightCNN network, and then the mean square loss value is calculated, and the expression is:

[0055] (12);

[0056] The adversarial loss is described as:

[0057] (13);

[0058] Wherein, The discriminator is represented by D, The probability that the reconstruction result is more false than the real image is represented by P;

[0059] When the discriminant network and the perceptual loss are not added, the total loss function is described as:

[0060] (14);

[0061] When the discriminant network, the adversarial loss and the perceptual loss are added, the total loss function is represented as:

[0062] (15);

[0063] Wherein, , , The constant factor is used to balance each loss function, and is used to allocate the weight of each loss.

[0064] The beneficial effects of the present application are as follows:

[0065] 1. The overall network structure is composed of a super-resolution reconstruction main branch and a face heat map prior estimation branch, and the two are based on a feedback mechanism and cooperate with each other to realize super-resolution reconstruction from coarse to fine.

[0066] 2. The super-resolution reconstruction process of each round can be disassembled into 4 times of reconstruction, and the reconstruction result and the heat map prior information of the last time can be used for each reconstruction from the second time, further making the hidden features more fully utilized.

[0067] ​​3. In terms of heat map prior information utilization, the structural prior represented by the face heat map is utilized by the attention mechanism and the feature modulation module in multiple levels, so as to more fully utilize the face prior information to assist in super-resolution reconstruction, and promote the reconstructed edge to be clearer, the texture to be more abundant, and the high-resolution face image to be more realistic. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 is the super-resolution network structure diagram adopted in the face image super-resolution method based on heat map prior guidance of the present application;

[0069] Figure 2 is the prior extraction network structure diagram in the face image super-resolution method based on heat map prior guidance of the present application;

[0070] Figure 3 is the feature modulation module structure diagram in the face image super-resolution method based on heat map prior guidance of the present application;

[0071] Figure 4 is the discriminator network structure diagram in the face image super-resolution method based on heat map prior guidance of the present application;

[0072] Figure 5 is the result comparison diagram of embodiment 2 in the face image super-resolution method based on heat map prior guidance of the present application;

[0073] Figure 6 is the result comparison diagram of embodiment 3 in the face image super-resolution method based on heat map prior guidance of the present application. DETAILED DESCRIPTION

[0074] The present application will be described in detail below in combination with the drawings and specific embodiments.

[0075] The face image super-resolution method based on heat map prior guidance of the present application is composed of a super-resolution main branch and a heat map prior estimation branch in the network model structure. The network structure diagram is shown in Figure 1The super-resolution main branch receives a low-resolution input image, and a high-resolution image is obtained after super-resolution reconstruction. The prior estimation branch takes the output of the super-resolution main branch as input, estimates the key points, and then calculates the heat map. The heat map is then fed back to the super-resolution main branch as structural prior information to guide the super-resolution reconstruction. In the training stage, the two branches are optimized together. In order to maximize the guiding effect of the heat map prior, the heat map information is utilized from two levels. Finally, a discriminant network is introduced to supervise the training of the reconstruction network, and a higher quality SR result is reconstructed. Since the face super-resolution task not only requires a powerful super-resolution main network to learn the accurate mapping relationship from LR to SR, but also requires efficient and accurate prior information to guide the generation of high-resolution images. Given that the size of the input face image is very small, most existing face heat map estimation methods are difficult to accurately estimate the face feature heat map. In order to obtain more accurate face feature structure information, the hourglass structure face key point estimation network is used to obtain the face heat map information.

[0076] In general, the method mainly includes two stages: training stage and test stage. The specific implementation steps of the training stage include the following:

[0077] Step 1, training data set preprocessing, step 1 is implemented according to the following steps:

[0078] Select 2000 and 168800 face images from Helen and CelebA face data sets respectively, and adjust them to 128x128 size (high-resolution image) as target images of the training set ; The bicubic interpolation algorithm is used to downsample the high-resolution images in the training set, and the output size is 16x16 low-resolution image, which is used as the input image of the training set .

[0079] Step 2, the first super-resolution reconstruction, i.e. the first forward propagation of the low-resolution input image in the super-resolution network.

[0080] Step 2 is implemented according to the following steps:

[0081] The super-resolution main network is composed of three parts: pre-up sampling module, reconstruction main body based on feedback network, and post-up sampling module.

[0082] Step 2.1, the pre-up sampling module up samples the LR image by 2 times, which can be represented as:

[0083] (1);

[0084] wherein, denotes pre-up sampling, represents the output feature of the pre-up sampling module.

[0085] Step 2.2, input to the reconstruction subject network. The reconstruction subject network is based on the feedback network, and the process can be represented as:

[0086] (2);

[0087] wherein, represents the reconstruction subject, which takes the low-resolution feature as input, and outputs the reconstructed feature map .

[0088] Step 2.3, input to the post-up sampling module, represented as:

[0089] (3);

[0090] wherein, represents the post-up sampling module, and the output of the post-up sampling module is the reconstructed residual image. After adding the reconstructed residual image to the interpolated up-sampling result of the input low-resolution image, the high-resolution image is obtained, represented as:

[0091] (4);

[0092] wherein, () represents the global interpolation up-sampling operation, represents the interpolated up-sampling result of the low-resolution image.

[0093] Step 3, first prior estimation, according to estimate the heat map prior information

[0094] Step 3 is implemented according to the following steps:

[0095] input the first obtained high-resolution image to the heat map prior estimation branch, which is mainly composed of an hourglass network, and is used to estimate the heat distribution map from the key point position coordinates (the key point position coordinates are obtained in advance by the OpenFace model). The heat distribution map is used to generate the facial feature probability map and the prior feature. The process can be represented as:

[0096] (5);

[0097] wherein, represents the heat map extraction operation, represents the key point coordinates obtained from OpenFace, represents the obtained heat distribution map.

[0098] Step 4, the 2nd super-resolution reconstruction, i.e. the 2nd forward propagation of the low-resolution input image in the super-resolution network.

[0099] Step 4 is implemented according to the following steps:

[0100] Step 4.1, the pre-up-sampling module processes the LR image Up-sampling 2 times, the process can be represented as:

[0101] (6);

[0102] wherein, denotes the interpolation pre-up-sampling, denotes the output feature of the pre-up-sampling module.

[0103] Step 4.2, the is input to the reconstruction main network. In the reconstruction main network, the output feature of the pre-up-sampling module is fused with the five-probability maps (in the attention fusion module), and the feature fine-tuning module takes the depth heat map prior as input to perform attention enhancement and feature modulation on the intermediate features in the network, the detailed process is shown in steps 4.2.1-4.2.3. At the same time, the reconstruction main network also receives the last forward propagation feature and the pre-up-sampling input, the process can be represented as:

[0104] (7);

[0105] wherein, denotes the reconstruction main network, and the output obtains the reconstructed feature map .

[0106] Step 4.2.1, since the heat map cannot be directly used by the attention fusion module and the feature modulation module, it is necessary to first divide the 68-channel keypoint heat map into P groups, each group representing a five-organ structure, such as eyes, nose and mouth, etc., denoted as . The Softmax function is used to convert the numerical value in the heat map to a probability value, and finally the probability map information is calculated, which is mathematically expressed as:

[0107] (8);

[0108] wherein, denotes the probability that the keypoint at position belongs to the P group of five-organ categories. This probability map will be used by the attention fusion module and the feature modulation module respectively.

[0109] Step 4.2.2, the probability map information is input into the attention fusion module in the super-resolution backbone branch. The attention fusion module is at the input end of the reconstruction main body, which divides the low-resolution input feature map into groups, and then performs a point multiplication operation with the five facial probability maps P to obtain the attention-enhanced feature map , and the calculation formula is represented as:

[0110] (9);

[0111] wherein, represents a point-to-point multiplication operation, represents a low-resolution feature map of P channels. Since the probability value of the facial feature region is high, after the multiplication operation, the reconstruction main body will pay more attention to the recovery of the facial feature region, thereby improving the model's ability to recover high-frequency details in important regions of the face.

[0112] Step 4.2.3, the probability map information is input into the feature modulation module in the super-resolution backbone branch. The main body of the feature modulation module is a spatial feature transformation layer, which receives the deep structure prior in the heat map to generate a set of modulation parameters and , and then performs affine transformation on the high-level features to achieve the purpose of feature modulation. Among them, the deep structure prior is obtained from the facial feature heat map through a lightweight feature extraction module, and the process can be represented as:

[0113] (10);

[0114] (11);

[0115] wherein, represents the facial feature probability map, represents the feature extraction module, which is composed of a set of convolutional layers, see Figure 2 . represents the feature modulation module, which is mainly composed of convolutional layers, see the structure diagram Figure 3 , and the modulation parameters and are learned by the network. Next, according to the modulation parameters, the affine transformation of the current feature is performed, and the calculation formula is represented as:

[0116] (12);

[0117] wherein, represents a point-to-point multiplication operation, ​​output features of the previous layer.

[0118] Step 4.3, finally to the post-up sampling module, denoted as:

[0119] (13);

[0120] wherein, denotes the post-up sampling module, and the output of the post-up sampling module is a reconstructed residual image; the high-resolution image is obtained by adding the reconstructed residual image to the interpolated up-sampling result of the input low-resolution image, denoted as:

[0121] (14);

[0122] wherein, denotes the global interpolation up-sampling operation.

[0123] Step 5, second prior estimation, according to estimating the heat map prior information

[0124] Step 5 is implemented according to the following steps:

[0125] Step 5.1, inputting the high-resolution image to the heat map prior estimation branch to estimate the heat distribution map (the same as step 3), and the process can be denoted as:

[0126] (15);

[0127] wherein, denotes the heat map extraction operation, denotes the key point coordinates obtained from OpenFace, denotes the obtained heat distribution map.

[0128] Step 6, third super-resolution reconstruction, i.e., the third forward propagation of the low-resolution input image in the super-resolution network. Step 6 is implemented approximately the same as step 4, and specifically:

[0129] Step 6.1, pre-up sampling is performed, and the process can be denoted as:

[0130] (16);

[0131] wherein, denotes the interpolation pre-up sampling, denotes the output features of the third pre-up sampling module.

[0132] Step 6.2, inputting the output features of the third pre-up sampling module The input is fed into the reconstruction network. Within the reconstruction network, the output features of the pre-upsampling module are compared with the facial feature probability map. The feature fine-tuning module performs fusion using a depth heatmap as a priori. As input, attention enhancement and feature modulation are applied to the intermediate features of the network. Details on obtaining the facial probabilistic map and depth heatmap priors are provided in sections 4.2.1-4.2.3. Simultaneously, the reconstructed subject also receives features from the previous forward propagation. With the pre-upsampled input, this process can be represented as:

[0133] (17);

[0134] in, The main body is represented, and the output is the reconstructed feature map. .

[0135] Step 6.3, finally... The input to the upsampling module is represented as:

[0136] (18);

[0137] in, () indicates the post-upsampling module, whose output is the reconstructed residual image. Adding this residual image to the interpolated upsampling result of the input low-resolution image yields the high-resolution image, represented as:

[0138] (19);

[0139] in, () indicates a global interpolation upsampling operation.

[0140] Step 7, the third prior estimation, based on Estimate the prior information of the heatmap;

[0141] Step 7 is implemented in much the same way as step 5, specifically as follows:

[0142] Step 7.1: The high-resolution image obtained from the third reconstruction... Input the heatmap prior estimation branch to estimate the heat distribution map (same as step 3). This process can be represented as:

[0143] (20);

[0144] in, This indicates the heatmap extraction operation. This represents the keypoint coordinates obtained from OpenFace. This represents the obtained thermal distribution map.

[0145] Step 8, the fourth super-resolution reconstruction, is the fourth forward propagation of the low-resolution input image in the super-resolution network. For the specific implementation of Step 8, refer to Step 6. The entire process can be expressed as:

[0146] Step 8.1, perform pre-upsampling, which can be represented as:

[0147] (twenty one);

[0148] Step 8.2, use the results of the fourth pre-upsampling. The result of the third reconstruction The third prior information And the probability diagram of facial features in the third estimation Input to the reconstructed main network and output .

[0149] (twenty two);

[0150] Step 8.3, finally... The input is fed into the upsampling module, where it is added to the interpolated upsampling result of the input low-resolution image to obtain the high-resolution image representation. :

[0151] (twenty three);

[0152] (twenty four);

[0153] This completes one round of iterative training, and the final output is a high-resolution image. The number of iterations is determined by the fluctuation of the loss function during training. It is assumed that the total loss fluctuates within a range of variance. The training is now complete.

[0154] The stability calculation method is as follows:

[0155] (25);

[0156] Let x be the total loss value obtained in the x-th iteration. The average value of the loss is obtained through a total of i iterations.

[0157] Step 9: Backpropagate the error gradient to optimize network parameters; Step 9 is implemented in the following steps:

[0158] Step 9.1, calculate the pixel loss, alignment loss, perception loss, adversarial loss, total loss between the output result and the high-resolution target image. Among them, the network model of the present application uses only pixel loss and alignment loss without adding a discriminator network, at this time the method of the present application is named "the method of the present application"; in order to improve the perception quality, the method will also introduce a discriminator network to distinguish the true and false of the super-resolution generation result, the structure of the discriminator network is shown in Figure 4 , after adding the discriminator network, the perception loss and the adversarial loss are also introduced, at this time the method of the present application is named "the method of the present application GAN".

[0159] The pixel loss represents the Manhattan distance between the super-resolution result and the target image, compared with Loss, Loss is beneficial to restore clearer edges and improve the convergence speed of the network model. Therefore, the present application adopts Loss measures the reconstruction error between the reconstructed image and the target image. The pixel loss is expressed as:

[0160] (26);

[0161] Among them, Indicates the nth target high-resolution image, Indicates the nth super-resolution reconstruction result.

[0162] The alignment loss is used to measure the difference between the position of the face key point of the reconstructed output result And the target position The expression is:

[0163] (27);

[0164] The perception loss is used to improve the perception quality of the image, which measures the difference between the reconstructed result and the real target in the feature domain The present application adopts a pre-trained LightCNN network to extract perception features, and then calculates the mean square loss value, the expression is:

[0165] (28);

[0166] In order to make the reconstructed result more realistic to the target image, this chapter introduces a discriminator network, and the adversarial loss can be described as:

[0167] (29);

[0168] Among them, Indicates the discriminator, which indicates the probability that the real image is more real than the reconstructed result, and similarly, The probability that the reconstruction result is more false relative to the real image.

[0169] If we want to obtain a more stable and higher PSNR reconstruction result, we do not need to add a discriminant network and a perceptual loss, and the total loss function can be described as:

[0170] (30);

[0171] If we want to obtain a more clear and realistic perceptual oriented result, we need to add a discriminant network, an adversarial loss and a perceptual loss, and the total loss can be expressed as:

[0172] (31);

[0173] wherein, , , is a constant factor for balancing each loss function, which is used to assign weights to each loss, and the method sets , , .

[0174] Step 9.2, calculate the error gradient direction, and update the parameters through the Adam optimizer.

[0175] (32);

[0176] wherein, represents the model parameters, denotes the step size, is the gradient descent direction, is a tensor, is a constant.

[0177] Apply the network model after updating the parameters to the next set of training samples, and repeat this process until the total loss of the network on the validation set is within a floating range of variance . In the method experiment, after training all the training samples for 800 rounds, the total loss of the validation set is stable at , at which time the training can be completed.

[0178] In the test phase, the model parameters trained in the training phase are loaded into the method model, and after inputting the low resolution image, the high resolution image can be output, and the propagation process of the characteristics is similar to the reconstruction process in the training phase.

[0179] Example 1

[0180] Table 1 shows the PSNR values and SSIM values obtained by the prior art such as bicubic interpolation algorithm, EDSR, SRFBN, PSFR, FSRNet, FSRGAN, DIC, DICGAN and the method of the present application, respectively as shown in the following table:

[0181] Table 1: Comparison of indicators of the method of the present application and prior super-resolution methods

[0182]

[0183] By comparison, the method of the present application is superior to several other mainstream comparison methods in objective evaluation indicators.

[0184] Example 2

[0185] Figure 5 The comparison results of the images randomly selected from the face dataset Helen under 8 times magnification are shown; Figure 5 a in the above table is a super-resolution result obtained by directly using bicubic interpolation, Figure 5 b in the above table is a gradual semantic perception style transformation method, which uses a multi-scale pyramid to adjust and restore the features from coarse to fine, Figure 5 c in the above table uses a deep end-to-end network to realize super-resolution reconstruction using landmark heat maps and analysis maps and other geometric information, Figure 5 d in the above table proposes to perform iterative collaboration between two recursive networks, and focuses on face image restoration and landmark estimation, Figure 5 e in the above table is an extension of d in the above table, Figure 5 d in the above table is an extension of d in the above table, Figure 5 d in the above table is an extension of d in the above table, Figure 5 f in the above table is the method of the present application without using a generative adversarial network, Figure 5 g in the above table is the method of the present application, Figure 5 f in the above table is the method of the present application without using a generative adversarial network, Figure 6 h in the above table is a high-resolution target image, and by comparison, the method of the present application is superior to several other mainstream comparison methods in subjective quality evaluation, especially in the super-resolution reconstruction effect of the facial features, greatly reducing the facial feature asymmetry, artifacts and distorted texture, etc.

[0186] Example 3

[0187] Figure 6 The comparison results of the images randomly selected from the face dataset CelebA under 8 times magnification are shown; Figure 6 a in the above table is a super-resolution result obtained by directly using bicubic interpolation, Figure 6 b in the above table is a gradual semantic perception style transformation method, which uses a multi-scale pyramid to adjust and restore the features from coarse to fine, Figure 6The c in the method uses a deep end-to-end network, and landmark heat maps and parsing maps are used to realize super-resolution reconstruction, Figure 6 The d in the method is an extension of the d in the method, Figure 6 The e in the method is an extension of the d in the method, Figure 6 The d in the method is an extension of the d in the method, Figure 6 The d in the method is an extension of the d in the method, Figure 6 The f in the method does not use a generative adversarial network, Figure 6 The g in the method is an extension of the f in the method, ​ The g in the method is an extension of the f in the method, ​ The h in the method is a high-resolution target image, and the method is superior to other mainstream comparison methods in subjective quality evaluation, and when the magnification is large, the texture model is more conducive to restoring the high-frequency details lost in the low-resolution image, especially in the facial feature area, significantly reducing the facial feature asymmetry, artifacts and distorted texture.

Claims

1. A face image super-resolution method based on a heat map prior guide, characterized in that: Specifically comprising the following steps: Step 1, pre-process the training data set to obtain the input image of the training set; the specific process of the step 1 is: selecting the face image from the face data set as the target image of the training set ; using bicubic interpolation algorithm to down-sample the high-resolution image in the training set, outputting a low-resolution image as the input image of the training set ; Step 2, the first super-resolution reconstruction is carried out on the input image obtained in step 1; the specific process of the step 2 is: Step 2.1, pre-upsample the LR image with the pre-upsample module Up-sample by 2, the process is represented as: （1）； wherein, denotes pre-up-sampling, denotes output features of the pre-up-sampling module; Step 2.2, will The input is fed into the reconstruction master network, which is a super-resolution reconstruction network based on a feedback network. This process is represented as follows: （2）； wherein, denotes a reconstruction subject which takes as input low resolution features and outputs reconstructed feature maps ; Step 2.3, to the post-upsample module, is represented as: input to the post-upsample module, is represented as: （3）； wherein, denotes a post-up-sampling module, and the output is the reconstructed residual image. The high-resolution image is obtained by adding the reconstructed residual image and the interpolated up-sampling result of the input low-resolution image, denoted as: （4）； wherein () denotes a global interpolation up-sampling operation; Step 3, the first prior estimation is carried out according to the super-resolution reconstruction result obtained in step 2, and the heat map prior information is estimated; the specific process of the step 3 is: The high-resolution image obtained in step 2 The input heatmap prior estimation branch is used to extract the coordinates of 68 key points on the face, and then calculate the heat map of the key points. This process is represented as follows: （5） wherein, denotes the heat map extraction, denotes the obtained 68 key heat map coordinates; Step 4, the second super-resolution reconstruction is carried out; Step 5, the second prior estimation is carried out according to the super-resolution reconstruction result obtained in step 4, and the heat map prior information is estimated; Step 6, the third super-resolution reconstruction is carried out; Step 7, the third prior estimation is carried out according to the super-resolution reconstruction result obtained in step 6, and the heat map prior information is estimated; Step 8, fourth super-resolution reconstruction is performed, outputting a high resolution image ; Step 9, the error gradient is back propagated, and the network parameters are optimized.

2. The face image super-resolution method based on heat map prior guidance according to claim 1, characterized in that: The specific process of the step 4 is: Step 4.1, pre-upsample the LR image with a pre-upsample module Up-sample by 2, the process is represented as: （6）； wherein, denotes pre-up-sampling, denotes output features of the pre-up-sampling module; Step 4.2, will The input is fed into the reconstruction network, where the attention module estimates the probability maps of facial features in the branches based on prior information. As input, the feature modulation module uses the depth heatmap prior as input to perform attention enhancement and feature modulation on the intermediate features of the reconstructed subject network. At the same time, the reconstructed subject also receives the reconstructed features from the previous forward propagation. The process is represented as: （7）； wherein, represents a reconstruction subject, and outputs a feature map obtained after reconstruction ; Step 4.3, the input to the post-upsample module is represented as: input to the post-upsample module is represented as: （8）； wherein, () represents a post-up-sampling module, and the output result is a reconstructed residual image. After adding the reconstructed residual image to the interpolated up-sampling result of the input low-resolution image, a high-resolution image is obtained, which is represented as: （9）； wherein () denotes a global interpolation up-sampling operation.

3. The method of claim 2, wherein the method further comprises: The specific process of the step 9 is: Step 9.1, the pixel loss, the alignment loss, the perception loss, the adversarial loss and the total loss between the output result and the high-resolution target image are calculated; Step 9.2, calculate the error gradient direction, update the parameters by Adam optimizer, apply the network model after updating the parameters to the next set of training samples, and repeat this process until the total loss of the network on the validation set is in the floating range of variance .

4. The face image super-resolution method based on heat map prior guidance according to claim 3, characterized in that: In the step 9.1, the pixel loss is expressed as: （10）； wherein, denotes the nth target high-resolution image, denotes the nth super-resolution reconstruction result; The alignment loss is used to measure the difference between the reconstructed output result and the ground truth face key point positions and the target positions , expressed as: （11）； The pre-trained LightCNN network is used to extract the perception feature, and then the mean square loss value is calculated, and the expression is: （12）； The adversarial loss is described as: （13）； wherein, denotes the discriminator, denotes the probability that the reconstruction is more fake than the real image; When the discriminant network and the perception type loss are not added, the total loss function is described as: （14）； When the discriminant network, the adversarial loss and the perception loss are added, the total loss function is expressed as: （15）； wherein , , are constant factors for balancing the individual loss functions for assigning weights to the individual losses.

Images