Face image super-resolution method based on facial structure prior fusion network

Abstract

The invention discloses a facial image super-resolution method based on facial structure prior fusion network guidance. The method comprises the following steps: constructing a facial image super-resolution network based on facial structure prior; preprocessing the public data set, and dividing the data set into a training set and a test set; training a face image super-resolution network based on face structure prior guidance; and super-resolution is carried out on the images in the test set. A face image super-resolution model based on a generative adversarial network is constructed, a generator part is composed of a rough super-resolution module, a key point heat map extraction module, a fine super-resolution encoder and a fine super-resolution decoder, and a discriminator is composed of a convolutional layer and a maximum pooling layer. According to the face super-resolution model constructed by the method, the problems of detail missing and low trueness of a result image under a relatively large magnification factor in an existing face super-resolution method are solved, so that a high-resolution face image with high trueness and a maintained structure can be reconstructed by the method.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and face super-resolution, and in particular relates to a face image super-resolution method based on guidance of a face key point heat map. Background Art

[0002] High-definition facial images have very important application significance: in the academic field, facial image super-resolution technology can effectively improve the accuracy of downstream facial analysis tasks, such as: face alignment, face parsing, face recognition and 3D face reconstruction; in the industrial field, facial image super-resolution technology can provide higher accuracy for smart transportation, identity authentication, security monitoring, sentiment analysis and other systems.

[0003] With the rapid development of application fields such as face recognition, security monitoring, medical imaging, photography and video processing, the demand for high-quality face images is growing. However, due to the limitations of hardware equipment and differences in acquisition conditions, in practice, only low-resolution face images can be obtained. These images are obviously insufficient in detail, clarity and recognition, and it is difficult to meet application requirements. At the same time, it costs a lot to improve the performance of the equipment from the hardware level. Therefore, as a software-level method, face image super-resolution technology is simple to implement, low-cost, and has very important application value.

[0004] Facial image super-resolution algorithms can be divided into the following two categories: interpolation-based super-resolution algorithms and deep learning-based super-resolution methods. The former estimates the value of the newly inserted pixel by considering the neighborhood information of the pixel in the original image, and is less used at present. With the development of deep learning technology, the latter has become mainstream. However, most general deep learning-based super-resolution methods only train the model by minimizing the root mean square error, which easily produces over-smoothed image texture details. At the same time, many methods ignore the unique geometric structure of facial images and the human eye's perception of image style content. Therefore, designing a loss function that effectively improves model performance and effectively utilizing the structural information of the face is an important issue facing facial image super-resolution methods. Summary of the invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a face image super-resolution method based on a facial structure prior fusion network, which learns the facial structure prior, designs a prior fusion network, and promotes a more accurate face super-resolution process, thereby solving the problem that the existing face image super-resolution method ignores the facial structure and the image texture details are over-smoothed.

[0006] To achieve the above object, the technical solution of the present invention is: a facial image super-resolution method based on a facial structure prior fusion network, comprising the following steps:

[0007] S1: construct the coarse super-resolution module, key point heat map extraction module, fine super-resolution encoder, fine super-resolution decoder and discriminator module respectively, and jointly construct the facial structure prior fusion face super-resolution network;

[0008] S2: Perform Bicubic downsampling on high-definition face images in the public dataset, construct low / high resolution face image pairs, and divide them into training and test sets;

[0009] S3: Train the face super-resolution model based on the training set;

[0010] S4: Input the images in the test set into the trained face image super-resolution network guided by facial structure priors, and output the reconstructed high-resolution face image.

[0011] Furthermore, the construction process of the face image super-resolution network based on facial structure prior guidance in step S1 is specifically as follows:

[0012] A1: Construct a coarse super-resolution module to perform preliminary upsampling of the input low-resolution face image and reconstruct a coarse super-resolution result; construct a fine super-resolution encoder to encode the face feature map of the coarse super-resolution result; construct a key point heat map extraction module to generate a face key point heat map of the coarse super-resolution result;

[0013] A2: Build a fine super-resolution decoder to decode the parallel results of the face feature map and the face key point heat map to generate a fine high-resolution face image; build a discriminator module to act on the fine high-resolution face image to improve the image authenticity.

[0014] Furthermore, the construction process of the coarse super-resolution module in step A1 is specifically as follows:

[0015] B1: Constructing residual module: The residual module consists of a convolution layer with a kernel size of 3*3 and a stride of 1, a batch layer, a RELU activation function, and a skip connection;

[0016] B2: Construct a coarse super-resolution module: The coarse super-resolution module consists of a convolutional layer and a residual module. First, a convolutional layer with a convolution kernel size of 3*3, a step size of 1, and an output channel of 64 is stacked. Then three consecutive residual modules are cascaded. Finally, a convolutional layer with a convolution kernel size of 3*3, a step size of 1, and an output channel of 3 is cascaded to output a preliminary super-resolution face image.

[0017] Furthermore, the construction process of the key point heat map extraction module in step A1 is specifically as follows:

[0018] C1: Construct traditional hourglass network module;

[0019] C2: stack a convolution layer with a convolution kernel size of 7*7, a stride of 2, and an output channel of 64, then cascade three consecutive residual modules, and finally cascade two consecutive hourglass network modules to output the heat map features of facial key points.

[0020] Furthermore, the construction process of the fine super-resolution encoder in step A1 is specifically as follows:

[0021] D1: stack a convolutional layer with a kernel size of 3*3, a stride of 2, and an output channel of 64, then cascade 12 residual modules, and finally cascade a convolutional layer with a kernel size of 3*3, a stride of 1, and an output channel of 64.

[0022] Furthermore, the construction process of the fine super-resolution decoder in step A2 is specifically as follows:

[0023] E1: stack a convolution layer with a kernel size of 3*3, a stride of 1, and an output channel of 64, then cascade a transposed convolution layer with a kernel size of 3*3, a stride of 2, and an output channel of 64, then cascade three residual modules, and finally cascade a convolution layer with a kernel size of 3*3, a stride of 1, and an output channel of 3.

[0024] Furthermore, the construction process of the discriminator in step A2 is specifically as follows:

[0025] F1: Construct a downsampling module consisting of a convolutional layer and a maximum pooling layer;

[0026] F2: Stack 5 consecutive downsampling modules, then cascade a reshape layer to convert the features into a one-dimensional vector, then cascade a fully connected layer, and finally cascade a sigmoid activation layer.

[0027] Furthermore, the process of step S2 is specifically as follows:

[0028] G1: Preprocess the published dataset and adjust the images in the dataset to the target size using bilinear interpolation;

[0029] G2: Normalize the pixel matrix of each image adjusted in step G1 as the original high-resolution image;

[0030] G3: Downsample the image adjusted in G2 by eight times using bilinear interpolation to obtain the corresponding low-resolution image;

[0031] G4: Randomly perturb the low-resolution original image and high-resolution original image data in pairs, and select 80% of the data as the training data set and the rest as the test data set.

[0032] Furthermore, the process of step S3 is specifically as follows:

[0033] H1: Construct mean square error as the loss function;

[0034] H2: Construct a content loss function based on the VGG19 network;

[0035] H3: Construct a content loss function of face key point heat map based on 2D-FAN network;

[0036] H4: Construct adversarial loss function;

[0037] H5: Initialize the parameters of each module in the face image super-resolution network guided by the face key point heat map;

[0038] H6: Set the learning rate, number of model training iterations, and number of batch training samples;

[0039] H7: Use the low-resolution images and corresponding high-resolution images in the training dataset to train the network, and use the optimization algorithm to update the parameters in the face image super-resolution network guided by the face key point heat map;

[0040] H8: Train the face image super-resolution network guided by the face key point heatmap until the number of iterations reaches a preset value.

[0041] Furthermore, the content loss function L based on the VGG19 network in step H2 C for:

[0042]

[0043] Among them, I LR ,I HR They represent the input low-resolution image and the corresponding high-resolution image, respectively, G(·) represents the high-resolution image generated by the network, W represents the feature map obtained by the jth convolution (after activation) before the i-th maximum pooling layer in the pre-trained VGG19 network, N represents the number of all feature encoding layers in the VGG19 network, and x and y represent the pixel-by-pixel coordinates of the current feature map. i，j , H i，j Represents the length and width of the feature map of the corresponding layer in the VGG network.

[0044] Furthermore, the face key point heat map content loss function L based on the 2D-FAN network in step H3 P for:

[0045]

[0046] Where C(·) represents the generated coarse super-resolution image, P(·) represents the prior information extracted by the key point heat map extraction module, FAN(·) represents the face key point heat map generated by 2D-FAN, and W and H represent the length and width of the face key point heat map generated by 2D-FAN, respectively.

[0047] Furthermore, the adversarial loss function in step H4 is:

[0048]

[0049]

[0050] Among them, L G is the adversarial loss function constraining the generator, L D is the adversarial loss function constraining the discriminator, h is the generated face image of the generator, is the true face image, D(·) represents the output of the discriminator, Pg represents the distribution of high-resolution images generated by the network, and Pd represents the distribution of the original high-resolution images.

[0051] Compared with the prior art, the present invention has the following significant advantages: the face super-resolution model constructed by the present invention overcomes the problems of missing image details and low authenticity in the existing face super-resolution methods at a large magnification, so that the present invention can reconstruct high-resolution face images with high authenticity and preserved structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0053] Figure 2 Schematic diagram of the face image super-resolution network structure based on facial structure prior guidance.

[0054] Figure 3 Schematic diagram of the structure of the residual module.

[0055] Figure 4 Schematic diagram of the hourglass network module structure.

[0056] Figure 5 The figure is a comparison diagram of the test results with those of other similar methods. DETAILED DESCRIPTION

[0057] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments.

[0058] The present invention provides a facial image super-resolution method based on a facial structure prior fusion network, referring to Figure 1 , which comprises the following steps:

[0059] Step 1: Construct a coarse super-resolution module, which includes steps 1.1-1.2:

[0060] Step 1.1: Construct a residual module: The residual module consists of a convolution layer with a convolution kernel size of 3*3 and a step size of 1, a batch layer, a RELU activation function, and a skip connection. The specific structure is as follows Figure 3 As shown;

[0061] Step 1.2: Construct a rough super-resolution module: The rough super-resolution module consists of a convolutional layer and a residual module. First, a convolutional layer with a convolution kernel size of 3*3, a step size of 1, and an output channel of 64 is stacked, then three consecutive residual modules are cascaded, and finally a convolutional layer with a convolution kernel size of 3*3, a step size of 1, and an output channel of 3 is cascaded to output a preliminary super-resolution face image. The specific structure is as follows: Figure 2 shown.

[0062] Step 2: Construct a key point heat map extraction module, which includes steps 2.1-2.2:

[0063] Step 2.1: Construct an hourglass network module. The hourglass module is composed of a maximum pooling layer, a residual module, and a jump connection. It has a highly symmetrical network structure and is used to extract the heat map of key points in the face image. The specific structure is as follows: Figure 4 As shown;

[0064] Step 2.2: Stack a convolution layer with a convolution kernel size of 7*7, a stride of 2, and an output channel of 64, then cascade three consecutive residual modules, and finally cascade two consecutive hourglass network modules to output the heat map features of the key points of the face. The specific structure is as follows: Figure 3 shown.

[0065] Step 3: Construct a fine super-resolution encoder, which includes step 3.1:

[0066] Step 3.1: Stack a convolution layer with a kernel size of 3*3, a stride of 2, and an output channel of 64, then cascade 12 residual modules, and finally cascade a convolution layer with a kernel size of 3*3, a stride of 1, and an output channel of 64. The specific structure is as follows Figure 2 shown.

[0067] Step 4: Construct a fine super-resolution decoder, which includes step 4.1:

[0068] Step 4.1: Stack a convolution layer with a kernel size of 3*3, a stride of 1, and an output channel of 64, then cascade a transposed convolution layer with a kernel size of 3*3, a stride of 2, and an output channel of 64, then cascade three residual modules, and finally cascade a convolution layer with a kernel size of 3*3, a stride of 1, and an output channel of 3. The specific structure is as follows: Figure 2 shown.

[0069] Step 5: Build the discriminator module, which includes steps 5.1-5.2:

[0070] Step 5.1: Construct a downsampling module consisting of a convolutional layer and a maximum pooling layer;

[0071] Step 5.2: Stack 5 consecutive downsampling modules, then cascade a reshape layer to convert the features into a one-dimensional vector, then cascade a fully connected layer, and finally cascade a sigmoid activation layer, and finally output the probability that the input image is generated by the generator. The specific structure is as follows Figure 2 shown.

[0072] Step 6: Construct a facial image super-resolution network based on facial structure prior guidance, which includes steps 6.1-6.2:

[0073] Step 6.1: Connect the keypoint heatmap extraction module and the fine super-resolution encoder in parallel;

[0074] Step 6.2: Cascade the coarse super-resolution module with the structure mentioned in step 6.1, then cascade it with the fine super-resolution decoder, and finally cascade the discriminator part. The specific structure is as follows Figure 2 shown.

[0075] Step 7: Preprocess the dataset and divide it into dataset and test set:

[0076] Step 7.1: Preprocess the published CelebAMask-HQ dataset and adjust the images in the dataset to the target size using bilinear interpolation;

[0077] Step 7.2: Normalize the pixel matrix of each image adjusted in step 7.1 as the original high-resolution image;

[0078] Step 7.3: Downsample the image adjusted in step 7.2 by 8 times using bilinear interpolation to obtain a corresponding low-resolution image;

[0079] Step 7.4: Randomly perturb the low-resolution original image and the high-resolution original image data in pairs, and select 80% of their data as the training data set and the rest as the test data set.

[0080] Step 8: Training a facial image super-resolution network based on facial structure prior guidance, which includes steps 8.1-8.8:

[0081] Step 8.1: Construct a loss function based on the mean square error to constrain the coarse super-resolved face image and the fine super-resolved face image;

[0082] Step 8.2: Construct a content loss function based on the VGG19 network. Its specific structure is as follows:

[0083]

[0084] Among them, I LR ,I HR They represent the input low-resolution image and the corresponding high-resolution image, respectively, G(·) represents the high-resolution image generated by the network, represents the feature map obtained by the jth convolution (after activation) before the i-th maximum pooling layer in the pre-trained VGG19 network, W i，j , H i，j Represents the dimension of the feature map of the corresponding layer in the VGG network.

[0085] Step 8.3: Construct a face key point heat map content loss function based on the 2D-FAN network. Its specific structure is as follows:

[0086]

[0087] Where C(·) represents the generated coarse super-resolution image, P(·) represents the prior information extracted by the key point heat map extraction module, FAN(·) represents the face key point heat map generated by 2D-FAN, and W and H represent the dimensions of the face key point heat map generated by 2D-FAN, respectively.

[0088] Step 8.4: Construct the adversarial loss function, whose specific structure is as follows:

[0089]

[0090]

[0091] Where D(·) represents the output of the discriminator, Pg represents the distribution of high-resolution images generated by the network, and Pd represents the distribution of original high-resolution images.

[0092] Step 8.5: Initialize the parameters of each module in the face image super-resolution network guided by facial structure prior;

[0093] Step 8.6: Set the learning rate, number of model training iterations, and number of batch training samples;

[0094] Step 8.7: Use the low-resolution images and corresponding high-resolution images in the training dataset to train the network, and use the optimization algorithm to update the parameters in the face image super-resolution network guided by the facial structure prior;

[0095] Step 8.8: Train the face image super-resolution network guided by facial structure prior until the number of iterations reaches a preset value.

[0096] Step 9: Input the images in the test dataset into the trained face image super-resolution network guided by facial structure prior, and output the reconstructed high-resolution face image.

[0097] In this embodiment, the face image finally output by the above method is compared with the face image obtained by other methods. The comparison results are as follows: Figure 5 As shown, from left to right are low-resolution image, Bicubic interpolation, SRGAN, FHC, FSRNET, the result of the present invention, and the original high-resolution image. By comparison, it can be proved that the face image reconstructed by the present invention has richer texture details in visual senses and has higher authenticity.

Claims

1. A facial image super-resolution method based on facial structure prior fusion network, characterized in that: The steps include: S1: construct the coarse super-resolution module, key point heat map extraction module, fine super-resolution encoder, fine super-resolution decoder and discriminator module respectively, and jointly construct the facial structure prior fusion face super-resolution network; S2: Perform Bicubic downsampling on high-definition face images in the public dataset, construct low / high resolution face image pairs, and divide them into training and test sets; S3: Train the face super-resolution model based on the training set; S4: Input the images in the test set into the trained face image super-resolution network guided by facial structure priors, and output the reconstructed high-resolution face image.

2. The facial image super-resolution method based on facial structure prior fusion network according to claim 1, characterized in that: The construction process of the face image super-resolution network based on facial structure prior guidance in S1 is as follows: A1: Construct a coarse super-resolution module to perform preliminary upsampling of the input low-resolution face image and reconstruct a coarse super-resolution result; construct a fine super-resolution encoder to encode the face feature map of the coarse super-resolution result; construct a key point heat map extraction module to generate a face key point heat map of the coarse super-resolution result; A2: Build a fine super-resolution decoder to decode the parallel results of the face feature map and the face key point heat map to generate a fine high-resolution face image; build a discriminator module to act on the fine high-resolution face image.

3. The facial image super-resolution method based on facial structure prior fusion network according to claim 2 is characterized in that: The construction process of the coarse super-resolution module in step A1 is specifically as follows: B1: Constructing residual module: The residual module consists of a convolution layer with a kernel size of 3*3 and a stride of 1, a batch layer, a RELU activation function, and a skip connection; B2: Construct a rough super-resolution module: The rough super-resolution module consists of a convolutional layer and a residual module. First, a convolutional layer with a convolution kernel size of 3*3, a step size of 1, and an output channel of 64 is stacked, and then three consecutive residual modules are cascaded. Finally, a convolutional layer with a convolution kernel size of 3*3, a step size of 1, and an output channel of 3 is cascaded to output a preliminary super-resolution face image. The construction process of the key point heat map extraction module is specifically as follows: C1: Construct traditional hourglass network module; C2: stack a convolution layer with a convolution kernel size of 7*7, a stride of 2, and an output channel of 64, then cascade three consecutive residual modules, and finally cascade two consecutive hourglass network modules to output the heat map features of facial key points.

4. The facial image super-resolution method based on facial structure prior fusion network according to claim 2, characterized in that: The construction process of the fine super-resolution encoder in step A1 is specifically as follows: stacking a convolution layer with a convolution kernel size of 3*3, a step size of 2, and an output channel of 64, then cascading 12 residual modules, and finally cascading a convolution layer with a convolution kernel size of 3*3, a step size of 1, and an output channel of 64; The construction process of the fine super-resolution decoder in step A2 is specifically as follows: stack a convolution layer with a convolution kernel size of 3*3, a step size of 1, and an output channel of 64, then cascade a transposed convolution layer with a convolution kernel size of 3*3, a step size of 2, and an output channel of 64, then cascade three residual modules, and finally cascade a convolution layer with a convolution kernel size of 3*3, a step size of 1, and an output channel of 3.

5. The facial image super-resolution method based on facial structure prior fusion network according to claim 2, characterized in that: The construction process of the discriminator in step A2 is specifically as follows: F1: Construct a downsampling module consisting of a convolutional layer and a maximum pooling layer; F2: Stack 5 consecutive downsampling modules, then cascade a reshape layer to convert the features into a one-dimensional vector, then cascade a fully connected layer, and finally cascade a sigmoid activation layer.

6. The facial image super-resolution method based on facial structure prior fusion network according to claim 1, characterized in that: The process of step S2 is specifically as follows: G1: Preprocess the published dataset and adjust the images in the dataset to the target size using bilinear interpolation; G2: Normalize the pixel matrix of each image adjusted in step G1 as the original high-resolution image; G3: Downsample the image adjusted in G2 by eight times using bilinear interpolation to obtain the corresponding low-resolution image; G4: Randomly perturb the low-resolution original image and high-resolution original image data in pairs, and select 80% of the data as the training data set and the rest as the test data set.

7. The facial image super-resolution method based on facial structure prior fusion network according to claim 1, characterized in that: The process of step S3 is specifically as follows: H1: Construct mean square error as the loss function; H2: Construct a content loss function based on the VGG19 network; H3: Construct a content loss function of face key point heat map based on 2D-FAN network; H4: Construct adversarial loss function; H5: Initialize the parameters of each module in the face image super-resolution network based on facial structure prior guidance: H6: Set the learning rate, number of model training iterations, and number of batch training samples; H7: Use the low-resolution images and corresponding high-resolution images in the training dataset to train the network, and use the optimization algorithm to update the parameters in the face image super-resolution network guided by the face key point heat map; H8: Train the face image super-resolution network guided by the face key point heatmap until the number of iterations reaches a preset value.

8. The facial image super-resolution method based on facial structure prior fusion network according to claim 7, characterized in that: The content loss function L based on the VGG19 network in step H2 C / i,j for: Among them, I LR ,I HR They represent the input low-resolution image and the corresponding high-resolution image, respectively, G(·) represents the high-resolution image generated by the network, W represents the feature map obtained by the jth convolution (after activation) before the i-th maximum pooling layer in the pre-trained VGG19 network, N represents the number of all feature encoding layers in the VGG19 network, and x and y represent the pixel-by-pixel coordinates of the current feature map. i,j , H i,j Represents the length and width of the feature map of the corresponding layer in the VGG network.

9. The facial image super-resolution method based on facial structure prior fusion network according to claim 7, characterized in that: The face key point heat map content loss function L based on the 2D-FAN network in step H3 P for: Where C(·) represents the generated coarse super-resolution image, P(·) represents the prior information extracted by the key point heat map extraction module, FAN(·) represents the face key point heat map generated by 2D-FAN, and W and H represent the length and width of the face key point heat map generated by 2D-FAN, respectively.

10. The facial image super-resolution method based on facial structure prior fusion network according to claim 7, characterized in that: The adversarial loss function in step H4 is: Among them, L G is the adversarial loss function constraining the generator, L D is the adversarial loss function constraining the discriminator, h is the generated face image of the generator, is the true face image, D(·) represents the output of the discriminator, Pg represents the distribution of high-resolution images generated by the network, and Pd represents the distribution of the original high-resolution images.

Images