A face super-resolution reconstruction method based on progressive training and face semantic segmentation

Abstract

The application provides a face super-resolution reconstruction method based on progressive training and face semantic segmentation, which mainly comprises the following steps: a light and efficient face super-resolution network is proposed, the network mainly comprises a residual aggregation module and three up-sampling modules; a progressive training method is used to make the model develop in three stages in one iteration training, and the trained model can perform two-fold, four-fold and eight-fold super-resolution image reconstruction on a low-resolution face image; a face semantic segmentation network is used to obtain face prior information, and a face segmentation loss is added in the model training to assist the network in generating a more realistic face structure. The application is improved based on the RFDN network, is suitable for super-low resolution face image input, can output three kinds of high-resolution reconstruction images with different magnification, and solves some defects of traditional models.

Description

Technical Field

[0001] This invention relates to the field of face image reconstruction technology, and in particular to a face super-resolution reconstruction method based on progressive training and face semantic segmentation. Background Technology

[0002] Image super-resolution refers to the process of restoring a low-resolution image or image sequence to a high-resolution image. Face image super-resolution, also known as face phantom reconstruction, is a branch of image super-resolution research that aims to reconstruct high-resolution face images from low-resolution ones. Currently, several industrial applications require large amounts of high-resolution face information: First, computer vision tasks such as face recognition, face parsing, and face alignment often require extracting rich semantic information from input images. These model algorithms can extract more image features from high-resolution faces, thereby improving the accuracy of the algorithm. Second, in the public security system, police officers frequently need to obtain facial information of criminal suspects through cameras. However, due to limitations in hardware and shooting environments, images captured by cameras are often low-resolution, making it extremely difficult to extract information from blurry images. Police need high-resolution face images to identify suspects.

[0003] Image interpolation is a simple and fast face super-resolution algorithm, widely used in various scenarios. However, it performs poorly when performing high-magnification magnification on extremely low-resolution images (e.g., 16×16 pixels). Deep learning-based face super-resolution algorithms can achieve good super-resolution results by leveraging the advantages of convolutional neural networks. However, existing deep learning-based face super-resolution algorithms have the following problems: First, many models stack more convolutional blocks or add more complex substructures to improve algorithm performance, which inevitably increases the computational complexity of the model. Models with a large number of parameters are difficult to train due to limited computing resources and are also difficult to deploy on some lightweight mobile devices. Second, a face super-resolution model can usually only perform magnification at a specific magnification. When facing the need for multiple magnification magnifications, additional resources must be spent to train a new model. Third, many current face super-resolution models use a large amount of manually labeled face prior information during training, which increases the workload for researchers. Summary of the Invention

[0004] To address the problems of existing technologies, this invention proposes a lightweight and efficient face super-resolution reconstruction method. Based on an improved RFDN (Residual Feature Distillation Network), it introduces a face semantic segmentation model to assist the network in acquiring prior face information. A progressive training approach is employed, allowing the model to be trained in three stages within a single iteration. Furthermore, a face segmentation loss is incorporated into the model training process to generate more realistic faces. This face super-resolution reconstruction method is applicable to ultra-low resolution (e.g., 16×16 pixels) face image inputs and can output high-resolution reconstructed images with three different magnifications.

[0005] To achieve the aforementioned objectives, the present invention employs the following technical solution: a face super-resolution reconstruction method based on progressive training and face semantic segmentation, comprising the following steps:

[0006] S1: Prepare training data, specifically including low-resolution face images I LR16 (16×16 pixels), high-resolution face image I HR32 (32×32 pixels), I HR64 (64×64 pixels), I HR128 (128×128 pixels).

[0007] S2: Propose a lightweight face super-resolution network LFSRNet, which consists of a residual aggregation block (RAB) and three upsampling blocks.

[0008] 1) The Residual Aggregation Module (RAB) is used to obtain image features of low-resolution images. The RAB module is composed of four Residual Feature Distillation Blocks (RFDBs) through residual aggregation. This structure can greatly improve the network's ability to extract features.

[0009] The residual aggregation takes the following form: residual learning is performed on consecutive RFDBs, and the features F1, F2, F3, and F4 output by the first RFDB, are stacked. The stacked features are then fused through a convolution and output.

[0010] 2) The three upsampling blocks are mainly composed of convolution and subpixel convolution. Each upsampling block can perform twice the image magnification and output the reconstructed image.

[0011] The first upsampling block consists of convolution, LeakyReLU, PixShuffle (2x magnification), and convolution in sequence; the second upsampling block consists of convolution, convolution, LeakyReLU, PixShuffle (2x magnification), and convolution in sequence; and the third upsampling block consists of convolution, convolution, LeakyReLU, PixShuffle (2x magnification), and convolution in sequence.

[0012] S3: Change the I in S1 LR16 The LFSRNet network fed into S2 passes through the RAB module and the first upsampling block, respectively, and forward inference yields a super-resolution image I magnified by two times. SR32 .

[0013] S4: Based on the following loss function, perform backpropagation on the network to complete the first stage of network parameter update and optimization.

[0014]

[0015] Where N represents the amount of data in a batch during model training.

[0016] S5: Change the I in S1 LR16 The LFSRNet network fed into S2 passes through the RAB module, the first upsampling block, and the second upsampling block, respectively, and forward inference yields a super-resolution image I magnified four times. SR64 .

[0017] S6: Based on the following loss function, perform backpropagation on the network to complete the second stage of network parameter update and optimization.

[0018]

[0019] Where N represents the amount of data in a batch during model training.

[0020] S7: Change the I in S1 LR16 The LFSRNet network fed into S2 passes through the RAB module, the first upsampling block, the second upsampling block, and the third upsampling block, respectively, and forward inference yields an eight-fold magnified super-resolution image I. SR128 .

[0021] S8: Change the I in S1 HR128 The data is fed into a face semantic segmentation network to obtain a high-resolution face semantic segmentation prediction map, PSeg. HR128 PSeg HR128 Data processing yields a high-resolution face semantic segmentation map, Seg. HR128。

[0022] The face semantic segmentation network is pre-trained and can perform face semantic segmentation on multiple parts of a face image, such as the left eye, right eye, nose, and lips.

[0023] Among them, the face semantic segmentation prediction map PSeg HR128 The shape and size are (128×128×C), where C represents the number of categories that the face semantic segmentation network can segment, PSeg HR128 Each pixel value represents the prediction confidence for each class at that location.

[0024] Furthermore, PSeg HR128 The class with the highest confidence in each pixel is used as the final segmentation result to generate a high-resolution face semantic segmentation map Seg. HR128 Seg HR128 It is a grayscale image with a pixel size of (128×128).

[0025] S9: Change the I in S7 SR128 The data is fed into the face semantic segmentation network in S8 to obtain the super-resolution face semantic segmentation prediction map PSeg. SR128 .

[0026] Among them PSeg SR128 The shape and size are (128×128×C), where C represents the number of categories that the face semantic segmentation network can segment.

[0027] S10: Based on the following loss function, perform backpropagation on the network to complete the third stage of network parameter update and optimization.

[0028]

[0029] Where N represents the amount of data in a batch during model training, C represents the number of categories that the face semantic segmentation network can segment, and α and β represent the weights of the two losses, respectively.

[0030] In this stage, a face semantic segmentation network is used to obtain prior face information, and the Segmentation Network is used to obtain the prior face information. HR128 With PSeg SR128 The cross-entropy loss is incorporated into the total loss function to update the network parameters, helping the network generate more realistic face structures.

[0031] Furthermore, face semantic segmentation networks have high requirements for the resolution of input images. In the last stage of training iteration, the super-resolution image generated has a pixel size of (128×128), which contains more semantic information and can meet the input requirements of face semantic segmentation networks.

[0032] S11: Iterate through S3 to S10 to train the model until it converges.

[0033] The ADAM gradient descent optimization algorithm was used to train the model with an initial learning rate of 0.001. The learning rate was adjusted using a linear step size learning rate adjustment strategy, and the model was trained for several epochs until it converged.

[0034] Furthermore, the trained convergent model can perform super-resolution image reconstruction of low-resolution (16×16 pixels) face images at times of 2x, 4x, and 8x.

[0035] The beneficial effects of this invention are:

[0036] 1. The network model in this invention is lightweight and efficient, and one model can achieve face super-resolution reconstruction at three different magnifications.

[0037] 2. This invention is applicable to face image input with extremely low resolution (e.g., 16×16 pixels) and can produce high-quality reconstructed images.

[0038] 3. This invention uses a face semantic segmentation network model to obtain prior information about faces and proposes a face segmentation loss to help the model reconstruct more realistic facial structures. Attached Figure Description

[0039] Figure 1 This is a structural diagram of the SRB (Shallow Residual Block) module in the method of this invention;

[0040] Figure 2 This is a structural diagram of the RFDB (Residual Feature Distillation Block) module in the method of this invention;

[0041] Figure 3 This is a structural diagram of the RAB (Residual Aggregation Block) module in the method of this invention;

[0042] Figure 4 This is a structural diagram of the lightweight face super-resolution network LFSRNet in the method of this invention;

[0043] Figure 5 This is a segmentation result diagram of the face semantic segmentation network in this embodiment;

[0044] Figure 6 This is a comparison image of LFSRNet (Lightweight Face Super-Resolution Network) and RFDN (RFDN) generated face images with 8x super-resolution resolution in this embodiment.

[0045] Figure 7 This is a comparison image of the super-resolution face images generated with and without face segmentation loss in this embodiment. Detailed Implementation

[0046] Specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the present invention. It should be noted that the embodiments of the present invention are not limited thereto.

[0047] like Figures 1 to 7 As shown in the figure, the face super-resolution reconstruction method based on progressive training and face semantic segmentation provided in this embodiment includes the following steps:

[0048] S1: Prepare training data. The img_align_celeba dataset contains 202,559 aligned face images. First, the dataset is cropped by centering each image to a size of (128×128×3) pixels, which is used as I. HR128 .

[0049] Furthermore, the dataset was randomly divided into a training set, a validation set, and a test set in a ratio of 8:1:1, with 162,559 images in the training set, 20,000 images in the validation set, and 20,000 images in the test set.

[0050] Furthermore, the I values ​​in the training set, validation set, and test set will be... HR128 Bicubic interpolation was used to downsample to (64×64×3) pixels as I. HR64 , will I HR64 Bicubic interpolation is used to downsample to (32×32×3) pixels as I. HR32 , will I HR32 Bicubic interpolation is used to downsample to (16×16×3) pixels as I. LR16 .

[0051] In this method, images from the training set are randomly flipped horizontally and input into the network for training.

[0052] S2: A lightweight face super-resolution network, LFSRNet, is proposed. The LFSRNet network consists of a residual aggregation block (RAB) and three upsampling blocks, such as... Figure 4 As shown, the LFSRNet network in this invention is an improvement based on the RFDN (Residual Feature Distillation Network) network:

[0053] 1) The Shallow Residual Block (SRB) proposed by RFDN, such as Figure 1 As shown, this module can perform residual learning without adding any additional parameters. In SRB, the convolution kernel is 3×3, the number of filters is 64, and the stride is 1.

[0054] 2) The Residual Feature Distillation Block (RFDB) proposed by RFDN is built from the SRB. The method of this invention removes the attention mechanism module in the original RFDB, such as... Figure 2 As shown:

[0055] Furthermore, the feature maps are first input in parallel into a convolutional block (1×1 kernel, 32 filters, stride of 1) and an SRB to generate D1 and R1.

[0056] Furthermore, R1 is input in parallel into a convolutional block (1×1 kernel, 32 filters, stride of 1) and an SRB to generate D2 and R2;

[0057] Furthermore, R2 is input in parallel into a convolutional block (1×1 kernel, 32 filters, stride of 1) and an SRB to generate D3 and R3;

[0058] Furthermore, R3 is input into a convolutional block (with a kernel size of 3×3, a filter size of 32, and a stride of 1) to generate D4;

[0059] Furthermore, D1, D2, D3, and D4 are stacked, and the stacked feature maps are output through a convolution block (1×1 kernel, 64 filters, and 1 stride).

[0060] 3) This invention improves the RFDN network by proposing a residual aggregation module (RAB). The RAB module is composed of four RFDB blocks aggregated through residual aggregation. This structure can greatly enhance the network's ability to extract features, such as... Figure 3 As shown: Residual learning is performed on consecutive RFDBs. The features F1, F2, F3, and F4 output by the first RFDB are stacked. The stacked feature maps are then fused and output through a convolutional block (1×1 kernel, 64 filters, stride of 1).

[0061] 4) The three upsampling blocks mainly consist of convolution and subpixel convolution. Each upsampling block can perform a 2x image magnification and output a reconstructed image, such as... Figure 4 As shown:

[0062] Furthermore, the first upsampling block consists of convolution (3×3 kernel, 256 filters, stride 1), LeakyReLU, PixShuffle (2x magnification), and convolution (1×1 kernel, 3 filters, stride 1).

[0063] Furthermore, the second upsampling block consists of convolution (3×3 kernel, 32 filters, stride 1), convolution (3×3 kernel, 128 filters, stride 1), LeakyReLU, PixShuffle (2x magnification), and convolution (1×1 kernel, 3 filters, stride 1).

[0064] Furthermore, the third upsampling block consists of convolution (3×3 kernel, 16 filters, stride 1), convolution (3×3 kernel, 64 filters, stride 1), LeakyReLU, PixShuffle (2x magnification), and convolution (1×1 kernel, 3 filters, stride 1).

[0065] S3: Change the I in S1 LR16 The LFSRNet network fed into S2 passes through RAB and the first upsampling block, respectively, and forward inference yields a super-resolution image I magnified by two times. SR32 .

[0066] S4: Based on the following loss function, perform backpropagation on the network to complete the first stage of network parameter update and optimization.

[0067]

[0068] Where N represents the amount of data in a batch during model training, and in this embodiment, N equals 256.

[0069] S5: Change the I in S1 LR16 The LFSRNet network fed into S2 passes through RAB, the first upsampling block, and the second upsampling block, respectively, and forward inference yields a super-resolution image I magnified four times. SR64 .

[0070] S6: Based on the following loss function, perform backpropagation on the network to complete the second stage of network parameter update and optimization.

[0071]

[0072] Where N represents the amount of data in a batch during model training, and in this embodiment, N equals 256.

[0073] S7: Change the I in S1 LR16The LFSRNet network fed into S2 passes through RAB, the first upsampling block, the second upsampling block, and the third upsampling block, respectively, and forward inference yields an eight-fold magnified super-resolution image I. SR128 .

[0074] S8: Change the I in S1 HR128 The data is fed into a face semantic segmentation network to obtain a high-resolution face semantic segmentation prediction map, PSeg. HR128 PSeg HR128 Data processing yields a high-resolution face semantic segmentation map, Seg. HR128 .

[0075] In this embodiment, the face semantic segmentation network used is BiSeNet. BiSeNet can perform semantic segmentation on 19 categories in a face image, including the left eye, right eye, lips, and background. The segmentation effect is as follows: Figure 5 As shown.

[0076] Among them, the face semantic segmentation prediction map PSeg HR128 The shape and size are (128×128×C), where C represents the number of categories that the face semantic segmentation network can segment. In this embodiment, C equals 19, PSeg HR128 Each pixel value represents the prediction confidence for each class at that location.

[0077] Furthermore, PSeg HR128 The class with the highest confidence in each pixel is used as the final segmentation result to generate a high-resolution face semantic segmentation map Seg. HR128 Seg HR128 It is a grayscale image with a pixel size of (128×128).

[0078] S9: Change the I in S7 SR128 The data is fed into the BiSeNet network in S8 to obtain the super-resolution face semantic segmentation prediction map PSeg. SR128 Among them, PSeg SR128 The shape and size are (128×128×C), where C represents the number of categories that the face semantic segmentation network can segment. In this embodiment, C equals 19.

[0079] S10: Based on the following loss function, perform backpropagation on the network to complete the final stage of network parameter update and optimization.

[0080]

[0081] Where N represents the amount of data in a batch during model training, C represents the number of categories that the face semantic segmentation network can segment, and α and β represent the weights of the two loss functions, respectively. In this embodiment, N equals 256, C equals 19, α equals 1, and β equals 1.

[0082] Furthermore, this stage uses a face semantic segmentation network to obtain prior face information, and then segments the face into segments. HR128 With PSeg SR128 The cross-entropy loss is incorporated into the total loss function as the face segmentation loss to update the network parameters, thereby helping the network generate more realistic face structures.

[0083] Furthermore, the face semantic segmentation network has a high resolution for the input image. The super-resolution image generated in the last stage has a pixel size of (128×128) and contains more semantic information, which can meet the input requirements of the face semantic segmentation network.

[0084] S11: Iterate through S3 to S10 to train the model until it converges.

[0085] The ADAM gradient descent optimization algorithm was used as the optimizer for model training. The initial learning rate was 0.001, and the learning rate was adjusted using a linear step size learning rate adjustment strategy. After each epoch of training, the learning rate was adjusted to 0.99 of the previous epoch. The batch size was 256, and the model was trained for 300 epochs.

[0086] Furthermore, the trained convergent model can perform super-resolution image reconstruction of low-resolution (16×16 pixels) face images at times of 2x, 4x, and 8x.

[0087] Furthermore, objective evaluation metrics are used to compare the performance of the method of this invention and the RFDN network on the test set in S1, and the I of the test set in S1 is used to... LR16 The super-resolution images were reconstructed by inputting LFSRNet and RFDN respectively, with 4x and 8x super-resolution. Peak noise ratio (PSNR) and structural similarity index (SSIM) were used as objective evaluation indicators. Table 1 compares the performance of the two methods at 4x super-resolution, and Table 2 compares the performance of the two methods at 8x super-resolution. The results of the two tables show that the method of the present invention has better super-resolution performance.

[0088] Table 1

[0089] PSNR SSIM RFDN 25.349 0.800 LFSRNet 27.614 0.870

[0090] Table 2

[0091] PSNR SSIM RFDN 23.794 0.676 LFSRNet 25.911 0.749

[0092] Furthermore, the performance of the method of the present invention and RFDN on the test set in S1 was compared using human visual perception quality, and the I of the test set in S1 was... LR16 8x super-resolution image reconstruction was performed by inputting data into LFSRNet and RFDN respectively. Figure 6 The comparison between the method of the present invention and the face super-resolution reconstructed image generated by RFDN in the test set is shown, and it can be seen that the image reconstructed by the method of the present invention is higher resolution.

[0093] Furthermore, the method of this invention verifies the gain brought by adding face segmentation loss to the model training. The test set in S1 is used to test the effect of the model trained with face segmentation loss added, such as... Figure 7 As shown, after incorporating face segmentation loss, the model achieves better results in restoring facial structures such as eyes, lips, and teeth in face images. This confirms that face segmentation loss can help the network reconstruct more realistic facial structures.

[0094] Furthermore, the method of this invention verifies the gains brought to the model by the progressive training method. The test set in S1 is used to test the super-resolution effect of the model using progressive training. Peak noise ratio (PSNR) and structural similarity (SSIM) are used as objective evaluation indicators. Table 3 shows the performance comparison of the model with and without progressive training on a 4x super-resolution scale, and Table 4 shows the performance comparison of the model with and without progressive training on an 8x super-resolution scale. The results of the two tables show that the method of this invention has good generalization ability, and progressive training can improve super-resolution performance.

[0095] Table 3

[0096] PSNR SSIM Without using progressive training 27.598 0.870 Use progressive training 27.614 0.870

[0097] Table 4

[0098] PSNR SSIM Without using progressive training 25.833 0.749 Use progressive training 25.911 0.749

[0099] The above embodiments are relatively ideal implementations of the present invention. However, the method of the present invention is not limited to the specific implementation scope. Other changes, substitutions, combinations, etc., made without departing from the principle and essence of the present invention should be equivalent substitutions and should be included within the protection scope of the present invention.

Claims

1. A face super-resolution reconstruction method based on progressive training and face semantic segmentation, characterized in that, Includes the following steps: S1: Prepare training data, including low-resolution 16×16 pixel face images. LR16 High-resolution face image 32×32 pixels HR32 64×64 pixels HR64 128×128 pixels HR128 ; S2: Propose a lightweight face super-resolution network LFSRNet. The LFSRNet network consists of a residual aggregation module RAB and three upsampling blocks. The three upsampling blocks are composed of convolution and subpixel convolution. Each upsampling block can perform image magnification by 2 times and output the reconstructed image. S3: Change the I in S1 LR16 The LFSRNet network fed into S2 passes through the RAB module and the first upsampling block, respectively, and forward inference yields a super-resolution image I magnified by two times. SR32 ; S4: Based on the following loss function, perform backpropagation on the network to complete the first stage of network parameter update and optimization; Where N represents the amount of data in a batch during model training; S5: Change the I in S1 LR16 The LFSRNet network fed into S2 passes through the RAB module, the first upsampling block, and the second upsampling block, respectively, and forward inference yields a super-resolution image I magnified four times. SR64 ; S6: Based on the following loss function, perform backpropagation on the network to complete the second stage of network parameter update and optimization; S7: Change the I in S1 LR16 The LFSRNet network fed into S2 passes through the RAB module, the first upsampling block, the second upsampling block, and the third upsampling block, respectively, and forward inference yields an eight-fold magnified super-resolution image I. SR128 ; S8: Change the I in S1 HR128 The data is fed into a face semantic segmentation network to obtain a high-resolution face semantic segmentation prediction map, PSeg. HR128 PSeg HR128 Data processing yields a high-resolution face semantic segmentation map, Seg. HR128 ; S9: Change the I in S7 SR128 The data is fed into the face semantic segmentation network in S8 to obtain the super-resolution face semantic segmentation prediction map PSeg. SR128 ; S10: Based on the following loss function, perform backpropagation on the network to complete the third stage of network parameter update and optimization; Where C represents the number of categories that the face semantic segmentation network can segment, and α and β represent the weights of the two losses, respectively; S11: Iterate through S3 to S10 to train the model until it converges.

2. The face super-resolution reconstruction method based on progressive training and face semantic segmentation according to claim 1, characterized in that, The face super-resolution network LFSRNet in step S2 is an improvement based on RFDN. The LFSRNet network model is a lightweight model that is suitable for ultra-low resolution face image input and can output high-resolution reconstructed images with three different magnifications, including the residual aggregation module RAB and three upsampling blocks.

3. The face super-resolution reconstruction method based on progressive training and face semantic segmentation according to claim 2, characterized in that, The residual aggregation module RAB is constructed by four consecutive residual feature distillation modules RFDB through residual aggregation: the RFDBs connected one after the other perform residual learning, the output features of the four RFDBs are stacked, and finally the stacked features are fused by a convolution and output.

4. The face super-resolution reconstruction method based on progressive training and face semantic segmentation according to claim 2, characterized in that, Each upsampling block consists of convolutions and subpixel convolutions. Each upsampling block can perform a 2x image upscaling and output a reconstructed image, including: 1) The first upsampling block consists of convolution, LeakyReLU, PixShuffle (2x magnification), and convolution in sequence; 2) The second upsampling block consists of convolution, convolution, LeakyReLU, PixShuffle 2x magnification, and convolution in sequence; 3) The third upsampling block consists of convolution, convolution, LeakyReLU, PixShuffle 2x amplification, and convolution in sequence.

5. The face super-resolution reconstruction method based on progressive training and face semantic segmentation according to claim 1, characterized in that, Steps S3 to S10 constitute a progressive training method for face super-resolution networks. This training method divides the model into three stages in a single iteration of training, including: 1) Steps S3 to S4 are the first stage in an iterative training, in which the model is trained for a double super-resolution task. 2) Steps S5 to S6 are the second stage in an iterative training, in which the model is trained for a four-fold super-resolution task. 3) Steps S7 to S10 are the third stage in an iterative training, in which the model is trained for an eight-fold super-resolution task.

6. The face super-resolution reconstruction method based on progressive training and face semantic segmentation according to claim 1, characterized in that, Steps S8 to S10 use a face semantic segmentation network model to obtain prior information about the face and propose a face segmentation loss. In the third stage of an iterative training, the face segmentation loss is incorporated into the total loss function to update the network parameters, thereby assisting the network in generating more realistic face structures.

Images