A deep learning-based image super-resolution reconstruction method

Abstract

The application discloses a kind of based on deep learning's image super-resolution reconstruction method, its method includes the steps as follows: first, obtain image sample, it is handled to construct training set;Second, the construction of image super-resolution network model, third, using the generator network in the above-mentioned obtained SRGAN model, the image to be super-resolution reconstruction is handled to realize super-resolution reconstruction.There is beneficial effect: it is beneficial to the detail generation when image super-resolution reconstruction is generated.Improve the depth of calculation, and more can be reserved to shallow feature, so that the completion effect of image super-resolution reconstruction this shallow task is better.Batch normalization layer is removed to the blurring influence of feature, further provide the expression ability of model.Improve the quality of image super-resolution reconstruction, and better generate high-resolution image.

Description

Technical Field

[0001] This invention relates to an image super-resolution reconstruction method, and more particularly to an image super-resolution reconstruction method based on deep learning. Background Technology

[0002] Currently, deep learning technology has made significant progress in the field of computer vision. Super-resolution refers to the process of reconstructing a low-resolution image or video into a high-resolution image or video. Generative Adversarial Networks (GANs), as a deep learning model, are widely used in image generation. Super-resolution GANs introduce GAN design into image super-resolution reconstruction tasks, dividing the image super-resolution reconstruction network into a discriminator and a generator. The generator network is responsible for generating high-resolution images from the input low-resolution image. The discriminator network is trained adversarially against the generator. The high-resolution image generated by the generator and the ground truth high-resolution image (the original image) are input into the discriminator, which performs a binary classification judgment on the input set of images. Through the adversarial training of the two networks, the generation ability of the generator and the discrimination ability of the discriminator are continuously improved, making the images generated by the generator increasingly closer to real high-resolution images. At the same time, the discriminator's ability to correctly distinguish whether an image is a ground truth image or an image generated by the network generator is further improved, achieving a level of capability that increasingly approaches generating real high-resolution images. However, the feature extraction capability of this method still has room for improvement. The original model does not incorporate an attention mechanism, failing to fully utilize the attention information of features in image super-resolution. Meanwhile, the reconstruction work of the generator when generating high-resolution images is all performed at the same scale, missing feature information at different levels across multiple scales. Furthermore, the original model's loss function uses a VGG network for feature extraction, which lacks sufficient detail. Introducing a new loss function can help improve the model's super-resolution reconstruction performance. Improvements are made to address these shortcomings of the SRGAN network: First, feature information from different levels is fused and extracted; second, an attention mechanism is introduced to allow the deep learning model to better perform image super-resolution reconstruction tasks; third, the loss function is improved by replacing the VGG network with a ResNet50 network for better feature extraction, and a high-frequency loss is introduced to enhance image reconstruction performance. Summary of the Invention

[0003] The purpose of this invention is to address the problems of existing generative adversarial networks (GANs) where the original model does not incorporate an attention mechanism, thus failing to fully utilize the attention information of features in image super-resolution. Furthermore, the reconstruction work of the generator when generating high-resolution images is performed entirely at the same scale, resulting in the loss of feature information at different levels and scales. Therefore, this invention provides a deep learning-based image super-resolution reconstruction method.

[0004] The deep learning-based image super-resolution reconstruction method provided by this invention includes the following steps:

[0005] The first step is to obtain image samples and process them to construct a training set;

[0006] The second step is to construct the image super-resolution network model, and the specific steps are as follows:

[0007] The image super-resolution model consists of a generator network and a discriminator network, and the construction steps are as follows:

[0008] Step 1: Construct the generator network, which consists of convolutional layers, eight residual blocks, a multi-scale high-frequency ResUNet module, and an upsampling module. The specific steps are as follows:

[0009] (1) Construct a convolutional layer module to extract feature information of the input image. After the 3-channel input RGB image is passed through a convolutional layer with 64 convolutional kernels, a feature map with 64 channels is generated.

[0010] (2) Construct residual blocks, each residual block has two convolutional layers, the first convolutional layer is followed by a PRelu layer;

[0011] (3) Construct an upsampling module, which consists of sub-pixel convolutional layers and PReLU layers;

[0012] (4) Construct a high-frequency residual attention module. This module includes an average pooling layer, a residual connection block, and Sigmoid normalization. Input X, subtract the result obtained by the global average pooling layer from X, and then calculate it through the residual connection block and perform Sigmoid normalization to obtain the result attention layer. Output the dot product result of the original input X and the attention. This module can implicitly obtain the global attention information of the feature by adjusting the attention weight through deep learning, which is beneficial to the effect of super-resolution reconstruction.

[0013] (5) Construct a multi-scale high-frequency ResUNet module. This module consists of 21 small modules and is divided into a three-level cascaded structure. Each cascaded structure consists of three residual connection blocks, one max pooling downsampling layer, two high-frequency residual attention modules and one upsampling layer. The input is set as X. Starting from the third layer, X is first calculated by the residual connection block of the third layer. Then X is simultaneously input into another sequence. In this sequence, X is first downsampled to half the size by the max pooling downsampling layer. After passing through a residual connection block, the first high-frequency residual attention calculation of this layer is performed. The result is used as input and recursively input into the next level of multi-scale attention calculation. The result is an intermediate image of the same size after recursion of multi-scale attention. The output of the recursive result after calculation of this layer continues to enter the residual connection block and upsampling module of this level of calculation. It is upsampled to the size of the original input and then the second high-frequency residual attention calculation is performed. After the second high-frequency residual attention calculation of this level, the result is added to the output of the first branch and the result of this level is output.

[0014] (6) The generator network is constructed by combining the convolutional layer module, residual connection block, upsampling module and multi-scale high-frequency ResUNet module. First, the input image dimension is increased to 64 using the convolutional layer module. Then, it passes through three residual connection blocks and enters the multi-scale high-frequency ResUNet module. By extracting multi-scale attention, the model can implicitly extract relevant information of each scale, so that the model can weight attention on the part that is more relevant to the part to be super-resolution reconstructed. At the same time, the use of multi-scale attention also makes up for the model's ability to reconstruct high-frequency details. After the attention extraction and weighting calculation of multi-scale features, the result is calculated by three residual connection blocks and then upsampling is performed. Upsampling is performed through sub-pixel convolutional layers to reduce network parameters and speed up reconstruction. In addition, in order not to affect the reconstruction quality, the batch normalization operation used in the original SRGAN is abandoned in the generator network model.

[0015] (7) Construct a loss function. First, use perceptual loss in the loss function and select ResNet50 for feature extraction. Replace the ReLU activation function with the PReLU activation function and remove the last classification layer. Then, introduce high-frequency loss to obtain high-frequency information in the image reconstruction process. By using mean filtering, obtain low-frequency information of the image. Perform difference operation between the image information and the low-frequency information to obtain high-frequency information of the image. Calculate the MSE loss of the high-frequency part of the original image and the super-reconstructed image to obtain the high-frequency loss.

[0016] Step 2: Construct the discriminator network. The network is built in the following order: convolutional layer, Leaky ReLU layer, and BatchNorm layer. These are grouped together and consist of seven combinations within the discriminator. The dimension is increased from 3 to 64, and then gradually increased to 512. Finally, the classification section consists of an adaptive average pooling layer and two convolutional layers. The convolutional layers increase the 512-dimensional features to 1024 dimensions, and then reduce them to 1 dimension to obtain the binary classification result. Here, the cross-entropy loss of the binary classifier is used.

[0017] Step 3: Initialize the two network models and use their respective loss functions to obtain the SRGAN model through alternating iterative training;

[0018] The third step is to use the generator network in the SRGAN model obtained above to process the image to be super-resolution reconstructed in order to achieve super-resolution reconstruction.

[0019] The beneficial effects of this invention are:

[0020] This invention provides a deep learning-based image super-resolution reconstruction method. It trains an improved SRGAN model on a publicly available dataset to obtain an SRGANv2 model for image super-resolution reconstruction. The SRGANv2 model includes a high-frequency residual attention module, residual connection blocks, and adversarial neural network training methods. This method introduces a multi-scale high-frequency ResUNet module, enabling the model to focus on multi-scale feature information during image super-resolution reconstruction, which is highly beneficial for detail generation. In calculating the perceptual loss function, the feature extractor is changed from VGG19 to ResNet50, increasing computational depth while preserving shallow features, resulting in better performance in this shallow task. A high-frequency loss is also introduced into the loss function, allowing the model to focus on high-frequency information in the image during super-resolution reconstruction, preserving image texture and style. All batch normalization layers are removed from the generator, eliminating their blurring effect on features and further enhancing the model's expressive power. This invention can improve the quality of image super-resolution reconstruction and generate higher-resolution images more effectively. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the image super-resolution reconstruction method described in this invention.

[0022] Figure 2 This is a schematic diagram of the process of training the model for reconstructing the dataset according to the present invention.

[0023] Figure 3 This is a schematic diagram of the residual connection block structure described in this invention.

[0024] Figure 4 This is a schematic diagram of the high-frequency residual attention module described in this invention.

[0025] Figure 5 This is a schematic diagram of the multi-scale high-frequency ResUNet module described in this invention.

[0026] Figure 6 This is a schematic diagram of the generator network module described in this invention.

[0027] Figure 7 This is a schematic diagram of the discriminator network structure described in this invention. Detailed Implementation

[0028] Please see Figures 1 to 7 As shown:

[0029] Requirements: A standard PC running Ubuntu 20.04.1, with an NVIDIA GeForce RTX 3090 graphics card. PyTorch 1.7.1 is the deep learning framework.

[0030] The image super-resolution reconstruction method based on deep learning provided by this invention includes the following steps:

[0031] The following will describe, with reference to the accompanying drawings in the embodiments of the present invention, a complete process for improving the SRGAN model using a publicly available image super-resolution reconstruction dataset to obtain a prediction model, such as... Figure 1 and Figure 2 As shown, the model's network structure consists of a generator network and a discriminator network. The network construction is described in step two.

[0032] The first step is to acquire image samples, scale them to a factor inverse of the desired magnification, and then perform a series of image enhancement operations, such as rotation and translation, to build a training set.

[0033] The second step is the construction of the image super-resolution network, and the specific process is as follows:

[0034] Step 1: Construct the generator network, which consists of convolutional layers, eight residual blocks, a multi-scale high-frequency ResUNet module, and an upsampling module. The specific steps are as follows:

[0035] (1) Construct a convolutional layer module to extract feature information from the input image. After passing the 3-channel input RGB image through a convolutional layer with 64 convolutional kernels, a feature map with 64 channels is generated.

[0036] (2) Construct 8 residual blocks. Each residual block consists of two convolutional layers and one PReLU layer. The input to the residual block is passed through a convolutional layer followed by a PReLU layer, and then through another convolutional layer to obtain the output of that block. This output is then residually concatenated with the input to obtain the final output of the residual block. This module is as follows: Figure 3 As shown.

[0037] (3) Construct an upsampling module, which consists of subpixel convolutional layers and PReLU layers.

[0038] (4) Constructing a high-frequency residual attention module. This module includes an average pooling layer, a residual connective block, and sigmoid normalization. For the input X, X is subtracted from the result obtained by the global average pooling layer, then processed by the residual connective block and normalized using sigmoid normalization to obtain the resulting attention layer. The dot product of the original input X and the attention layer is then output. This module implicitly utilizes deep learning to adjust the attention weights to obtain global attention information for the features, which is beneficial for the super-resolution reconstruction effect. This module is as follows... Figure 4 As shown.

[0039] (5) Constructing a multi-scale high-frequency ResUNet module. This module consists of 21 sub-modules in a three-level cascaded structure. Each cascaded layer comprises three residual connection blocks, one max-pooling downsampling layer, two high-frequency residual attention modules, and one upsampling layer. Assuming the input is X, starting from the third layer, X is first processed through the residual connection block of the third layer. Then, X is simultaneously input into another sequence. In this sequence, X is first downsampled to half its original size by the max-pooling downsampling layer, then passes through a residual connection block, and undergoes the first high-frequency residual attention calculation of this layer. The result is used as input and recursively fed into the next level of multi-scale attention calculation. The result is an intermediate image of the same size obtained through recursive multi-scale attention. The output of the recursive result, after being processed by this layer, continues to be fed into the residual connection block and upsampling module of this level, upsampled to the original input size, and then undergoes the second high-frequency residual attention calculation. After the second high-frequency residual attention calculation of this level, the result is added to the output of the first branch and the result of this level is output. For example... Figure 5 As shown, the High Frequency Residual Attention module is abbreviated as HFRA attention.

[0040] (6) The generator network is constructed by combining the convolutional layer module, residual connection block, upsampling module, and multi-scale high-frequency ResUNet module described above. First, the input image dimension is increased to 64 using the convolutional layer module. Then, it passes through three residual connection blocks and enters the multi-scale high-frequency ResUNet module. By extracting multi-scale attention, the model implicitly extracts relevant information at each scale, allowing it to weight attention towards parts more relevant to the super-resolution reconstruction. The use of multi-scale attention also compensates for the model's inability to reconstruct high-frequency details. After attention extraction and weighted calculation of multi-scale features, the result is processed through three residual connection blocks and then upsampled using sub-pixel convolutional layers, reducing network parameters and accelerating reconstruction. Furthermore, to avoid affecting reconstruction quality, the batch normalization operation used in the original SRGAN is discarded in the generator network model. The generator network is as follows: Figure 6 As shown.

[0041] (7) Constructing the loss function. The generator's loss function includes content loss and perceptual loss. Content loss is the calculation of the difference between the generated image and the ground truth, using the MSE loss function. Perceptual loss is achieved by feeding the image set generated by the generator and the image set of the ground truth image into the feature extraction network. The feature extraction network in this invention uses the ResNet50 network, and its structure is shown in the table below:

[0042]

[0043] The ResNet architecture can significantly accelerate neural network training and improve model accuracy. Furthermore, ResNet has excellent generalization capabilities; therefore, this invention uses the ResNet50 network, removing the final classification layer as the feature extraction network and introducing residual connections to enhance its feature extraction capabilities. When an image is input into the ResNet50, its corresponding feature dimension output is obtained. The loss is calculated by comparing the feature output of the image generated by the generator with the feature output generated by the ground truth network, thus obtaining the feature dimension loss, which helps the generator network train better. To prevent the loss of high-frequency information during super-resolution reconstruction, this invention also introduces a high-frequency loss. Low-frequency information is obtained by using mean filtering, and the difference between the image information and the low-frequency information is calculated to obtain the high-frequency information. The high-frequency loss is obtained by calculating the MSE loss of the high-frequency components of the original image and the super-reconstructed image.

[0044] Step 2: Discriminator Network Construction: The discriminator network is a binary classification network, so building it only requires 2D convolutional layers, batch normalization layers, and non-linear layers. The network is constructed in the following order: convolutional layers, LeakyReLU layers, and BatchNorm layers. This is a combination, consisting of seven combinations within the discriminator, increasing the dimension from 3 to 64 and then gradually increasing it to 512 dimensions. Finally, the classification section consists of an adaptive average pooling layer and two convolutional layers, increasing the 512-dimensional features to 1024 dimensions and then reducing them to 1 dimension to obtain the binary classification result. The discriminator network is as follows: Figure 7 As shown.

[0045] Step 3: Initialize the two network models, determine the number of images and training epochs for each batch, and use their respective loss functions to iteratively train the SRGAN model. The specific steps are as follows:

[0046] (1) Input the processed training set images into the generator network in batches. After the generator network calculates, it outputs the target image to be magnified by a factor of 1. At this time, the output image is compared with the image before the initial processing, i.e. the true image. The loss is calculated, and the result is backpropagated once to update the parameters in the network so that the generator network can generate an output that is closer to the true image.

[0047] (2) After backpropagation of the generator, adversarial training is performed. The generator output and random ground truth images are randomly combined to form a batch of data, which is then input into the discriminator network. After the discriminator calculates, a binary classification result is obtained. That is, the discriminator judges whether each image is a ground truth image. If it is a ground truth image, its label is 1. If it is an image generated by the generator, its label is 0. After the discriminator network calculates, it will output a set of data, which correspond to the probability that each image is a ground truth image. The loss is calculated by combining this set of data with the label. After obtaining the loss result, the discriminator network is backpropagated to update the parameters of the discriminator network, so that the discriminator network can correctly judge whether an image is judged by the generator or a ground truth image. In this way, it can better judge whether the image is generated by the generator.

[0048] (3) Train the generator network and the discriminator network alternately until the network converges, and save the final trained model parameters for use in subsequent super-resolution reconstruction of the target image.

[0049] The third step involves using the trained model parameters to input the low-resolution image to be super-resolved into the constructed SRGAN network, importing the parameters into it, performing depth calculations, and finally obtaining the high-resolution image after super-resolved reconstruction.

Claims

1. A deep learning-based image super-resolution reconstruction method, characterized in that: The method includes the following steps: The first step is to obtain image samples and process them to construct a training set; The second step is to construct the image super-resolution network model, and the specific steps are as follows: The image super-resolution model consists of a generator network and a discriminator network, and the construction steps are as follows: Step 1: Construct the generator network, which consists of convolutional layers, eight residual blocks, a multi-scale high-frequency ResUNet module, and an upsampling module. The specific steps are as follows: (1) Construct a convolutional layer module to extract feature information of the input image. After the 3-channel input RGB image is passed through a convolutional layer with 64 convolutional kernels, a feature map with 64 channels is generated. (2) Construct residual blocks, each residual block has two convolutional layers, the first convolutional layer is followed by a PRelu layer; (3) Construct an upsampling module, which consists of sub-pixel convolutional layers and PReLU layers; (4) Construct a high-frequency residual attention module. This module includes an average pooling layer, a residual connection block, and Sigmoid normalization. Input X, subtract the result obtained by the global average pooling layer from X, and then calculate it through the residual connection block and perform Sigmoid normalization to obtain the result attention layer. Output the dot product result of the original input X and the attention. This module can implicitly obtain the global attention information of the feature by adjusting the attention weight through deep learning, which is beneficial to the effect of super-resolution reconstruction. (5) Construct a multi-scale high-frequency ResUNet module. This module consists of 21 small modules and is divided into a three-level cascaded structure. Each cascaded structure consists of three residual connection blocks, one max pooling downsampling layer, two high-frequency residual attention modules and one upsampling layer. The input is set as X. Starting from the third layer, X is first calculated by the residual connection block of the third layer. Then X is simultaneously input into another sequence. In this sequence, X is first downsampled to half the size by the max pooling downsampling layer. After passing through a residual connection block, the first high-frequency residual attention calculation of this layer is performed. The result is used as input and recursively input into the next level of multi-scale attention calculation. The result is an intermediate image of the same size after recursion of multi-scale attention. The output of the recursive result after calculation of this layer continues to enter the residual connection block and upsampling module of this level of calculation. It is upsampled to the size of the original input and then the second high-frequency residual attention calculation is performed. After the second high-frequency residual attention calculation of this level, the result is added to the output of the first branch and the result of this level is output. (6) The generator network is constructed by combining the convolutional layer module, residual connection block, upsampling module and multi-scale high-frequency ResUNet module. First, the input image dimension is increased to 64 using the convolutional layer module. Then, it passes through three residual connection blocks and enters the multi-scale high-frequency ResUNet module. By extracting multi-scale attention, the model can implicitly extract relevant information of each scale, so that the model can weight attention on the part that is more relevant to the part to be super-resolution reconstructed. At the same time, the use of multi-scale attention also makes up for the model's ability to reconstruct high-frequency details. After the attention extraction and weighting calculation of multi-scale features, the result is calculated by three residual connection blocks and then upsampling is performed. Upsampling is performed through sub-pixel convolutional layers to reduce network parameters and speed up reconstruction. In addition, in order not to affect the reconstruction quality, the batch normalization operation used in the original SRGAN is abandoned in the generator network model. (7) Construct a loss function. First, use perceptual loss in the loss function and select ResNet50 for feature extraction. Replace the ReLU activation function with the PReLU activation function and remove the last classification layer. Then, introduce high-frequency loss to obtain high-frequency information in the image reconstruction process. By using mean filtering, obtain low-frequency information of the image. Perform difference operation between the image information and the low-frequency information to obtain high-frequency information of the image. Calculate the MSE loss of the high-frequency part of the original image and the super-reconstructed image to obtain the high-frequency loss. Step 2: Construct the discriminator network. The network is built in the following order: convolutional layer, Leaky ReLU layer, and Batch Norm layer. These are grouped together and consist of seven combinations within the discriminator. The dimension is increased from 3 to 64, and then gradually increased to 512. Finally, the classification section consists of an adaptive average pooling layer and two convolutional layers. The convolutional layers increase the 512-dimensional features to 1024 dimensions, and then reduce them to 1 dimension to obtain the binary classification result. Here, the cross-entropy loss of the binary classifier is used. Step 3: Initialize the two network models and use their respective loss functions to obtain the SRGAN model through alternating iterative training; The third step is to use the generator network in the SRGAN model obtained above to process the image to be super-resolution reconstructed in order to achieve super-resolution reconstruction.

2. The image super-resolution reconstruction method based on deep learning according to claim 1, characterized in that: The specific steps for obtaining the SRGAN model through alternating iterative training in step 3 of the second step are as follows: (1) Input the processed training set images into the generator network in batches. After the generator network calculates, it outputs the target image to be magnified by a factor of 1. At this time, the output image is compared with the image before the initial processing, i.e. the true image. The loss is calculated, and the result is backpropagated once to update the parameters in the network so that the generator network can generate an output that is closer to the true image. (2) After backpropagation of the generator, adversarial training is performed. The generator output and random ground truth images are randomly combined to form a batch of data, which is then input into the discriminator network. After the discriminator calculates, a binary classification result is obtained. That is, the discriminator judges whether each image is a ground truth image. If it is a ground truth image, its label is 1. If it is an image generated by the generator, its label is 0. After the discriminator network calculates, it will output a set of data, which correspond to the probability that each image is a ground truth image. The loss is calculated by combining this set of data with the label. After obtaining the loss result, the discriminator network is backpropagated to update the parameters of the discriminator network, so that the discriminator network can correctly judge whether an image is judged by the generator or a ground truth image. In this way, it can better judge whether the image is generated by the generator. (3) Train the generator network and the discriminator network alternately until the network converges, and save the final trained model parameters for use in subsequent super-resolution reconstruction of the target image.

Images