A Generative Adversarial Network Method for Image Super-resolution Reconstruction

Abstract

The invention discloses a generative adversarial network method for image super-resolution reconstruction, comprising: adopting a residual connection module and a dense connection module to construct a residual dense connection module for connecting feature information of different layers of an image; establishing a feature weight allocation attention mechanism for allocating different weights according to the importance of different features; obtaining a mixed loss function by combining a content loss function, a perception loss function and an adversarial loss function; establishing a generative adversarial network based on residual dense connection through the residual dense connection module, the feature weight allocation attention mechanism and the mixed loss function; training the generative adversarial network based on the residual dense module according to an image training set; inputting an image to be reconstructed with resolution into the trained generative adversarial network based on residual dense connection to obtain a reconstructed super-resolution image, thereby improving the fusion of multi-layer information, suppressing background noise, preventing gradient vanishing and gradient explosion problems, and enhancing the clarity of the image.

Description

Technical Field

[0001] The present invention relates to the technical field of image super-resolution reconstruction, and in particular to a generative adversarial network method applied to image super-resolution reconstruction. Background Art

[0002] Image super-resolution methods are mainly divided into the following categories. These image super-resolution methods have their own advantages, but also have their own disadvantages; Image super-resolution methods based on convolution operations. This method first extracts the features of low-resolution images through convolution layers, then maps the features to high-resolution image dimensions through nonlinear mapping, and finally maps the feature maps back to high-resolution images through reconstruction layers; however, this method is heavily dependent on the context content of the image area. Image super-resolution methods based on dense residual connection networks. This method aims to enhance the performance of the model through dense connections and residual connections. This connection method helps to alleviate the over-smoothing problem that may occur when the network depth increases, improves the training efficiency of the model on deep network structures and the ability to capture complex structures; however, this method leads to an increase in network parameters and increases computing resource consumption. Image super-resolution methods based on multi-scale fusion. This method will focus on extracting features at different scales, and capture the features of images or data at different scales by using convolution kernels of different scales or adopting multi-scale pooling modules; however, this method is sensitive to image noise and requires a large number of high-resolution images. An image super-resolution method based on generative adversarial networks is an unsupervised learning method. Through the binary minimax game method between the generator and the discriminator, the generative model can estimate the super-resolution image of the sample data distribution. This method can effectively solve the problem of insufficient high-resolution images, but this method has the problems of feature gradient disappearance and gradient explosion. Summary of the invention

[0003] Purpose of the invention: In order to overcome the shortcomings of the prior art, the present invention provides a generative adversarial network method for image super-resolution reconstruction, constructs a residual dense connection module, a feature weight allocation attention mechanism and a hybrid loss function to establish a generative adversarial network based on residual dense connection, and reconstructs the initial image for image super-resolution; improves the fusion of multi-layer information, suppresses background noise, and prevents gradient vanishing and gradient explosion problems.

[0004] Technical solution: To achieve the above-mentioned purpose, a generative adversarial network method for image super-resolution reconstruction of the present invention comprises the following steps:

[0005] S1. A residual connection module and a dense connection module are used to construct a residual dense connection module that connects feature information of different layers of the image;

[0006] S2. Establish a feature weight allocation attention mechanism that assigns different weights based on the importance of different features;

[0007] S3, combining the content loss function, the perceptual loss function and the adversarial loss function to obtain a hybrid loss function;

[0008] S4. Establish a generative adversarial network based on residual dense connection through residual dense connection module, feature weight allocation attention mechanism and hybrid loss function;

[0009] S5. Training a generative adversarial network based on a residual dense module according to a training set in the image data set to obtain a trained generative adversarial network based on a residual dense connection;

[0010] S6. Input the image to be reconstructed into the trained residual densely connected generative adversarial network to obtain a reconstructed super-resolution image.

[0011] Furthermore, in step S1, the residual connection module connects the input image feature x i-1 And the result after input image feature mapping F(x i-1 ) is accumulated to obtain the residual mapping output F Res (x i ), the calculation process is as follows:

[0012] F Res (x i )=F(x i-1 )+x i-1

[0013] The densely connected module converts x i All previous image features and x i-1 The accumulation of residual mapping outputs gives the dense mapping output F DB (x i ), the calculation process is as follows:

[0014]

[0015] The residual dense connection module is constructed by the residual connection module and the dense residual module. The calculation process is as follows:

[0016]

[0017] Where β is the residual scaling parameter, x 0 is the initial variable.

[0018] Furthermore, in step S2, the feature weight allocation attention mechanism inputs the initial features P and G through convolution operations and adds the corresponding position features, and processes them through ReLU functions and Sigmoid functions to obtain the processed results, and multiplies the processed results with the initial features P and G channel by channel to obtain the weighted features P after the weights are allocated. l , the calculation process is as follows:

[0019]

[0020] In the formula, σ 1 is the ReLU function, W P T , W g T , and ψ T denote the unit transposed vector, and b g and b Ψ Both represent offsets, and b Ψ ∈R; R is a real number, σ 2 is the Sigmoid activation function, Θ att Contains linear transformation variable W p , W g , ψ and offset b g , b Ψ .

[0021] Furthermore, in step S3, the mixed loss function L SR The calculation process is as follows:

[0022] L SR (I lr , I hr , I sr )=L MSE (I hr , I sr )+6×10 -3 ×L prec (I hr , I sr )+1×10 -3 ×L WGANGP (I lr , I hr )

[0023] Where, L MSE is the content loss function, L prec is the perceptual loss function, L WGANGP is the adversarial loss function; I lr is the low-resolution image of the I image, I hris the high-resolution image of the I image, I sr is the super-resolution image of the I image, and the image size is H×W.

[0024] Furthermore, the content loss function L in the hybrid loss function MSE The calculation process is as follows:

[0025]

[0026] Where, L MSE is the loss function of the mean square error, which is the square value of the pixel difference at the corresponding position of the calculated image.

[0027] Furthermore, the perception loss function L in the hybrid loss function prec The calculation process is as follows:

[0028]

[0029] Where V x,y represents the feature map obtained after the yth convolution activation before the xth maximum pooling layer in the VGG19 network; W x,y and H x,y represents the H and W dimensions of the feature map in the VGG network.

[0030] Furthermore, the adversarial loss function in the hybrid loss function adopts the WGAN loss function with gradient penalty, and the calculation process is as follows:

[0031]

[0032] In the formula, E represents the mean, G represents the operation of the generator, D represents the operation of the discriminator, || || p represents the p-criterion, Indicates downsampling of I picture.

[0033] Beneficial effects: The generative adversarial network method for image super-resolution reconstruction of the present invention realizes the transmission of feature information of different layers through the residual dense connection module, prevents the deepening of the network structure and the problem of gradient attenuation, and then suppresses the interference of noise information through the feature weight allocation attention mechanism, thereby improving the signal-to-noise ratio of the super-resolution image; finally, the content loss, perceptual loss and adversarial loss are combined into a new hybrid loss function to prevent gradient vanishing and gradient explosion, thereby improving the fusion of multiple information. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flowchart of the generative adversarial network based on residual dense connection;

[0035] Figure 2 To generate a diagram of the adversarial network structure;

[0036] Figure 3 It is the residual network structure diagram;

[0037] Figure 4 This is a densely connected network structure diagram;

[0038] Figure 5 This is the residual dense connection network structure diagram;

[0039] Figure 6 Assigning attention mechanism structure diagram to feature weights;

[0040] Figure 7 This is a diagram of the structure of a generative adversarial network based on residual dense connections;

[0041] Figure 8 Reconstruct contrast images for various super-resolution methods;

[0042] Fig. 9 This is a comparison diagram of the ablation experiment results of the generative adversarial network based on residual dense connection. DETAILED DESCRIPTION

[0043] The present invention will be further described below in conjunction with the accompanying drawings.

[0044] like Figure 1 As shown, a generative adversarial network method applied to image super-resolution reconstruction is characterized by comprising the following steps:

[0045] S1. A residual connection module and a dense connection module are used to construct a residual dense connection module that connects feature information of different layers of the image;

[0046] S2. Establish a feature weight allocation attention mechanism that assigns different weights based on the importance of different features;

[0047] S3, combining the content loss function, the perceptual loss function and the adversarial loss function to obtain a hybrid loss function;

[0048] S4. Establish a generative adversarial network based on residual dense connection through residual dense connection module, feature weight allocation attention mechanism and hybrid loss function;

[0049] S5. Training the residual densely connected generative adversarial network according to the training set in the image data set to obtain a trained residual densely connected generative adversarial network;

[0050] S6. Input the image to be reconstructed into the trained residual densely connected generative adversarial network to obtain a reconstructed super-resolution image.

[0051] like Figure 2-3As shown, in step S1, according to the principle of the residual connection module, the residual connection module connects the input image feature x through an internal skip connection i-1 And the result after input image feature mapping F(x i-1 ) is accumulated to obtain the residual mapping output F Res (x i ); weaken or eliminate the input image feature x i-1 The gradient attenuation problem that occurs during the mapping process strengthens the output results, and as the number of network layers increases, overfitting will not occur; the calculation process is as follows:

[0052] F Res (x i )=F(x i-1 )+x i-1

[0053] like Figure 2-4 As shown in the figure, the dense connection module mainly fuses local feature information and strengthens the fusion between internal features; the dense connection module converts x i All previous image features and x i-1 The residual mapping output is accumulated to obtain the dense mapping output F DB (x i ), the calculation process is as follows:

[0054]

[0055] like Figure 5 As shown in the figure, in the dense connection module structure, only the internal feature fusion of the block is focused on, and the connection between blocks is lacking, which leads to the problem of gradient attenuation between blocks. Therefore, a residual dense connection module is constructed based on the residual connection module and the dense connection module. This is a global residual connection to solve the problem of gradient attenuation between blocks. The calculation process is as follows:

[0056]

[0057] Where β is the residual scaling parameter, β = 0.2 in practice; x 0 is the initial variable; when i∈[1,n-1], the variable x i-1 With variable x i-1 The dense mapping output results are connected; when i = n, the initial variable x 0 Make a skip connection with the output data of the residual dense connection module.

[0058] like Figure 6As shown, in step S2, the feature weight allocation attention mechanism can effectively highlight important feature information and suppress background feature information. The core idea is to allocate different weights according to the importance of the features. The feature weight allocation attention mechanism inputs the initial features P and G through a 1*1*1 convolution operation and adds the corresponding position features, suppresses negative features through the ReLU function, normalizes the features through the Sigmoid function to obtain the processed result, and multiplies the processed result with the initial features P and G channel by channel to obtain the weighted feature p after the weight is allocated. l , the calculation process is as follows:

[0059]

[0060] In the formula, σ 1 is the ReLU function, W P T , W g T , and ψ T denote the unit transposed vector, and b g and b Ψ Both represent offsets, and b Ψ ∈R; R is a real number, σ 2 is the Sigmoid activation function, Θ att Contains linear transformation variable W p , W g , ψ and offset b g , b Ψ .

[0061] In step S3, the mixed loss function L SR The calculation process is as follows:

[0062] L SR (I lr , I hr , I sr )=L MSE (I hr , I sr )+6×10 -3 ×L prec (I hr , I sr )+1×10 -3 ×L WGANGP (I lr , I hr )

[0063] Where, L MSE is the content loss function, L prec is the perceptual loss function, LWGANGP is the adversarial loss function; I lr is the low-resolution image of the I image, I hr is the high-resolution image of the I image, I sr is the super-resolution image of the I image, and the image size is H×W.

[0064] The content loss function L in the mixed loss function MSE The calculation process is as follows:

[0065]

[0066] Where, L MSE is the loss function of the mean square error, which is the square value of the pixel difference at the corresponding position of the calculated image.

[0067] The perceptual loss function L in the hybrid loss function prec The calculation process is as follows:

[0068]

[0069] Where V x,y W represents the feature map obtained after the yth convolution activation before the xth maximum pooling layer in the VGG network; x,y and H x,y represents the W dimension and H dimension of the feature map in the VGG network.

[0070] The reasoning process of the adversarial loss function in the hybrid loss function is based on the principle of the Wasserstein GAN loss function, i.e., the WGAN loss function, in order to better prevent the occurrence of gradient loss in the generative adversarial network GAN. An important trick of Wasserstein GAN is to clip all the weights of the discriminator to a constant range [-c, c] to satisfy the condition that the Wasserstein distance can be derived. However, under the clipping strategy, the weights of the discriminator either tend to the minimum or the maximum, which causes the discriminator to behave like a binary network and weakens the nonlinear simulation ability of GAN. Therefore, it is proposed to replace the clipping operation with a gradient penalty, and a new term is added to the adversarial loss to limit the rapid change of the discriminator gradient. The adversarial loss function in the hybrid loss function adopts the WGAN loss function with a gradient penalty, and the calculation process is as follows:

[0071]

[0072] In the formula, E represents the mean, G represents the operation of the generator in the generative adversarial network, D represents the operation of the discriminator in the generative adversarial network, || || p represents the p-criterion, Indicates downsampling of I picture.

[0073] like Figure 7 As shown in the figure, a generative adversarial network based on residual dense connection is established through the residual dense connection module, the feature weight allocation attention mechanism and the hybrid loss function;

[0074] Example

[0075] In step S5, the image dataset may be the LUNA16 dataset, which is a subset of the largest public lung nodule dataset LIDC-IDRI; the dataset is deployed by the National Cancer Institute of the United States and is widely used in detecting and classifying early lung cancer; it contains 1,186 medical imaging data from low-dose lung CT images of 888 patients in the MHD format; after data sorting and analysis, it contains 1,186 PNG format images.

[0076] The minimum value of each dimension is [300, 300], the median value of each dimension is [360, 360], and the median value of each dimension is [400, 400]. The LUNA16 dataset is randomly divided into training set, test set and validation set according to the ratio of 70%, 20% and 10%. The training set is used to train the generative adversarial network based on residual dense connection, and the test set is input into the generative adversarial network based on residual dense connection to obtain the reconstructed super-resolution image, and the accuracy of the super-resolution image is verified by the validation set.

[0077] like Figure 8 As shown in the figure, Original PNSR / SSIM is the original image in the image dataset, Bicubic is an image super-resolution method based on multi-scale fusion, SRCNN is an image super-resolution method based on convolution operation, SRGAN is an image super-resolution method based on generative adversarial network, RDN is an image super-resolution method based on dense residual connection network, and ARDCGAN is a generative adversarial network method based on residual dense connection in this scheme; the above different methods and the method of the present invention are used to compare the effect diagrams on the same dataset. Through comparative analysis, the method of the present invention has better effects on PNSR and SSIM evaluation indicators than other methods. Therefore, the superiority of the method of the present invention can be proved, and the super-resolution image obtained is clearer, clearer and more accurate.

[0078] like Fig. 9As shown, in order to prove the improved effect of the present invention, an ablation experiment is adopted to verify the performance of the improved part. The first column is the original image; the second column is the super-resolution image generated by only the generative adversarial network in the generative adversarial network method based on residual dense connection; the third column is the super-resolution image generated by only the residual density connection block and the generative adversarial network structure in the generative adversarial network method based on residual dense connection; the fourth column is the super-resolution image generated by only the attention mechanism and the generative adversarial network structure in the generative adversarial network method based on residual dense connection, and the fifth column is the super-resolution image obtained by the generative adversarial network method based on residual dense connection of the present invention. Through the PSNR and SSIM evaluation indicators, it can be seen that the experimental effect of the method of the present invention is the best. The super-resolution image obtained by the method of the present invention is clearer than the super-resolution image obtained by other methods, and is more conducive to the subsequent observation of the picture; therefore, the effectiveness of the present invention can be proved.

[0079] The above is only a description of the preferred embodiments of the present invention. Ordinary technicians in this technical field can make several modifications and optimizations based on the above disclosure without departing from the above basic principles. These improvements and optimizations should be regarded as the understood protection scope of the present invention.

Claims

1. A generative adversarial network method for image super-resolution reconstruction, characterized by: The following steps are involved: S1. A residual connection module and a dense connection module are used to construct a residual dense connection module that connects feature information of different layers of the image; S2. Establish a feature weight allocation attention mechanism that assigns different weights based on the importance of different features; S3, combining the content loss function, the perceptual loss function and the adversarial loss function to obtain a hybrid loss function; S4. Establish a generative adversarial network based on residual dense connection through residual dense connection module, feature weight allocation attention mechanism and hybrid loss function; S5. Training the residual densely connected generative adversarial network according to the training set in the image data set to obtain a trained residual densely connected generative adversarial network; S6, inputting the image to be reconstructed into the trained residual densely connected generative adversarial network to obtain a reconstructed super-resolution image; In step S1, the residual connection module connects the input image feature x i-1 And the result after input image feature mapping F(x i-1 ) is accumulated to obtain the residual mapping output F Res (x i ), the calculation process is as follows: F Res (x i )=F(x i-1 )+x i-1 The densely connected module converts x i All previous image features and x i-1 The residual mapping output is accumulated to obtain the dense mapping output F DB (x i ), the calculation process is as follows: The residual dense connection module is constructed by the residual connection module and the dense connection module. The calculation process is as follows: Where β is the residual scaling parameter and x0 is the initial variable; In step S2, the feature weight allocation attention mechanism inputs the initial features P and G through convolution operations and adds the corresponding position features, and processes them through ReLU function and Sigmoid function to obtain the processed results, and multiplies the processed results with the initial features P and G channel by channel to obtain the weighted features P after the weights are allocated. l , the calculation process is as follows: Where σ1 is the ReLU function, W P T , W g T , and ψ T denote the unit transposed vector, and b g and b Ψ Both represent offsets, and b Ψ ∈R; R is a real number, σ2 is the Sigmoid activation function, Θ att Contains linear transformation variable W p , W g , ψ and offset b g , b Ψ .

2. The method of generating adversarial networks for image super-resolution reconstruction according to claim 1, characterized in that: In step S3, the mixed loss function L SR The calculation process is as follows: L SR (I lr ,I hr ,I sr ) =L MSE (I hr ,I sr )+6×10 -3 ×L prec (I hr ,I sr )+1×10 -3 ×L WGANGP (I lr ,I hr ) Where, L MSE is the content loss function, L prec is the perceptual loss function, L WGANGP is the adversarial loss function; I lr is the low-resolution image of the I image, I hr is the high-resolution image of the I image, I sr is the super-resolution image of the I image, and the image size is H×W.

3. The method of generating adversarial networks for image super-resolution reconstruction according to claim 2, characterized in that: The content loss function L in the mixed loss function MSE The calculation process is as follows: Where, L MSE is the loss function of the mean square error, which is the square value of the pixel difference at the corresponding position of the calculated image.

4. The method of generating adversarial networks for image super-resolution reconstruction according to claim 2, characterized in that: The calculation process of the perceptual loss function Lprec in the hybrid loss function is as follows: Where V x,y W represents the feature map obtained after the yth convolution activation before the xth maximum pooling layer in the VGG network; x,y and H x,y represents the W dimension and H dimension of the feature map in the VGG network.

5. The method of generating adversarial networks for image super-resolution reconstruction according to claim 2, characterized in that: The adversarial loss function in the hybrid loss function adopts the WGAN loss function with gradient penalty, and the calculation process is as follows: In the formula, E represents the mean, G represents the operation of the generator, D represents the operation of the discriminator, || || p represents the p-criterion, Indicates downsampling of I picture.

Images