Network structure and method for multi-scale super-resolution reconstruction of a single image

Abstract

The present invention discloses a network structure and method for multi-scale super-resolution reconstruction of a single image. The network structure of the present invention comprises: a feature extraction layer, the input of which is the original image, and the output of which is connected to the input layer of a dilated convolutional neural network; a cascade expansion layer, the input of which is connected to the output layer of a dilated convolutional neural network; an aggregation layer, which is used to fuse the low-level coding information of different scales of the feature extraction layer and the decoding information of the cascade expansion layer; a reconstruction module, the input of which is the output of the cascade expansion layer, and the output of which is the reconstructed image. The present invention adopts a back-end upsampling network to perform image mapping transformations in a low-resolution space, thereby reducing the computational complexity and spatial complexity.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and artificial intelligence, and in particular to a single image multi-scale super-resolution reconstruction network structure and method. Background Art

[0002] Digital image super-resolution reconstruction technology is an important part of information processing technology. It is one of the main means to generate high-resolution images from low-resolution images. It has been widely used in many fields: for example, urban drainage network defect detection requires reconstruction of high-definition images from low-resolution images to achieve defect identification and classification; when using remote sensing images to detect objects, it is necessary to reconstruct high-resolution remote sensing images with more prominent details from low-resolution images; in water quality monitoring, image data is easily blurred by water vapor images and poor channel signals, so super-resolution reconstruction is needed to restore more detailed information; in medicine, fuzzy image reconstruction technology is used to magnify subtle lesions that cannot be distinguished by the human eye without losing image information, etc. Therefore, improving image resolution without losing image detail texture is one of the main technologies in the field of image processing.

[0003] CN202011318695.X discloses a single image super-resolution method based on a multi-scale channel attention mechanism, and proposes a super-resolution method using multi-scale channel attention reconstruction. However, this method has the following two problems:

[0004] (1) The upsampling method pre-set in this method will introduce problems such as blurring (noise) and noise amplification (noise amplification). At the same time, because the network is interpolated to a high-resolution space at the front end, the required storage space and time consumption are much higher than other types of super-resolution networks;

[0005] (2) This method uses multi-scale channel attention to better extract global information, but it fails to consider the fusion of low-level image encoding information and high-level decoding information. The network cannot make up for the detailed information lost in the convolution process, resulting in artifacts after reconstruction. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a single image multi-scale super-resolution reconstruction network structure and method in view of the shortcomings of the existing technology, so as to increase the amount of image information in the reconstruction stage and avoid artifacts in the reconstructed image.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: a single image multi-scale super-resolution reconstruction network structure, comprising:

[0008] Feature extraction layer, the input is the original image, and the output is connected to the input layer of the hole convolutional neural network (NLA);

[0009] Cascade expansion layer, the input is connected to the output layer of the dilated convolutional neural network;

[0010] An aggregation layer, used to fuse the low-level coded information of different scales in the feature extraction layer and the decoded information after upsampling in the reconstruction module;

[0011] The reconstruction module has the output of the cascade expansion layer as input and the reconstructed image as output.

[0012] The present invention uses an aggregation layer to fuse low-level coding information of different scales of the feature extraction layer and decoding information of the cascade expansion layer, thereby increasing the amount of image information in the image reconstruction stage, avoiding artifacts in the reconstructed image, and solving the problem of serious image information loss during image reconstruction. In view of the problem that the prior art upsampling method easily introduces noise into the image and has high computational and spatial complexity, the present invention uses a back-end upsampling network (cascade expansion layer) to perform image mapping transformations in a low-resolution space, and combines a dilated convolutional neural network (NLA module) to reduce computational complexity and spatial complexity.

[0013] The feature extraction layer comprises:

[0014] Convolution module, used to extract shallow features of the original image;

[0015] M residual components are connected to the convolution module to extract deep features; wherein M≥1.

[0016] The convolution operation maps the image to a low-resolution space, reducing the computational complexity and spatial complexity, and the residual component can extract deeper semantic information from the image.

[0017] The cascade expansion layer includes M cascade decoders; the first decoder is connected to the output layer of the hollow convolutional neural network, and the Mth decoder is connected to the reconstruction module. The cascade expansion layer adopts transposed convolution cascade decoding, which can adaptively learn the image encoding-decoding mapping relationship, obtain more detailed image information recovery features, and further improve the image reconstruction accuracy.

[0018] The input of the convolution module is superimposed with the output of the M-1th decoder as the input of the Mth decoder. Fusion of low-level features is beneficial to the reconstruction of image detail texture.

[0019] In the present invention, the aggregation layer includes N aggregation modules; wherein 1≤N≤M.

[0020] The polymer layer of the present invention has a variety of implementation structures, as follows:

[0021] When N=1, the M residual components are all connected to the aggregation module, and the output of the aggregation module is superimposed with the output of the hole convolutional neural network, or superimposed with the output of the i-th decoder, 1≤i<M. The aggregation module can fuse multi-scale low-level coding information, and superimposing with the output of the hole convolutional neural network can fuse high-level feature semantic information, and superimposing with the output of the i-th decoder is conducive to enhancing the image reconstruction information of this part.

[0022] When N=M, each aggregation module is connected to a residual component, and the output of each aggregation module is respectively superimposed on the output of the dilated convolutional neural network, the output of the first decoder, ..., and the output of the M-1 decoder. The aggregation module can fuse multi-scale low-level coding information, and can fuse high-level feature semantic information by superimposing the output of the dilated convolutional neural network, and can be superimposed on the output of the first decoder, ..., and the output of the M-1 decoder, thereby alleviating the image information loss caused by multiple upsampling operations.

[0023] When 1<N<M, each aggregation module is connected to at least one residual component, and the output of each aggregation module is superimposed on the output of the dilated convolutional neural network and the output of the 1st to jth decoders; where j≤N. The aggregation module can fuse multi-scale low-level coding information, and superimposing with the output of the dilated convolutional neural network can fuse high-level feature semantic information, and superimposing with the output of the 1st to jth decoders is conducive to compensating for the image information loss in each upsampling stage.

[0024] Preferably, in the present invention, M=4; N=3; the first aggregation module is connected to the four residual components, the second aggregation module is connected to the second to fourth residual components, and the third aggregation module is connected to the third and fourth residual components; the output of the third aggregation module is superimposed on the output of the hole convolutional neural network as the input of the first decoder; the output of the second aggregation module is superimposed on the output of the first decoder as the input of the second decoder; the output of the first aggregation module is superimposed on the output of the second decoder as the input of the third decoder. The four residual components can maximize the extraction of effective image features, and the three aggregation modules will not cause information redundancy on the basis of fusing multi-scale features. The connection method of the aggregation module with the residual components, the hole convolutional neural network, and the decoder can optimally fuse high-level semantic information and make up for the image information loss in each upsampling stage.

[0025] The aggregation module includes L first upsampling layers, and the outputs of the L first upsampling layers are connected to the input layer of the aggregation unit formed by cascading multiple upsampling layers of different multiples; wherein L is the number of residual components connected to the aggregation module. The aggregation module structure is conducive to the fusion of multi-scale feature information.

[0026] The dilated convolutional neural network (NLA) includes a plurality of parallel cascade branches, the input of the plurality of parallel cascade branches is the output of the feature extraction layer, and the output of the plurality of parallel cascade branches is connected to a fusion module; the cascade branch includes one or more convolution layers connected in series. The fusion module is used to superimpose (from the channel direction) the outputs of all cascade branches. In the present invention, a single cascade branch can greatly improve the receptive field and increase local feature aggregation by using multiple dilated convolutions.

[0027] The number of the cascade branches is 5; the number of convolution layers on the 5 cascade branches from top to bottom are 1, 2, 0, 3, and 4. The multiple cascade branches can extract multi-scale feature information and perform information fusion.

[0028] The present invention also provides a single image multi-scale super-resolution reconstruction method, which comprises the following steps:

[0029] S1. Acquire multiple images from an image data set as a sample set, and randomly divide the sample set into a training set and a validation set;

[0030] S2, using the training set to train a super-resolution reconstruction network, and using the validation set to adjust the model structure and hyperparameters to obtain a reconstruction model;

[0031] Wherein, the super-resolution reconstruction network is the network structure described in the present invention.

[0032] After step S1 and before step S2, the method further includes: performing enhancement processing on the images of the training set.

[0033] The method of the present invention further comprises: S3, using the original picture as input of the reconstruction model to obtain a reconstructed image.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1) In view of the problem that the upsampling method preset in the prior art easily introduces noise into the image and has high computational and spatial complexity, the present invention uses a back-end upsampling network to perform image mapping transformation in a low-resolution space, thereby reducing computational and spatial complexity;

[0036] 2) In view of the serious problem of image information loss in the reconstruction stage, the present invention adopts an aggregation module to realize the fusion of low-level and high-level information, which increases the amount of image information in the reconstruction stage and avoids the appearance of artifacts in the reconstructed picture; the aggregation layer effectively integrates low-level semantic information, which is conducive to the restoration of image information in the reconstruction stage and eliminates image artifacts;

[0037] 3) The present invention weakens the model's dependence on prior knowledge graphs and has a certain degree of self-learning. Even if there is no prior knowledge, it can effectively self-learn and self-reconstruct existing images;

[0038] 4) The dense atrous convolution structure greatly improves the image receptive field in the model, realizes multi-scale feature fusion, greatly reduces the information loss of the image during the convolution process, eliminates ringing noise and edge blur, and is conducive to the reconstruction of texture details in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a network structure diagram of an embodiment of the present invention;

[0040] Figure 2 This is a structural diagram of the aggregation module according to an embodiment of the present invention;

[0041] Figure 3 This is a structural diagram of a dense atrous convolution module according to an embodiment of the present invention;

[0042] Figure 4 This is a structural diagram of a super-resolution reconstruction module in Example 1 of the present invention;

[0043] Figure 5(a) to Figure 5(e) Fig. 5(a) is the result of the artifact processing experiment; Fig. 5(b) is the result of the reconstruction using the SRCNN method; Fig. 5(c) is the result of the reconstruction using the RCAN method; Fig. 5(d) is the result of the reconstruction using the IGNN method; Fig. 5(e) is the result of the reconstruction using the SRDCNN method (the method of the present invention); Figure 6(a) to Figure 6(e) Fig. 6 is the experimental result diagram of texture reconstruction details; Fig. 6(a) is the original low-resolution image; Fig. 6(b) is the reconstruction result diagram of the SRCNN method; Fig. 6(c) is the reconstruction result diagram of the RCAN method; Fig. 6(d) is the reconstruction result diagram of the IGNN method; Fig. 6(e) is the reconstruction result diagram of the SRDCNN method (the method of the present invention). DETAILED DESCRIPTION

[0044] like Figure 1 As shown, the super-resolution reconstruction network structure of the parallel stacking structure of Embodiment 1 of the present invention includes:

[0045] The feature extraction module includes 1 convolutional layer and 4 serially connected residual components, where the output of the convolutional layer is connected to the output of the cascaded expansion layer, and the outputs of the 2nd, 3rd, and 4th residual blocks (i.e., residual components) are connected to the input of the global pyramid aggregation module.

[0046] The global pyramid aggregation module (i.e., aggregation layer) is as follows Figure 2As shown in FIG, it is composed of three G modules (i.e., aggregation modules); the first G module input is the output of the 1st, 2nd, 3rd, and 4th residual blocks superimposed after upsampling at different multiples, the second G module input is the output of the 2nd, 3rd, and 4th residual blocks superimposed after upsampling at different multiples, and the third G module input is the output of the 3rd and 4th residual blocks superimposed after upsampling at different multiples.

[0047] like Figure 3 , the dense hole convolution module includes five parallel branches, Figure 3 In the figure, the five branches are arranged in order from top to bottom. The first branch contains one 3x3 dilated convolution with a dilation coefficient of 1; the second branch contains two 3x3 dilated convolutions with dilation coefficients of 3 and 1 respectively; the third branch directly connects the input with the output of other branches; the fourth branch contains three 3x3 dilated convolutions with dilation coefficients of 1, 3, and 1 respectively; the fifth branch contains four 3x3 dilated convolutions with dilation coefficients of 1, 3, 5, and 1 respectively.

[0048] The cascade expansion layer contains 4 serially connected Decoder block modules, where the Decoder block module contains two 1x1 convolutions and two 3x3 transposed convolutions.

[0049] The super-resolution reconstruction module consists of 1 convolution layer and a Pixelshuffle layer.

[0050] The image reconstruction method of embodiment 2 of the present invention is as follows:

[0051] S1. Extract 1000 RGB images from the public image super-resolution reconstruction dataset DIV2K as samples, and divide them into training set and validation set in a ratio of 8:2;

[0052] S2, to prevent the image content information from changing drastically and obvious boundaries from appearing, the Blend method and RGBpermute method were used to enhance the training set images in S1. After enhancement, the number of training sets was 5800;

[0053] S3. Construct a super-resolution reconstruction network based on a parallel stacking structure. Figure 1 As shown;

[0054] S4: Use the data set after data augmentation in S2 to train the network, use the validation set in S1 to adjust the model structure and hyperparameters, and set the loss function to L(θ).

[0055] Furthermore, the S3 is specifically:

[0056] S3.1 constructs an input layer and four residual components connected in series. The first convolutional layer and the series residual blocks form a feature extraction layer.

[0057] The first convolution layer in S3.1 is constructed by a common convolution layer and a maximum pooling layer, which is used to extract shallow features of the input image. The size of the common convolution kernel is set to 7X7, the stride is 2, and the pooling layer uses Maxpool.

[0058] The residual component layer in S3.1 is used to extract deep features of the image. The first residual component consists of three Resnet basic blocks, where the Resnet basic block contains two 3x3 convolutions. The activation function between the convolution layers uses Relu. The input and output are concat-operated at the channel level through skip connections. The mathematical expression of the residual block is:

[0059] x l+1 =x l +F(x l ,W l )

[0060] where x l+1 is the output, x l is the input, W l is the convolution kernel, F(x l ,W l ) is the residual part;

[0061] For a deeper layer L, its relationship with the lth layer can be expressed as:

[0062]

[0063] The L layer can be expressed as the sum of any l layers shallower than it and the residual part between them;

[0064] S3.1 The second, third, and fourth residual components are composed of 4, 6, and 3 resnet basic blocks respectively, and each residual component is followed by a maxpool pooling layer to downsample the image;

[0065] S3.2 The global pyramid aggregation module consists of three G modules; the first G module input is the output of the first, second, third, and fourth residual blocks superimposed after being upsampled by different multiples, and the second and third G modules input are the superposition of 2, 3, 4 and 3, 4 respectively;

[0066] The G module in S3.2 above is as follows Figure 2 As shown in the figure, for the input of multiple branches, a 1x1 convolution layer is first used to make the number of channels in each layer the same, and then upsampling is used to get the same size and then concat, and then a separable dilated convolution with different dilation coefficients is performed. Here, the dilation coefficient is set to 1, 2, and 4 to obtain the fusion information of each layer with different receptive fields, and then a 3×3 convolution layer is used to get the final result;

[0067]

[0068] Among them G k is the output of step k, F k is the feature map of the encoding layer at the kth step, Indicates that the upsampling ratio is 2 i-k , represents the feature concatenation operation, D Sconv @2 i-k Represents separable atrous convolution;

[0069] The S3.3 dense atrous convolution module has 5 parallel branches. Each branch is stacked in a cascade mode with a kernel of 3x3 atrous convolution. The first branch contains 1 3x3 atrous convolution with a dilation coefficient of 1; the second branch contains 2 3x3 atrous convolutions with dilation coefficients of 3 and 1 respectively; the third branch directly connects the input with the outputs of other branches; the fourth branch contains 3 3x3 atrous convolutions with dilation coefficients of 1, 3, and 1 respectively; the fifth branch contains 4 3x3 atrous convolutions with dilation coefficients of 1, 3, 5, and 1 respectively. This parallel structure utilizes the multi-resolution function, and the 5 branch structures are concat-operated at the end to achieve multi-scale information fusion.

[0070] The dense hole convolution module in S3.3 is as follows Figure 3 As shown in the figure, its role is to reduce the loss of image information and expand the receptive field to ensure the amount of information for the image resolution reconstruction task in the decoding stage. In each hole convolution branch, we apply a 1x1 convolution for rectified linear activation.

[0071] The receptive field calculation method here is:

[0072]

[0073] Among them, RF l+1 is the receptive field size corresponding to the current feature map, RF l is the receptive field size corresponding to the feature map of the previous layer, f l+1 is the convolution kernel size of the current convolution layer, Represents the stride product of the previous convolutional layer.

[0074] The multi-branch parallel structure in S3.3 uses the idea of ​​feature layering to directly stack the outputs of dilated convolutions with different expansion coefficients in layers, achieving the addition of different receptive fields, compensating for the grid effect brought by dilated convolutions, and solving the problems of local information loss and irrelevant information obtained at long distances.

[0075] S3.4 Cascade expansion layer Figure 1As shown in the figure, there are 4 Decoder blocks (decoder modules). The decoder module adopts a combination of 1x1 convolution layer + 3x3 transposed convolution + 1x1 convolution layer, and uses transposed convolution to learn adaptive mapping to obtain more detailed information recovery features. The input of the first Decoder blocks is the output of the dense void convolution module and the output of the third G module fused through the concat operation, and the input of the corresponding second Decoder blocks is the output of the first Decoder blocks and the output of the second G module fused through the concat operation. The input of the third Decoderblocks is the output of the second Decoder blocks and the output of the first G module fused through the concat operation, and the input of the fourth Decoder blocks is the output of the third Decoder blocks and the output of the first convolution layer of the feature extraction layer fused through the concat operation.

[0076] S3.4 The structure of super-resolution reconstruction module is as follows Figure 4 As shown, it includes a 1x1 convolution and a pixel shuffle (pixel reorganization) upsampling layer, which is used to gradually enlarge the image size and restore pixel information.

[0077] The super-resolution reconstruction module in S3.4 is as follows Figure 4 As shown in the figure, the Pixelshuffle upsampling layer is used to convert a H×W image into a high-resolution image of rH×rW, and a 1X1 convolution is used to obtain r 2 A channel feature map is generated (the size of the feature map is consistent with the input low-resolution image), and then the high-resolution image is obtained through a periodic screening method, where r is the upsampling factor, that is, the image enlargement ratio.

[0078] Furthermore, the S4 is specifically:

[0079] S4.1 uses Pytorch as the deep learning framework. All models are trained on 2 NVIDIA GTX2080 GPUs.

[0080] S4.2 uses mean square error to define the objective function. Given the data set {(x1,y1),…(x n ,y n )}, the objective function is defined as follows, where n is the number of samples in the training set.

[0081]

[0082]

[0083] Where F(x i ; θ) is the feature extraction function; x i is the input feature; i is the corresponding true value; θ is the optimal parameter to be solved, θ0, θ1, ... θ n is the optimal parameter obtained in each iteration; L(θ) is the objective function;

[0084] S4.3 uses stochastic gradient descent (SGD) to minimize the objective function during training. The update process can be written as follows:

[0085]

[0086] Where α is the step size, is the gradient;

[0087] The specific steps are as follows: First, the network parameters θ are randomly initialized, and then the parameters θ are updated according to the following rules to minimize the objective function:

[0088]

[0089]

[0090] …

[0091]

[0092] Perform n iterations until the requirements are met, the loop ends, and the optimal θ value is obtained.

[0093] S4.4 uses the pytorch framework API and the torch.save() function to save the best training weights, and loads the model parameters through the torch.load() function. The trained model is used to perform inference tests on the test set in S.1 to achieve super-resolution reconstruction of the model.

[0094] Test results:

[0095] The trained model is used to test the test set data. The experimental results and model parameters are as follows:

[0096] Compared with other methods, the present invention has smaller computational complexity and space complexity, and has better peak signal-to-noise ratio and structural similarity evaluation.

[0097]

[0098]

[0099] The reconstructed image test results with a magnification scale of 4 are selected for comparison.

[0100] 1. Artifact phenomenon

[0101] according to Figure 5(a)~5(e) The results shown show that, Figures 5(a) to 5(d) In the enlarged image of the roof in the lower right corner, there are extra lines at the junction of the roof and the sky. In the enlarged image of the roof in the lower left corner of Figure 5(e), this problem is better eliminated, which shows that the method of the present invention can effectively eliminate the artifact phenomenon in the edge transition area of ​​the image.

[0102] 2. Texture reconstruction details

[0103] according to Figure 6(a)~6(e) Compared with the results of other methods, it can be clearly seen that Figure 6(a) to 6(d) The restored detail textures in the enlarged image of the window in the lower right corner are relatively sparse, while the restored detail textures in the enlarged image of the window in the lower right corner of FIG6(e) are relatively more, which shows that the inventive method has a better effect on texture detail reconstruction.

Claims

1. A single image multi-scale super-resolution reconstruction network structure, characterized in that: include: Feature extraction layer, the input is the original image, and the output is connected to the input layer of the hole convolutional neural network; Cascade expansion layer, the input is connected to the output layer of the dilated convolutional neural network; An aggregation layer, used to fuse the low-level coding information of different scales extracted by the feature extraction layer and the decoding information obtained by the cascade expansion layer; A reconstruction module, whose input is the output of the cascade expansion layer and whose output is the reconstructed image; The feature extraction layer comprises: Convolution module, used to extract shallow features of the original image; M residual components, connected to the convolution module, for extracting deep features; wherein M ≥ 1; The cascade expansion layer includes M cascaded decoders; the first decoder is connected to the atrous convolutional neural network output layer, and the Mth decoder is connected to the reconstruction module; The input of the convolution module is superimposed with the output of the M-1th decoder as the input of the Mth decoder; The aggregation layer includes N aggregation modules; wherein 1≤N≤M; When N=1, the M residual components are all connected to the aggregation module, and the output of the aggregation module is superimposed with the output of the hole convolutional neural network, or superimposed with the output of the i-th decoder, 1≤i<M; When N=M, each aggregation module is connected to one of the residual components, and the output of each aggregation module is respectively superimposed on the output of the dilated convolutional neural network, the output of the first decoder, ..., and the output of the M-1th decoder; When 1<N<M, each aggregation module is connected to at least one residual component, and the output of each aggregation module is superimposed on the output of the hole convolutional neural network and the output of the 1st to jth decoders; wherein j≤N.

2. The single image multi-scale super-resolution reconstruction network structure according to claim 1, characterized in that: M=4; N=3; the first aggregation module is connected to the four residual components, the second aggregation module is connected to the second to fourth residual components, and the third aggregation module is connected to the third and fourth residual components; the output of the third aggregation module is superimposed on the output of the void convolutional neural network as the input of the first decoder; the output of the second aggregation module is superimposed on the output of the first decoder as the input of the second decoder; the output of the first aggregation module is superimposed on the output of the second decoder as the input of the third decoder.

3. The single image multi-scale super-resolution reconstruction network structure according to claim 2, characterized in that: The aggregation module includes L first upsampling layers, and the outputs of the L first upsampling layers are connected to the input layer of an aggregation unit formed by cascading multiple upsampling layers with different multiples; wherein L is the number of residual components connected to the aggregation module.

4. The single image multi-scale super-resolution reconstruction network structure according to any one of claims 1 to 3, characterized in that: The hole convolutional neural network includes multiple parallel cascade branches, the input of the multiple parallel cascade branches is the output of the feature extraction layer, and the output of the multiple parallel cascade branches is connected to the fusion module; the cascade branches include one or more convolutional layers connected in series.

5. The single image multi-scale super-resolution reconstruction network structure according to claim 4, characterized in that: The number of the cascade branches is 5; the numbers of the convolutional layers connected in series on the 5 cascade branches are 1, 2, 0, 3, and 4 respectively.

6. A single image multi-scale super-resolution reconstruction method, characterized in that: The following steps are involved: S1. Acquire multiple images from an image data set as a sample set, and randomly divide the sample set into a training set and a validation set; S2, using the training set to train a super-resolution reconstruction network, and using the validation set to adjust the super-resolution reconstruction network structure and hyperparameters to obtain a reconstruction model; Wherein, the super-resolution reconstruction network is a network structure as described in any one of claims 1 to 5.

7. The method according to claim 6, characterized in that After step S1 and before step S2, the method further includes: performing enhancement processing on the images of the training set.

Images