Single blurred image super-resolution reconstruction method and device and storage medium

Abstract

The invention provides a single blurred image super-resolution reconstruction method and device, and a storage medium. The method comprises the following steps: S1, obtaining a data set; s2, constructing a super-resolution reconstruction model; s3, training a super-resolution reconstruction model; s4, testing the trained super-resolution reconstruction model; s2 comprises the steps of S21, constructing a shallow feature extraction module SFE; s22, constructing a deep feature extraction module; s23, a multi-feature fusion module MFF module is constructed; and S24, constructing an image reconstruction module IRB. The problems that the calculation amount of a super-resolution reconstruction task is large, a reconstructed image loses high-frequency information seriously, and the visual effect of the reconstructed image is poor are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image reconstruction, and in particular relates to a method, a device and a storage medium for super-resolution reconstruction of a single blurred image. Background Art

[0002] The image quality is often affected by various factors during the image acquisition process, which has a serious impact on the application of the subsequent images. Compared with the design of complex optical systems to improve imaging quality, super-resolution technology has a good ability to restore image texture details, which not only greatly reduces the imaging cost, but also breaks the limit of the optical imaging system. Therefore, it can be applied to many fields such as monitoring systems and medical images, and has important research value.

[0003] Super-resolution image reconstruction technology has gone through three stages: interpolation-based methods, reconstruction-based methods, and learning-based methods. However, these methods have many limitations. For example, the use of interpolation methods will make the edges of SR images too smooth and blurred, especially in the high-frequency information area of ​​the image. The defects of the interpolation method are particularly obvious; when using the learning method, there are many restrictions on the training samples, and the noise in the image has a significant impact on the learning method. Super-resolution image reconstruction methods based on deep learning have gradually shown excellent performance and have become a hot spot in current super-resolution work.

[0004] Traditional super-resolution reconstruction methods focus on finding the mapping relationship potential inside the blurred image, and combine the prior knowledge of the image to improve the resolution of the image. The edges of the reconstructed high-definition image are too smooth and blurred, especially in the high-frequency information area of ​​the image. The defects of the interpolation method are particularly obvious and cannot restore the high-frequency information. In addition, the traditional method has many restrictions on training samples and cannot meet the requirements of actual super-resolution image tasks. Although the learning-based method has made great progress compared to the traditional method, it can only extract local image features. It is restricted by the size limit of the convolution kernel of the convolution operation and cannot learn global information. It cannot achieve satisfactory efficiency in long-distance modeling, which is not conducive to the restoration of global texture details of the image.

[0005] In recent years, Transformer has received great attention. Different from the CNN structure, Transformer can not only capture global information but also utilize the self-similarity of images through the self-attention mechanism. With the increasing demand for high-resolution images and videos in modern life, high-resolution images are flooding our lives and providing indispensable help for work and study. At present, traditional super-resolution reconstruction methods have many defects: (1) The existing network has a large amount of computation, and the pursuit of deeper and wider networks leads to a large number of parameters; (2) The reconstructed image has serious high-frequency information loss and cannot effectively restore some key information; (3) The image reconstructed by the existing super-resolution network is significantly different from the original high-definition original image in terms of color, details, and edge texture, resulting in poor visual effects.

[0006] Therefore, there is an urgent need for an effective blurred image super-resolution reconstruction method, device and storage medium to address the defects of traditional super-resolution reconstruction methods. Summary of the invention

[0007] The purpose of the embodiments of the present invention is to provide a single blurred image super-resolution reconstruction method, device and storage medium to solve the problems of large computational complexity of the super-resolution reconstruction task, serious loss of high-frequency information of the reconstructed image, and poor visual effect of the reconstructed image.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is a single blurred image super-resolution reconstruction method, comprising the following steps:

[0009] S1: Get the dataset;

[0010] S2: Build a super-resolution reconstruction model;

[0011] S3: training super-resolution reconstruction model;

[0012] S4: Test the trained super-resolution reconstruction model.

[0013] Further, the S2 includes:

[0014] S21: construct shallow feature extraction module SFE;

[0015] S22: build deep feature extraction module;

[0016] S23: construct a multi-feature fusion module MFF module;

[0017] S24: construct an image reconstruction module IRB.

[0018] Furthermore, in S21:

[0019] Shallow feature extraction module SFE, including 1 3×3 convolution;

[0020] In the shallow feature extraction module SFE, the image extraction process is as follows:

[0021]

[0022] Among them, H SFE represents the convolution layer SFE with weight W0, represents the convolution operation, and I LR Represents the original blurred image of the input, F0 SFE Represents the extracted shallow features;

[0023] The S22 includes:

[0024] S221: Build ESA module;

[0025] In the ESA module, the input feature map Through a 1×1 convolutional layer, the number of channels is compressed from C to C / 12, and the channel-compressed feature map F1 is obtained. ESA ; Then pass a 3×3 convolution layer with a step size of 2 to perform spatial downsampling to halve the spatial size of the feature map; perform a maximum pooling operation on the downsampled feature map; then use a 3×3 convolution layer with a padding of 1 and a ReLU function to extract padding and ReLU activation; then perform a spatial upsampling operation to obtain the upsampled feature map F2 ESA , the channel-compressed feature map F1 ESA Feature map F2 after 1×1 convolution operation and upsampling ESA Add, and then restore the number of channels to the input state through a 1×1 convolution layer; finally, generate an adaptive weight map through the Sigmoid activation function; input feature map Multiply the generated adaptive weight map element by element, and the generated feature map is the output of the ESA module The calculation formula of the ESA module is as follows:

[0026]

[0027]

[0028] in, Represents the characteristic graph of the input ESA module; H conv1 Represents a 1×1 convolution operation, F1 ESA Represents the feature map obtained after 1×1 convolution operation; H conv2 represents a 3×3 convolution with a stride of 2, H maxpool represents the maximum pooling operation, H gconv H represents a 3×3 convolutional layer with padding of 1 and ReLU function, which performs extraction padding and ReLU activation; up represents the upsampling recovery layer; F2 ESA Indicates F1 ESA Feature map obtained through convolution, pooling, group convolution, and upsampling operations; The output characteristic diagram of the ESA module, H sigmoid is the Sigmoid function, * represents element-by-element multiplication operation, and + represents element-by-element addition operation.

[0029] Furthermore, the S22 further includes:

[0030] S222: build CCFN module;

[0031] In the CCFN module, the input feature map After the first fully connected layer, GELU activation function, and 3×3 deep convolution operation, the first feature map is obtained and the first feature map is combined with the input feature map Add element by element to get the intermediate stage process feature map f1, and pass f1 through the Dropout layer, the second fully connected layer h fc2 , Dropout layer to obtain the second feature map, the second feature map is centrally output and then added to the second feature map to obtain the output feature of the CCFN module

[0032]

[0033] in, is the input feature map of the CCFN module, h fc1 、h fc2 is the first fully connected layer, the second fully connected layer, h dropo 、h GELU They represent the Dropout layer and activation function, respectively. DWConv represents the depth convolution, f1, They are the process feature map of the intermediate stage and the final output of the CCFN module; H CB Represents a centralized output mechanism.

[0034] Furthermore, the S22 further includes:

[0035] S223: Build EATB module;

[0036] The lightweight CNN-Transformer module EATB consists of 2 LayerNorms, 1 ESA, 1 SimAM, 1 SW-MSA and 1 CCFN;

[0037] In the EATB module, the input feature map is normalized by the LayerNorm layer. Normalize all features to get the feature map Then the extracted feature map Divided into 2 branches;

[0038] The first branch uses the traditional convolution route, that is, the feature map As the input of the ESA enhanced spatial attention module of S221 Then the feature map obtained by the ESA module Perform SimAM parameter-free simple attention operation;

[0039] The second branch uses the Transformer route. When passing through the SW-MSA multi-head self-attention mechanism, the unfold function is used to Converted into input tensor X, the initial shape of input tensor X is (B, C, H, W), where B is the batch size, C is the number of channels, H is the height, and W is the width. Use the window_partition function to divide the input tensor X according to the window size. The shape of each sub-window tensor is (n, w, w', C), the number of sub-windows is n, and the width and height of the sub-window are w and w';

[0040] For each sub-window, use the fully connected layer to generate the query Q, key K, value V respectively, and adjust the shape of the Q, K, V tensors, then calculate the attention score attn, get the relative position bias, and adjust the shape and dimension order of the relative position bias, add the relative position bias to the attention score, and then perform softmax processing on the attention score and process it through the dropout layer, then calculate the attention output and process it through the linear layer and dropout layer to get the output tensor of the SW-MSA module;

[0041] Then, the outputs of the first branch and the second branch are added to obtain the feature map F2 EATB , and add the input feature map Then the features processed by LayerNorm layer and CCFN module are normalized and compared with F2 EATB , Add together to get the EATB module output

[0042] The EATB module performs deep extraction according to the following formula:

[0043]

[0044]

[0045] in, It represents the feature map obtained by adding the output of the first branch and the second branch. represents the input feature map of module EATB, H SW-MSA represents the sliding window self-attention mechanism SW-MSA, H SimAM Indicates SimAM operation, H CCFN Indicates CCFN operation, Indicates the output of the EATB module, H LN Represents the LayerNorm layer normalization operation, Represents the input feature map The feature map obtained by LayerNorm layer normalization; Represents the output feature map of the ESA module.

[0046] Furthermore, the S22 further includes:

[0047] S224: construct a PCA module; PCA consists of 2 LayerNorms, 1 OCA and 1 CCFN;

[0048] In the PCA module, the input feature map x is first layer-normalized, and then the OCA cross-overlapping block operation is performed: the shape and size of the input feature map x are adjusted, the query Q, key K, and value V are generated through the fully connected layer, the window_partition function is used to divide Q into small windows, and the unfold operation is used to divide K and V into small windows. The overlapping parts of each sub-window area are processed by the expansion operation; the shapes of Q, K, and V are adjusted and the attention weight attn is calculated, the relative position bias is obtained, the relative position bias is added to the attention score, and the attention weight is restored to the size of the input feature map x through the window_reverse function to obtain the output of the OCA module; this output is added to the input feature map x, and then layer normalization and CCFN are further processed. The output obtained after layer normalization and CCFN processing is added to the output of the OCA module and the input feature map x to obtain the output of the PCA module.

[0049] Furthermore, the S22 further includes:

[0050] S225: build deep feature extraction module;

[0051] The deep feature extraction module consists of four lightweight CNN-Transformer modules LHBCT; each LHBCT contains 3 embedding attention Transformer blocks EATB, 1 pixel cross-window attention PCA and 1 3×3 convolution block;

[0052] The LHBCT module performs deep feature extraction as a whole, and performs deep extraction according to the following formula:

[0053]

[0054] Among them, F0 LHBCT Represents the feature map of the input LHBCT module, represents the output of an LHBCT module, f 3×3Conv represents a 3×3 convolution operation, H PCA Indicates that after the PCA module operation, represents EATB operation, k∈{1, 2, 3}, k represents EATB at different levels;

[0055] The output F of the deep feature extraction module k The calculation formula is:

[0056]

[0057] in, represents the LHBCT operation, k∈{1, 2, 3, 4}, k represents different levels of LHBCT, F k represents the output F of the deep feature extraction module k .

[0058] Further, the S23 includes:

[0059] Multi-feature fusion MFF includes a 3×3 convolution and a residual structure. In the MFF module, the feature map F extracted by the deep feature extraction module is k After a 3×3 convolution operation, the feature map obtained by the 3×3 convolution is combined with the output feature map of the shallow feature extraction module SFE. Add them together to get the output of the MFF module;

[0060] The S24 includes:

[0061] The image reconstruction module IRB includes two 3×3 convolutions and one pixel shuffle upsampling layer. In the image reconstruction IRB module, the output feature map of the MFF module is used as input, first undergoes a 3×3 convolution operation, then an image upsampling operation, and finally a 3×3 convolution operation to obtain the final output F output ;

[0062] In the multi-feature fusion module MFF and the image reconstruction module IRB, the calculation formulas for the fusion and reconstruction process are as follows:

[0063]

[0064] Among them, F output represents the output SR image of the super-resolution reconstruction model, f 3×3Conv represents a 3×3 convolution operation, f PixelShuffle represents the upsampling operation, F k represents the output of the deep feature extraction module, F0 SFE Represents the output feature map of the SFE module.

[0065] Another technical solution adopted by the present invention is a single blurred image super-resolution reconstruction device, which adopts the above method to achieve single blurred image super-resolution reconstruction.

[0066] Another technical solution adopted by the present invention is a single blurred image super-resolution reconstruction storage medium, which adopts the above method to achieve single blurred image super-resolution reconstruction.

[0067] The beneficial effects of the present invention are

[0068] 1. The present invention mixes CNN and Transformer structures to give full play to their ability to restore and process different information of blurred images. For various low-resolution images, more detailed features are extracted. The reconstruction process of the present invention can ensure the prominence of significant features, reduce the loss of information during the reconstruction process, better restore the real scene, and solve the image blur problem caused by hardware defects of imaging equipment at a lower cost. The present invention can effectively restore the detailed information of low-resolution images for blurred images, improve contrast, and improve visual effects.

[0069] 2. The existing network has a large amount of calculation, and the pursuit of deeper and wider networks leads to a large number of parameters. The present invention constructs a deep feature extraction module LHBCT to achieve lightweight, so that a relatively good blurred image reconstruction effect can be achieved without a large number of parameters and computing resources.

[0070] 3. The existing network reconstructed images have serious high-frequency information loss and cannot effectively restore some key information. The present invention uses the EATB module to combine the convolution layer and the Transformer structure to better retain the high-frequency and low-frequency information of the image. It can also effectively extract complementary features and better retain the high-frequency and low-frequency information of the image.

[0071] 4. The present invention adds a convolution block based on the spatial attention ESA module to the Transformer to enhance the representation ability of the network and further focus the network's attention on the high-frequency information part. As a 2D attention mechanism, ESA can quickly obtain the spatial structural features of the image in the reconstruction network and treat different positions differently.

[0072] 5. The PCA module establishes cross-window connections in the Self-Attention in the network, which enhances the representation ability of the window self-attention. Since the addition of CCFN strengthens the nonlinearity of the network, the CCFN module of the present invention adds a deep separable convolution, an activation function and a centralized output mechanism. The deep separable convolution and activation function provide the ability to extract local features. The convolution layer not only captures local spatial information but also has the characteristics of parameter sharing, which is more efficient; the centralized output mechanism can make the data distribution more even, which plays a certain role in the stable training of the super-resolution reconstruction model and reduces the risk of gradient explosion or disappearance. Therefore, the CCFN module can further refine the deep features that represent the image details based on the existing features, capture the local spatial information and share parameters, aiming to bring more intensive interactions, thereby stabilizing the network and improving the generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0074] Figure 1 It is a flow chart of the architecture of the method of the present invention.

[0075] Figure 2a It is a flow chart of the EATB module of the method of the present invention.

[0076] Figure 2b It is a flow chart of the ESA module of the method of the present invention.

[0077] Figure 2c It is a flow chart of the LHBCT module of the method of the present invention.

[0078] Figure 3a It is a flow chart of the PCA module of the method of the present invention.

[0079] Figure 3b It is a flow chart of the CCFN module of the method of the present invention.

[0080] Figure 4 This is a comparison chart of the reconstruction effects of the content of the image Urban100:img076 at a 4-fold magnification using the method of the present invention and other methods.

[0081] Figure 5 This is a comparison chart of the reconstruction effects of the content of the image Urban100:imgg095 at a 4-fold magnification using the method of the present invention and other methods.

[0082] Figure 6 This is a comparison chart of the reconstruction effects of the method of the present invention and other methods on the content of picture Set14: barbara at a 4-fold magnification. DETAILED DESCRIPTION

[0083] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0084] This embodiment provides a method for super-resolution reconstruction of a single blurred image. Figure 1 The complete steps of the method of the present invention are as follows:

[0085] S1: Get a dataset. Use the public dataset DIV2K as the dataset, select the training set data and test set data in DIV2K, and perform data enhancement on them.

[0086] S2: Build a super-resolution reconstruction model. The model includes: shallow feature extraction module (SFE), deep feature extraction module, multi-feature fusion module (MFF), image reconstruction module (IRB). The specific steps are as follows:

[0087] S21: Build a shallow feature extraction module (SFE), including 1 3×3 convolution.

[0088] The original low-resolution image is input into SFE, and the image is extracted as follows:

[0089]

[0090] Among them, H SFE represents the convolutional layer SFE with weight W0, represents the convolution operation, Represents the extracted shallow features, such as contour information; it is not only used for the next step of deep feature extraction, but also directly sent to the MFF module for feature fusion. LR Represents the original blurred image to be processed.

[0091] S22: Construct a deep feature extraction module. The specific steps are as follows:

[0092] S221: Build ESA module.

[0093] like Figure 2b As shown, in the ESA module, the input feature map Through a 1×1 convolutional layer, the number of channels is compressed from C to C / 12, and we get Here, the number of channels is compressed from 48 to 4; then a 3×3 convolution layer with a step size of 2 is used to halve the spatial size of the feature map, that is, spatial downsampling is performed; the maxpool operation is performed on the downsampled feature map to aggregate local features; local convolution processing is then performed, using a 3×3 convolution layer with padding of 1 and a ReLU function to extract padding and ReLU activation, and convolution processing is performed on the features after maximum pooling to obtain local enhanced features, where the number of channels remains unchanged; and spatial upsampling is performed to obtain That is, use bilinear interpolation to restore the feature map to the spatial size of the input feature map x; perform a 1×1 convolution operation on the channel-compressed feature map and add it to the upsampled feature map, then restore the number of channels to the initial state through a 1×1 convolution layer; finally, generate an adaptive weight map through the Sigmoid activation function; Multiply the generated adaptive weight map element by element to generate an enhanced feature map as the final result The calculation formula of the ESA module is as follows:

[0094]

[0095]

[0096] in, Represents the feature map of the input ESA module, including texture detail information; H conv1 represents a 1×1 convolution operation, Represents the feature map obtained after the convolution operation; H conv2 represents a 3×3 convolution with a stride of 2, H maxpool represents the maximum pooling operation, H gconv H represents a group convolution consisting of three convolutional layers, i.e., a 3×3 convolutional layer with padding 1 and a ReLU function for extraction padding and ReLU activation; up Represents the upsampling recovery layer, which can expand the image resolution; Express Feature map obtained through convolution, pooling, group convolution, and upsampling operations; represents the output characteristic map of the ESA module, i.e., the feature map with the focused image area, H sigmoid is the Sigmoid function, * represents element-by-element multiplication operation, and + represents element-by-element addition operation.

[0097] S222: Build a CCFN module.

[0098] like Figure 3b In the CCFN module, CCFN can effectively perform complex nonlinear mapping and transformation on the input features through multiple linear transformations and nonlinear activation functions, enhance the nonlinear expression ability of the network, and thus better restore image details. The deep convolution layer DWConv is added to extract local features, enhance the model's ability to perceive details, improve the accuracy of feature extraction, reconstruct the high-frequency details of the image, and make the image look clearer and sharper. The implementation process of the CCFN module is as follows: Input feature map After the first fully connected layer, GELU activation function, and 3×3 deep convolution operation, the feature map is obtained. This feature map is combined with the first fully connected layer h fc1 , the feature maps obtained by the GELU activation function are added element by element to obtain the intermediate stage process feature map f1, and f1 is passed through the Dropout layer, the second fully connected layer h fc2 , Dropout layer to get the feature map, and combine this feature map with H CB Add together to get the output features of the CCFN module

[0099]

[0100] in, is the input feature map of the CCFN module, h fc1 、h fc2 is the first fully connected layer and the second fully connected layer, which maps the features of the input blurred image to a feature space of a higher dimension or target dimension. dropo 、h GELU They represent the Dropout layer and activation function, respectively, so that the model can learn more generalized features in image processing tasks. DWConv represents deep convolution, which can reduce the computational complexity of the image and enhance the preservation of image spatial information. They are the intermediate stage process feature map and the final output of the CCFN module containing the complex features in the enhanced image. CB Refers to the centralized output mechanism. The centralized output mechanism can make the data distribution more even, play a certain role in the stable training of the super-resolution reconstruction model, and reduce the risk of gradient explosion or disappearance.

[0101] A depthwise separable convolution, an activation function, and a centralized output mechanism are added to the CCFN module. The depthwise separable convolution and activation function provide the ability to extract local features. The convolution layer not only captures local spatial information but also has the characteristics of parameter sharing, which is more efficient; the centralized output mechanism can make the data distribution more even, which plays a certain role in the stable training of the super-resolution reconstruction model and reduces the risk of gradient explosion or disappearance. CCFN can realize nonlinear mapping of various features, aiming to bring more intensive interactions, optimize the overall visual Transformer model, and improve its generalization ability.

[0102] S223: Build EATB module.

[0103] The lightweight CNN-Transformer module EATB consists of 2 LayerNorms, 1 ESA, 1 SimAM, 1 SW-MSA, and 1 CCFN. Since the details in the image exist at different scales, the structure and shape of the contour are at a larger scale, and the edges and textures are usually at a smaller scale, EATB aims to enhance the representation capability of feature extraction.

[0104] like Figure 2a As shown, in the EATB module, the input feature map is normalized by the LayerNorm layer. Normalize all features to get the feature map Then the extracted feature map is divided into two branches;

[0105] The first branch uses the traditional convolution route, that is, As input of the ESA module of S221 By assigning different weights to different areas of the image, we can "focus" on the important parts of the image and automatically suppress irrelevant areas and background parts in the image. Execute the SimAM operation to implement traditional convolutional feature modeling on the input blurred image.

[0106] The second branch uses the Transformer route. When passing through the SW-MSA multi-head self-attention mechanism, the unfold function is used to Converted into input tensor X, the initial shape of input tensor X is (B, C, H, W), where B is the batch size, C is the number of channels, H is the height, and W is the width. Here, the initial shape of X is (32, 48, 48, 48). Use the window_partition function to divide the input tensor X according to the window size. Each window will contain a part of the original tensor X, and the shape of each sub-window tensor is (n, w, w', C), which is (1152, 8, 8, 48) here. The number of all sub-windows n is 1152, 8 and 8 are the width w and height w' of the sub-window, and 48 is the number of channels;

[0107] For each sub-window, a linear layer, i.e. a fully connected layer, is used to generate query Q, key K, and value V respectively, and the shapes of the Q, K, and V tensors are adjusted to (3, 1152, 6, 64, 8). Then the attention score attn is calculated, the relative position bias is obtained, and the shape and dimension order are adjusted. The relative position bias is added to the attention score, and then the attention score is softmax processed and dropout is applied to increase the generalization ability. Then the attention output is calculated and processed through the linear layer and dropout layer to obtain the final SW-MSA module output tensor.

[0108] Next, add the outputs of the first branch and the second branch to get the feature map F2 EATB , and add the input feature map A multi-feature, multi-scale feature map is obtained, which contains the original information, the global image information processed by the traditional convolution, and the local image information processed by the Transformer route. Finally, it is normalized by the LayerNorm layer, and then the high-frequency details of the image are reconstructed by the CCFN module constructed by S222, making the image look clearer and sharper. The features obtained by the LayerNorm layer and the CCFN module are compared with F2 EATB , Add together to get the final EATB module output

[0109] The EATB module performs deep extraction according to the following formula:

[0110]

[0111]

[0112] Among them, F2 EATB It represents the feature map of the image processing in the middle stage of EATB, that is, the feature map obtained by adding the outputs of the first branch and the second branch. H represents the input feature map of module EATB. SW-MSA represents the sliding window self-attention mechanism SW-MSA, H SimAM Indicates SimAM operation, H CCFN Indicates CCFN operation, That is the output result of the EATB module, H LN Represents the LayerNorm layer normalization operation. Indicates that the input feature map is normalized by the LayerNorm layer The feature map obtained by normalization; Represents the output feature map of the ESA module.

[0113] S224: Construct PCA module. PCA consists of 2 LayerNorms, 1 OCA and 1 CCFN, which aims to enhance the model's ability to extract features such as texture, details, and contours of images on sub-windows.

[0114] In the PCA module, such as Figure 3aAs shown in the figure, the input feature map x is first layer-normalized so that the features of each layer have the same mean and variance; then the OCA operation is performed: the shape and size of the input feature map x are adjusted, and then the query (Q), key (K), and value (V) are generated through linear transformation, that is, the fully connected layer; the window_partition function is used to divide Q into small windows, and the unfold operation is used to divide K and V into small windows. The overlapping parts of each sub-window area are processed by the expansion operation; the shapes of Q, K, and V are adjusted to adapt to the multi-head attention mechanism, and the attention weight attn is calculated, the relative position bias is obtained, the relative position bias is added to the attention score, and the attention weight is restored to the size of the input feature map x through the window_reverse function to obtain the output of the OCA module. This output is added to the input feature map x, and then further processed by layer normalization and CCFN. The output obtained after layer normalization and CCFN processing is added to the output of the OCA module and the input feature map x to obtain the final output of the PCA module. The PCA module can better capture the local features of the image, especially the details, through window division and overlapping operations; the multi-head attention mechanism can enhance the information exchange between different channels, improve the feature expression ability, and enhance the feature extraction ability of the model. The feature map is usually composed of multiple channels, each channel represents a different feature of the image, such as the edge, color, texture, etc. of the blurred image. Enhancing the information exchange between different channels means that the model can better integrate information from different channels, thereby improving the feature expression ability.

[0115] S225: Construct a deep feature extraction module. The deep feature extraction module is composed of four sets of lightweight CNN-Transformer modules LHBCT. Figure 2c As shown in the figure, each group of LHBCT consists of 3 embedded attention Transformer blocks EATB, 1 pixel cross-window attention PCA and 1 3×3 convolution block, aiming to extract more high-frequency information and low-frequency information. High-frequency information refers to the details in the image, such as edges, textures, noise, etc. This information changes quickly and represents subtle changes and sharp features in the image. Low-frequency information refers to the overall structure and general shape of the image, such as flat areas, gradients, etc. This information changes slowly and represents the overall outline and main content of the image.

[0116] The LHBCT module performs deep feature extraction as a whole, and performs deep extraction according to the following formula:

[0117]

[0118] Among them, F0 LHBCT Represents the feature map of the input LHBCT module, represents the output of each LHBCT module, f3×3Conv represents 3×3 convolution, H PCA represents the PCA enhanced image representation process, Represents EATB operation, k∈{1, 2, 3}, k represents EATB at different levels, and there are 3 levels in total.

[0119] The output F of the entire deep feature extraction module k The calculation formula is:

[0120]

[0121] in, represents the LHBCT operation, k∈{1, 2, 3, 4}, k represents different levels of LHBCT, there are 4 layers in total, F k It represents the final output F of the deep extraction module after LHBCT operation. k .

[0122] S23: Construct a multi-feature fusion module (MFF). The multi-feature fusion (MFF) module includes a 3×3 convolution and a residual structure, which aims to enhance the fusion of various features such as image color, shape, texture, contour and other information. In the MFF module, the feature map F extracted by the deep feature extraction module is k After a 3×3 convolution operation, the convolution feature map is combined with the output feature map of the shallow feature extraction module (SFE) Add them together to get the output of the MFF module.

[0123] S24: The image reconstruction module (IRB) includes two 3×3 convolutions and one pixel shuffle upsampling layer, which aims to increase the image resolution and output the final image. In the image reconstruction IRB, the output feature map of the MFF module is used as input, first undergoes a 3×3 convolution operation, then an image upsampling operation, and finally a 3×3 convolution operation to obtain the final output F output .

[0124] In multi-feature fusion (MFF) and image reconstruction (IRB), the calculation formulas for the fusion and reconstruction process are as follows:

[0125]

[0126] Among them, F output represents the output SR image of the super-resolution reconstruction model, f 3×3Conv represents a 3×3 convolution operation, f PixelShuffle represents the upsampling operation, F k represents the output of the deep feature extraction module, Represents the output feature map of the SFE module.

[0127] In the image reconstruction module at the end of the model, sub-pixel upsampling convolution is used, that is, the Pixel Shuffle operation is used to convolve the input low-resolution image or feature map to obtain a feature map with a specific number of channels. Then, these feature maps are rearranged into a larger high-resolution image. Suppose we need to enlarge the image by r times, then the number of channels of the feature map output by the first 3×3 convolution will be r times the number of original channels. 2 The pixel shuffle operation is used to rearrange the feature map into a high-resolution image. The advantage of the pixel shuffle operation is that it is more efficient, avoids complex interpolation calculations for each pixel, and requires less computing resources.

[0128] S3: Train the model using the training set. Update the model parameters during the training process.

[0129] S4: Test the trained model using the test set.

[0130] In order to test the model effect of the embodiment of the present invention, a comparative experiment was carried out. Two objective indicators, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), were used for quantitative evaluation. The validation sets were four data sets, Set5, Set14, B100, and Urban100, which contained pictures of different types from the training sets. Reconstruction and restoration were performed on the four data sets at three different magnifications of 2, 3, and 4 times. The super-resolution networks compared with the method of the present invention are Bicubic, FSRCNN, LAPAR-A, PAN, CARN, SMSR, IMDN, RLFN, LBNet, FDIWN, BSRN, ESRT, and MBFSR. The comparison data of PSNR / SSIM results of different methods on different data sets are shown in Table 1.

[0131] Table 1 Comparison of PSNR / SSIM results of different methods at different magnifications and on different datasets

[0132]

[0133]

[0134] In Table 1, the boldface indicates the best index. It can be seen that the ENLCT network performance of the method of the present invention is the best, and it also achieves good control in both network parameters. For example, in the comparison between ENLCT and LBNet, under a 2-fold magnification factor, in each data set, the PSNR value of the index increased by 0.1-0.19dB, and the SSIM value increased by 0.0005-0.0026. In the comparison between ENLCT and ESRT, under a 4-fold magnification factor, the PSNR value of the index increased by 0.56% and 0.21% in the Set5 and Set14 data sets, and the SSIM value increased by 0.25% and 0.24%. On the Urban100 data set, the algorithm was slightly lower than ESRT when magnified 2 and 4 times. It was found through the original data that the Urban100 data set was clearer than other data sets, and the algorithm had more advantages in processing more blurred data. In summary, the ENLCT model has a significant improvement compared with other methods, and can achieve relatively excellent super-resolution performance with a smaller number of model parameters.

[0135] In order to verify the efficiency advantage of the method of the present invention, the method ELCTT of the present invention is compared with LAPAR-A, PAN, CARN, SMSR, IMDN, RLFN, LBNet, FDIWN, BSRN, ESRT and MBFSR in terms of parameter quantity. The comparison results are shown in Table 2:

[0136] Table 2 Comparison of parameters, PSNR values, and SSIM values ​​of different methods in Set5×2 dataset

[0137]

[0138] It can be seen from Table 2 that the parameters of the embodiment of the present invention are relatively small, and the PSNR and SSIM indicators are both the highest. It can be seen that among the 12 methods, the method of the present invention achieves the best reconstruction effect with a moderate parameter amount.

[0139] like Figure 4 , Figure 5 , Figure 6 As shown in the figure, Bicubic, FSRCNN, PAN, CARN, IMDN, RLFN, LBNet, FDIWN, and BSRN algorithms are used to perform super-resolution reconstruction of buildings, ground, and people, respectively. The HR original image is used as a reference for comparison, and our represents the result obtained using the method of the present invention. It can be observed that CARN and RLFN have the worst visual effects, while BSRN and PAN have relatively good visual effects. The restoration effects of other algorithms are also blurred or distorted, while the ENLCT model (our) of the present invention can restore the details of the original image to a greater extent. Figure 4 In the middle image Urban100:img076, only the ELLCT model restores the line details of the building best; Figure 5 In the middle image Urban100:imgg095, only the image reconstructed by the ELCCT model has clear contours and strong contrast, which is closest to the high-resolution image. Figure 6 In the middle picture Set14: barbara, only ENLCT can restore clearer detailed textures of the books in the bookcase. Therefore, the method of the embodiment of the present invention effectively restores the contour of the original high-resolution image and improves the clarity of the blurred image.

[0140] In summary, the method of the present invention achieves a good balance between image reconstruction effect and parameter quantity, and can meet the application of mobile devices in terms of model lightweight.

[0141] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0142] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for super-resolution reconstruction of a single blurred image, characterized in that: The following steps are involved: S1: Get the dataset; S2: Build a super-resolution reconstruction model; S3: training super-resolution reconstruction model; S4: Test the trained super-resolution reconstruction model.

2. The method for super-resolution reconstruction of a single blurred image according to claim 1, characterized in that: The S2 includes: S21: construct shallow feature extraction module SFE; S22: build deep feature extraction module; S23: construct a multi-feature fusion module MFF module; S24: construct an image reconstruction module IRB.

3. The method for super-resolution reconstruction of a single blurred image according to claim 2, characterized in that: In S21: Shallow feature extraction module SFE, including 1 3×3 convolution; In the shallow feature extraction module SFE, the image extraction process is as follows: Among them, H SFE represents the convolution layer SFE with weight W0, represents the convolution operation, and I LR represents the original blurred image of the input, Represents the extracted shallow features; The S22 includes: S221: Build ESA module; In the ESA module, the input feature map Through a 1×1 convolutional layer, the number of channels is compressed from C to C / 12, and the channel-compressed feature map F1 is obtained. ESA ; Then pass a 3×3 convolution layer with a step size of 2 to perform spatial downsampling to halve the spatial size of the feature map; perform a maximum pooling operation on the downsampled feature map; then use a 3×3 convolution layer with a padding of 1 and a ReLU function to extract padding and ReLU activation; then perform a spatial upsampling operation to obtain the upsampled feature map The channel-compressed feature map F1 ESA Feature maps after 1×1 convolution and upsampling Add, and then restore the number of channels to the input state through a 1×1 convolution layer; finally, generate an adaptive weight map through the Sigmoid activation function; input feature map Multiply the generated adaptive weight map element by element, and the generated feature map is the output of the ESA module The calculation formula of the ESA module is as follows: in, Represents the characteristic graph of the input ESA module; H conv1 Represents a 1×1 convolution operation, F1 ESA Represents the feature map obtained after 1×1 convolution operation; H conv2 represents a 3×3 convolution with a stride of 2, H maxpool represents the maximum pooling operation, H gc onv represents a 3×3 convolutional layer with padding of 1 and ReLU function, which performs extraction padding and ReLU activation; H up represents the upsampling recovery layer; Indicates F1 ESA Feature map obtained through convolution, pooling, group convolution, and upsampling operations; The output characteristic diagram of the ESA module, H sigmoid is the Sigmoid function, * represents element-by-element multiplication operation, and + represents element-by-element addition operation.

4. The method for super-resolution reconstruction of a single blurred image according to claim 3, characterized in that: The S22 further includes: S222: build CCFN module; In the CCFN module, the input feature map After the first fully connected layer, GELU activation function, and 3×3 deep convolution operation, the first feature map is obtained and the first feature map is combined with the input feature map Add element by element to get the intermediate stage process feature map f1, and pass f1 through the Dropout layer, the second fully connected layer h fc2 , Dropout layer to obtain the second feature map, the second feature map is centrally output and then added to the second feature map to obtain the output feature of the CCFN module in, is the input feature map of the CCFN module, h fc1 、h fc2 is the first fully connected layer, the second fully connected layer, h dropo 、h GELU They represent the Dropout layer and activation function, respectively. DWConv represents the depth convolution, f1, They are the process feature map of the intermediate stage and the final output of the CCFN module; H CB Represents a centralized output mechanism.

5. The method for super-resolution reconstruction of a single blurred image according to claim 4, characterized in that: The S22 further includes: S223: Build EATB module; The lightweight CNN-Transformer module EATB consists of 2 LayerNorms, 1 ESA, 1 SimAM, 1 SW-MSA and 1 CCFN; In the EATB module, the input feature map is normalized by the LayerNorm layer. Normalize all features to get the feature map Then the extracted feature map Divided into 2 branches; The first branch uses the traditional convolution route, that is, the feature map As the input of the ESA enhanced spatial attention module of S221 Then the feature map obtained by the ESA module Perform SimAM parameter-free simple attention operation; The second branch uses the Transformer route. When passing through the SW-MSA multi-head self-attention mechanism, the unfold function is used to Converted into input tensor X, the initial shape of input tensor X is (B, C, H, W), where B is the batch size, C is the number of channels, H is the height, and W is the width. Use the window_partition function to divide the input tensor X according to the window size. The shape of each sub-window tensor is (n, w, w', C), the number of sub-windows is n, and the width and height of the sub-window are w and w'; For each sub-window, use the fully connected layer to generate the query Q, key K, value V respectively, and adjust the shape of the Q, K, V tensors, then calculate the attention score attn, get the relative position bias, and adjust the shape and dimension order of the relative position bias, add the relative position bias to the attention score, and then perform softmax processing on the attention score and process it through the dropout layer, then calculate the attention output and process it through the linear layer and dropout layer to get the output tensor of the SW-MSA module; Then, the outputs of the first branch and the second branch are added to obtain the feature map And add the input feature map Then the features processed by LayerNorm layer and CCFN module are normalized and processed by S222. Add together to get the EATB module output The EATB module performs deep extraction according to the following formula: in, It represents the feature map obtained by adding the output of the first branch and the second branch. represents the input feature map of module EATB, H SW-MSA represents the sliding window self-attention mechanism SW-MSA, H SimAM Indicates SimAM operation, H CCFN Indicates CCFN operation, Indicates the output of the EATB module, H LN Represents the LayerNorm layer normalization operation, Represents the input feature map The feature map obtained by LayerNorm layer normalization; Represents the output feature map of the ESA module.

6. The method for super-resolution reconstruction of a single blurred image according to claim 5, characterized in that: The S22 further includes: S224: construct a PCA module; PCA consists of 2 LayerNorms, 1 OCA and 1 CCFN; In the PCA module, the input feature map x is first layer-normalized, and then the OCA cross-overlapping block operation is performed: the shape and size of the input feature map x are adjusted, the query Q, key K, and value V are generated through the fully connected layer, the window_partition function is used to divide Q into small windows, and the unfold operation is used to divide K and V into small windows. The overlapping parts of each sub-window area are processed by the expansion operation; the shapes of Q, K, and V are adjusted and the attention weight attn is calculated, the relative position bias is obtained, the relative position bias is added to the attention score, and the attention weight is restored to the size of the input feature map x through the window_reverse function to obtain the output of the OCA module; this output is added to the input feature map x, and then layer normalization and CCFN are further processed. The output obtained after layer normalization and CCFN processing is added to the output of the OCA module and the input feature map x to obtain the output of the PCA module.

7. The method for super-resolution reconstruction of a single blurred image according to claim 6, characterized in that: The S22 further includes: S225: build deep feature extraction module; The deep feature extraction module consists of four lightweight CNN-Transformer modules LHBCT; each LHBCT contains 3 embedding attention Transformer blocks EATB, 1 pixel cross-window attention PCA and 1 3×3 convolution block; The LHBCT module performs deep feature extraction as a whole, and performs deep extraction according to the following formula: in, Represents the feature map of the input LHBCT module, represents the output of an LHBCT module, f 3×3Conv represents a 3×3 convolution operation, H PCA Indicates that after the PCA module operation, represents EATB operation, k∈{1, 2, 3}, k represents EATB at different levels; The output F of the deep feature extraction module k The calculation formula is: in, represents the LHBCT operation, k∈{1, 2, 3, 4}, k represents different levels of LHBCT, F k represents the output F of the deep feature extraction module k .

8. The method for super-resolution reconstruction of a single blurred image according to claim 2, characterized in that: The S23 includes: Multi-feature fusion MFF includes a 3×3 convolution and a residual structure. In the MFF module, the feature map F extracted by the deep feature extraction module is k After a 3×3 convolution operation, the feature map obtained by the 3×3 convolution is combined with the output feature map of the shallow feature extraction module SFE. Add them together to get the output of the MFF module; The S24 includes: The image reconstruction module IRB includes two 3×3 convolutions and one pixel shuffle upsampling layer. In the image reconstruction IRB module, the output feature map of the MFF module is used as input, first undergoes a 3×3 convolution operation, then an image upsampling operation, and finally a 3×3 convolution operation to obtain the final output F output ; In the multi-feature fusion module MFF and the image reconstruction module IRB, the calculation formulas for the fusion and reconstruction process are as follows: Among them, F output represents the output SR image of the super-resolution reconstruction model, f 3×3Conv represents a 3×3 convolution operation, f PixelShuffle represents the upsampling operation, F k represents the output of the deep feature extraction module, Represents the output feature map of the SFE module.

9. A single blurred image super-resolution reconstruction device, characterized in that: The method according to any one of claims 1 to 8 is used to realize super-resolution reconstruction of a single blurred image.

10. A storage medium for super-resolution reconstruction of a single blurred image, characterized in that: The method according to any one of claims 1 to 8 is used to realize super-resolution reconstruction of a single blurred image.

Images