Lightweight single image super-resolution reconstruction method combining convolutional neural network and transformer

Abstract

The application provides a light single-image super-resolution reconstruction method combining a convolutional neural network and a Transformer, and belongs to the fields of computer vision and image processing. First, original data in a training set of image super-resolution reconstruction is preprocessed, and paired LR-HR is constructed as a final training set. Second, the preprocessed training set is input into a light single-image super-resolution reconstruction network proposed by the application, L1 is used as a training loss for training, and thus a final image super-resolution reconstruction model is obtained. Finally, a model is loaded, a picture to be super-resolved is input into the trained image super-resolution reconstruction network model, and a high-resolution image corresponding to the low-resolution image is obtained. The application integrates the advantages of the convolutional neural network and the Transformer, achieves a good balance between the size and performance of the model, and has a significant improvement in image objective evaluation indexes.

Description

Technical Field

[0001] This invention relates to the fields of computer vision technology and image processing, specifically to a lightweight single-image super-resolution reconstruction method that combines convolutional neural networks with Transformer. Background Technology

[0002] Image super-resolution reconstruction refers to the process of restoring a given low-resolution image to its corresponding high-resolution image using a specific algorithm. Image resolution is a set of performance parameters used to evaluate the richness of detail contained in an image; high-resolution images often contain richer texture details and have higher reliability. It has wide applications in real life, such as super-resolution reconstruction of video surveillance images, which can better reconstruct people and objects in the video surveillance, thus accurately locating the desired target. However, while previous methods have achieved satisfactory results, the increased model size and computational complexity have limited their application in the real world. Therefore, achieving a balance between model size and performance in image super-resolution reconstruction has become an important and challenging task.

[0003] Currently, image super-resolution reconstruction methods are mainly divided into three types: interpolation-based methods, reconstruction-based methods, and learning-based methods. Early reconstruction methods mostly employed interpolation-based methods, the most common being bilinear interpolation and bicubic interpolation. These methods are computationally simple, easy to understand, and easy to implement, but the final reconstruction results often suffer from ringing artifacts and severe loss of high-frequency information. Subsequently, many reconstruction-based methods emerged. These methods start from image degradation models and combine prior knowledge of the image to constrain the generation of super-resolution images. Learning-based methods, on the other hand, acquire prior knowledge through a learning process to guide the generation of high-resolution images.

[0004] With the rise of deep learning, many deep learning-based image super-resolution reconstruction algorithms have emerged. One type primarily improves the reconstruction performance by continuously increasing the network depth and width, thereby producing high-quality reconstruction results, such as EDSR, RCAN, and SAN. While these methods can achieve high PSNR, the number of network parameters and computational cost also increase significantly, limiting their application in the real world. Another type focuses on lightweight image super-resolution reconstruction networks, such as IMDN, RFDN, SMSR, and LAPAR. These methods can recover some image details with a lower number of parameters, but they cannot achieve a good balance between model size and performance. These lightweight super-resolution reconstruction methods are often pure convolutional neural network models, meaning that these networks can only extract local features, which is not conducive to the recovery of image texture details. Summary of the Invention

[0005] To address the aforementioned problems in existing technologies, this invention proposes a lightweight single-image super-resolution reconstruction method that combines convolutional neural networks with Transformers. This method can recover more detailed features with lower parameter and computational requirements, alleviating the problem that existing lightweight image super-resolution reconstruction methods cannot achieve a balance between model performance and size.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] A lightweight single-image super-resolution reconstruction method combining convolutional neural networks and Transformers includes the following steps:

[0008] Step 1: Preprocess the raw data in the single-image super-resolution reconstruction training dataset to construct paired low-resolution image-high-resolution image training data;

[0009] Step 2: Construct a lightweight single-image super-resolution reconstruction network that combines convolutional neural networks and Transformers, and input the pre-processed training data into the network;

[0010] Step 3: Use L1 as the training loss function to train and obtain the final trained super-resolution reconstruction network model;

[0011] Step 4: Load the above model and input the image to be super-resolved into the super-resolution reconstruction network model to obtain the high-resolution image corresponding to the low-resolution image.

[0012] Furthermore, step 1 specifically includes:

[0013] Step 1.1: Crop the high-resolution (HR) images in the original dataset, cropping several 128×128 sub-images from each image to expand the dataset;

[0014] Step 1.2: Use the bicubic kernel function in MATLAB to downsample the high-resolution image by 4 times to obtain a low-resolution (LR) image. Use the LR-HR pair as the training set for the super-resolution reconstruction network.

[0015] Furthermore, step 2 specifically includes:

[0016] Step 2.1: First, put I LR The input is fed into the Asymmetric Convolution Block (ACB) for shallow feature extraction, as shown in the following formula:

[0017] F ACB =f ACB (I LR (1)

[0018] Where F ACB This represents the shallow features extracted by the ACB module; f ACB (·) represents the function of the ACB module, I LR This represents the input low-resolution image.

[0019] Furthermore, step 2.1 specifically includes:

[0020] Most previous single-image super-resolution reconstruction networks used a single 3×3 standard convolution as the first layer to extract shallow features. However, this method extracts features with a limited scale and insufficient feature content. This invention employs an ACB module in the first layer of the network (network structure as follows). Figure 2 (As shown) This involves using 3×3, 1×3, and 3×1 convolutional kernels to extract shallow features from the input image. Compared to using only a single 3×3 convolutional kernel for feature extraction, the ACB module can also improve the model's robustness to image flipping and rotation. Finally, the calculation results of the above three convolutional layers are fused to obtain the output of the final convolutional layer. The complete operation definition of the ACB module is shown below:

[0021] F ACB =f 3×3 (I LR )+f 1×3 (I LR )+f 3×1 (I LR (2)

[0022] Among them, f 3×3 (·), f 1×3 (·) and f 3×1 (·) represent convolutional functions with kernel sizes of 3×3, 1×3, and 3×1, respectively.

[0023] Step 2.2: Input the result obtained from the shallow feature extraction in Step 2.1 into the Multi-scale Feature Fusion CNN (MFF-CNN) of this invention (network structure as follows). Figure 3 As shown in the figure, its calculation formula is as follows:

[0024] F MFF-CNN =f MFF-CNN (F ACB (3)

[0025] Among them, F MFF-CNN This represents the local features extracted by the MFF-CNN network, f MFF-CNN (·) represents the function of the MFF-CNN network.

[0026] Furthermore, step 2.2 specifically includes:

[0027] Step 2.2.1: MFF-CNN mainly consists of 14 multiscale feature fusion modules (MFFM), which combine the extracted features... ACB The data fed into the MFFM module will first enter the Mobile Convolution module (MBConv). MBConv uses 1×1 convolutions and 3×3 deconvolutions for feature extraction, and its specific calculation is shown below:

[0028] F MBConv =f 1×1 (f SCA (f d3×3 (f 1×1 (LN(F ACB )))))+F ACB (4)

[0029]

[0030] Where F MBConv This represents the local features extracted by the MBConv module, where LN stands for LayerNorm normalization method, and f 1×1 (·) represents the convolution function with a kernel size of 1×1, fd3×3 (·) represents a 3×3 depthwise convolution, f SCA (·) represents the function of the efficient attention SCA module. The SCA module is based on the channel attention SENet, but removes redundant convolutional layers and activation functions. In order to have a certain computational complexity, the feature dimension of SCA is adjusted to 4 times the original.

[0031] Step 2.2.2: Convert the above result F MBConv The data is fed into a Multiscale Feedforward Network (MFFN). The MFFN uses LN normalization and two 1×1 convolutions for feature extraction to obtain the output F1 score, as shown in the following formula:

[0032] F1 = f 1×1 (f 1×1 (LN(F MBConv (6)

[0033] Step 2.2.3: Next, set F1 and F... MBConv The signal is fed into the Multi-Scale Channel Attention Module (MS-CAM) to obtain the output F2, the specific formula of which is shown below:

[0034] F2=f MS-CAM (F1+F MBConv (7)

[0035]

[0036] Y = F1 + F MBConv (9)

[0037] Where f MS-CAM (·) represents the function of the MS-CAM module. This represents the dot product operation; Y represents the product of F1 and F2. MBConv The sum of the two.

[0038] Step 2.2.4: Then, add the results F1 and F2 obtained above to obtain the final output F. MFFM The formula is as follows:

[0039]

[0040] Step 2.2.5: Finally, concatenate all the outputs input into the MFF-CNN network using Concat to obtain the final output result F. MFF-CNN The formula is shown below:

[0041]

[0042]

[0043]

[0044] Here, This represents the output of the i-th MFFM module. This represents the function of the i-th MFFM module, where n represents the number of MFFM modules, F MFF-CNN This represents the output of the MFF-CNN network, and Concat represents the concatenation operation along the channels.

[0045] Step 2.3: Extract the features F from the convolutional neural network that has undergone multi-scale feature fusion and attention. MFF-CNN The formula for feeding into the multi-attention module is as follows:

[0046] F MAB =f 3×3 (f MAB (f 3×3 (F MFF-CNN (14)

[0047] Among them, F MAB f represents the features learned by the MAB module. MAB (·) represents the function of the MAB module.

[0048] Furthermore, step 2.3 specifically includes:

[0049] Step 2.3.1: F MFF-CNN First, the input is fed into a module containing channel attention and a Transformer to obtain the output result F. mid The formula is as follows:

[0050] F mid =f MDTA (LN(f 3×3 (F MFF-CNN )))+f SCA (LN(f 3×3 (F MFF-CNN )))+f 3×3 (F MFF-CNN (15)

[0051] Among them, f MDTA (·) represents the function of the MDTA module in the original Restormer network.

[0052] Step 2.3.2: Apply the above result F mid The data is fed into the GDFN module to obtain the output result F. out Finally, the result is fed into a 3×3 convolution to obtain the final output F. MAB The calculation formula is as follows:

[0053] F out =F mid +f GDFN (LN(F mid (16)

[0054] F MAB =f 3×3 (F out (17)

[0055] Among them, f GDFN (·) represents the function of the GDFN module in the original Restormer network.

[0056] Step 2.4: Output the above result F MAB and F ACB The input is fed into the final upsampling and reconstruction module to obtain the final reconstructed high-resolution image. The training model and evaluation metrics are saved, and the output can be expressed by the following formula:

[0057] I SR =f Sub-pixel (F MAB +F ACB (18)

[0058] Among them, I SR This represents the high-resolution image reconstructed by the final network.

[0059] Furthermore, step 3 specifically includes:

[0060] Throughout the entire network training process, this invention uses only L1 as the loss function. Specifically, for a given pair of training data... The L1 loss function can be expressed as follows:

[0061]

[0062] in, This refers to the parameter settings of the network proposed in this invention, F(I) LR ) = I SR This represents the final reconstructed image, and N represents the number of images used for training.

[0063] Furthermore, step 4 specifically includes:

[0064] Load the model_best trained in step 3, input the low-resolution image into the model, and output the reconstructed SR image and the corresponding evaluation metrics.

[0065] The beneficial effects of this invention are as follows: This invention integrates the characteristics of convolutional neural networks and Transformers, effectively extracting both local and global information from images. Furthermore, by introducing channel attention into the Transformer, it activates more pixels, enabling the extraction of more features while maintaining a low computational load. Compared to previous lightweight image super-resolution reconstruction methods, this invention can reconstruct more texture details with lower parameter and computational requirements. Attached Figure Description

[0066] Figure 1 A complete architecture diagram of a lightweight image super-resolution reconstruction network (MMSR) that combines convolutional neural networks with Transformers;

[0067] Figure 2 Here is a detailed structural diagram of the ACB module;

[0068] Figure 3 Here is a structural diagram of the MFFM module;

[0069] Figure 4 This is a diagram of the MAB structure. Detailed Implementation

[0070] The embodiments of the present invention are implemented under the premise of the technical solution of the present invention, and detailed implementation methods and specific operation processes are given. However, the protection scope of the present invention is not limited to the following embodiments.

[0071] The present invention will be described in detail below with reference to embodiments, so that those skilled in the art can implement it after referring to this specification.

[0072] This embodiment uses Windows as the development environment, PyCharm as the development platform, and Python as the development language. It employs the lightweight single-image super-resolution reconstruction method of this invention, which combines convolutional neural networks with Transformers, to reconstruct low-resolution images into high-resolution images.

[0073] In this embodiment, a lightweight single-image super-resolution reconstruction method combining convolutional neural networks and Transformers includes the following steps:

[0074] Step 1: Preprocess the raw data in the single-image super-resolution reconstruction training dataset to construct paired low-resolution image-high-resolution image training data;

[0075] Step 2: Construct a lightweight single-image super-resolution reconstruction network that combines convolutional neural networks and Transformers, and input the pre-processed training data into the network (e.g., Figure 1 (as shown);

[0076] Step 3: Use L1 as the training loss function to train and obtain the final trained super-resolution reconstruction network model;

[0077] Step 4: Using the image requiring super-resolution as input, load the model trained and saved in Step 3 to obtain the high-resolution image corresponding to the low-resolution image and the corresponding evaluation metrics. Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) are used as objective evaluation metrics, while the number of parameters is used to measure model size. The formula for calculating PSNR is shown below:

[0078]

[0079] Here The maximum value of the color of an image point is represented by MSE, which stands for mean square error.

[0080] Based on the above steps, this invention proposes two models of different sizes (MMSR and MMSR_L), and compares them with MADNet, AWSRNM, LAPAR-A, SMSR, LBNet, and ESRT models at magnification ratios of ×2, ×3, and ×4 on five different test sets (Set5, Set14, BSD100, Urban100, and Manga109). As can be seen from Tables 1, 2, and 3, the method proposed in this invention can achieve superior PSNR values ​​with a reduced number of parameters on five common test sets.

[0081] Table 1 shows a quantitative comparison with state-of-the-art SISR models on ×2.

[0082]

[0083] Note: The best and second-best results are highlighted and underlined, respectively.

[0084] Table 2 shows a quantitative comparison with state-of-the-art SISR models on a ×3 scale.

[0085]

[0086] Note: The best and second-best results are highlighted and underlined, respectively.

[0087] Table 3 shows a quantitative comparison with state-of-the-art SISR models on a ×4 scale.

[0088]

[0089] Note: The best and second-best results are highlighted and underlined, respectively.

[0090] The foregoing description of specific exemplary embodiments of the invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A lightweight single-image super-resolution reconstruction method combining convolutional neural networks and Transformer, characterized in that, The method includes the following steps: Step 1: Preprocess the raw data in the single-image super-resolution reconstruction training dataset to construct paired low-resolution image-high-resolution image training data; Step 2: Construct a lightweight single-image super-resolution reconstruction network that combines convolutional neural networks and Transformers, and input the pre-processed training data into the network; Step 2 specifically includes: Step 2.1: First, The input is fed into the asymmetric convolutional module (ACB) for shallow feature extraction, as shown in the following formula: (1) in This represents the shallow features extracted by the ACB module; This refers to the function of the ACB module. This represents the input low-resolution image; Step 2.2: Input the result obtained from shallow feature extraction in Step 2.1 into the MFF-CNN convolutional neural network based on multi-scale feature fusion attention. The calculation formula is shown below: (2) in, This represents the local features extracted by the MFF-CNN network. This represents the function of the MFF-CNN network; Step 2.2 specifically includes: Step 2.2.1: MFF-CNN consists of 14 multi-scale feature fusion modules (MFFM), which will extract the features... The data fed into the MFFM module will first enter the moving convolution module MBConv. MBConv uses 1×1 convolutions and 3×3 deconvolutions for feature extraction, and its specific calculation is shown below: (6) (7) in This represents the local features extracted by the MBConv module. This refers to LayerNorm normalization technology. This represents a convolution function with a kernel size of 1×1. This represents a 3×3 depthwise convolution. This represents the function of the efficient attention SCA module. The SCA module is based on the channel attention SENet, but removes redundant convolutional layers and activation functions. In order to achieve a certain level of computational complexity, the feature dimension of SCA is adjusted to 4 times the original. Step 2.2.2: The results obtained above... The data is fed into a multi-scale feedforward network (MFFN), where LN normalization and two 1×1 convolutions are used for feature extraction to obtain the output. The formula is as follows: (8) Step 2.2.3: Next, and The data is fed into the multi-scale channel attention module MS-CAM to obtain the output results. The specific formula is as follows: (9) (10) (11) in This represents the function of the MS-CAM module. This represents the dot product operation. It means and The sum of the two; Step 2.2.4: Then, take the results obtained above. and Add them together to get the final output. The formula is as follows: (12) Step 2.2.5: Finally, all outputs input into the MFF-CNN network are concatenated using Contact to obtain the final output result. The formula is shown below: (13) (14) (15) Here, Indicates the first The output of each MFFM module, It indicates the first The functionalities of each MFFM module This indicates the number of MFFM modules. This represents the output of the MFF-CNN network. This indicates a series operation along the channel; Step 2.3: Extract the features from the network that has undergone multi-scale feature fusion and attention. The formula for feeding into the multi-attention module is as follows: (3) in, This represents the features learned by the MAB module. This represents the function of the MAB module; Step 2.4: Obtain the output results as well as The input is fed into the final upsampling and reconstruction module to obtain the final reconstructed high-resolution image. The training model and evaluation metrics are saved, and the output is expressed by the following formula: (4) in, This represents the high-resolution image reconstructed by the final network; Step 3: Use L1 as the training loss function to train and obtain the final trained super-resolution reconstruction network model; Step 4: Load the model obtained in Step 3, input the image to be super-resolved into the super-resolution reconstruction network model, and obtain the high-resolution image corresponding to the low-resolution image.

2. The method according to claim 1, characterized in that, Step 1 specifically includes: Step 1.1: Crop the high-resolution images (HR) in the original dataset, cropping several 128×128 sub-images from each image to expand the dataset; Step 1.2: Use the bicubic kernel function in MATLAB to downsample the high-resolution image by 4 times to obtain the low-resolution image LR. Use the paired LR-HR as the training set for the super-resolution reconstruction network.

3. The method according to claim 1, characterized in that, Step 2.1 specifically includes: The first layer of the network employs the ACB module, which uses 3×3, 1×3, and 3×1 convolutional kernels to extract shallow features from the input image. Finally, the calculation results of the above three convolutional layers are fused to obtain the output of the final convolutional layer. The complete operation definition of the ACB module is shown below: (5) in, , and These represent convolutional function types with kernel sizes of 3×3, 1×3, and 3×1, respectively.

4. The method according to claim 1, characterized in that, Step 2.3 specifically includes: Step 2.3.1: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation First, the data is fed into a module containing channel attention and a Transformer to obtain the output. The formula is as follows: (16) in, This represents the functionalities of the MDTA module in the original Restormer network; Step 2.3.2: Next, take the results obtained above... The data is fed into the GDFN module to obtain the output results. Finally, the result is fed into a 3×3 convolution to obtain the final output. The calculation formula is as follows: (17) (18) in, These represent the functionalities of the GDFN module in the original Restormer network.

5. The method according to any one of claims 1-4, characterized in that, Step 3 specifically includes: Given a pair of training data The L1 loss function is expressed as follows: (19) in, This refers to the parameter settings of the proposed network. This represents the final reconstructed image. This indicates the number of images used for training.

6. The method according to any one of claims 1-4, characterized in that, Step 4 specifically includes: Load the model_best trained in step 3, input the low-resolution image into the model, and output the reconstructed SR image and the corresponding evaluation metrics.

Images