An image super-resolution reconstruction method based on a lightweight double-branch network
Patent Information
- Application Number
- CN202310475255.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-28
- Publication Date
- 2026-10-20
- Estimated Expiration
- 2043-04-28
AI Technical Summary
当前的SR方法，特别是轻量级的SR方法，面临着一个共同的挑战：由于纹理等高频信息的丢失，重建的超分辨率图像经常遭受模糊和失真
[0032]本发明提出了一种用于SR的轻量级双分支网络(TBNet)，在模型复杂性和性能之间取得更好的平衡。设计了一种新的中间模块FEDAB，用于特征提取。使用反向传播模块BFM和递归Transformer的组合来建立更加有效的模型。提出了一种新的注意力机制(MA)。使用一个额外分支结构来获取更多的特征信息。大量的实验表明，本发明的方法优于目前可用的其他先进的方法，在模型复杂性和性能之间取得了更好的平衡。
Smart Images
Figure CN116664398B_ABST
Abstract
Description
Technical Field
[0001] This invention pertains to single-image super-resolution reconstruction technology, and particularly relates to an image super-resolution reconstruction method based on a lightweight dual-branch network. Background Technology
[0002] Super-resolution reconstruction (SR) of images is a classic task in the field of computer vision, aiming to recover a corresponding high-resolution image with rich details and better visual quality from a low-resolution image.
[0003] Convolutional Neural Network (CNN)-based super-resolution reconstruction methods have significantly improved SR performance compared to traditional methods due to their powerful feature extraction capabilities, dominating current SR research. However, CNN-based SR methods are highly dependent on the size of the CNN, i.e., the network's depth (number of layers) and width (number of channels). Larger SR networks are more expressive and generally exhibit better performance. For example, EDSR has 65 convolutional layers and 43M parameters, while RCAN has over 400 convolutional layers and 16M parameters. Although methods like EDSR and RCAN offer good performance, they require high computational and memory costs and are difficult to apply to resource-constrained devices (such as mobile phones). Lightweight SISR networks (generally considered to have fewer than 1M parameters) are highly desirable. Current SR methods, especially lightweight ones, face a common challenge: reconstructed super-resolution images often suffer from blurring and distortion due to the loss of high-frequency information such as texture.
[0004] The network architecture used in current SR methods mainly includes residual connections, dense connections, and channel attention mechanisms. Popular batch normalization layers and pooling layers have been removed from the network to reduce the loss of feature information and improve the utilization and expressive power of feature information. Summary of the Invention
[0005] To address the aforementioned problems, this invention provides an image super-resolution reconstruction method based on a lightweight dual-branch network.
[0006] The present invention provides an image super-resolution reconstruction method based on a lightweight dual-branch network, comprising the following steps:
[0007] Step 1: Place I LR I SR and I HRThe image is defined as an input low-resolution image (LR), a reconstructed super-resolution image (SR), and a corresponding high-resolution image (HR). At the beginning of the convolutional neural network, two 3x3 convolutional layers are applied for shallow feature extraction.
[0008] F1 = f conv (I LR )
[0009] Among them, f conv (.) represents a convolutional layer used to extract shallow features, and F1 represents the extracted shallow features.
[0010] Step 2: Input the data into the intermediate layer module for local feature extraction.
[0011] F cnn =f CNN，BFM (F1)+F1
[0012] Among them, f CNN，BFM (.) indicates that the feature extraction is performed by the intermediate layer module and the feature is fused by the Backward Fusion Module (BFM). At the same time, the residual is connected to an initial feature to avoid gradient vanishing.
[0013] The intermediate layer module consists of multiple Local Feature Fusion Modules (LEEM). Within the LEEM module, a Feature Extraction Dual-Attention Block (FEDAB) is used. Each FEDAB module uses residual connections to concatenate features. A 1x1 convolutional layer is used before each FEDAB module for dimensionality reduction. Finally, a Multipath Attention (MA) layer is applied.
[0014] The BFM module employs a reverse sequential cascade strategy to fuse features from different receptive fields, representing the output of the i-th FRDAB module as F i The features of each FEDAB module are first convolved with a 1×1 kernel, and then the features downsampled from the 1×1 convolution are concatenated and convolved from right to left.
[0015] Step 3: Input into the recursive Transformer, and then overlay it with a branch to get:
[0016] F RT =f RT (Fcnn )+F branch
[0017] Among them, f RT (.) indicates a recursive Transformer, F bra It is an additional branch. In deeper network layers, the network will lose some high-level and low-level features. The branch is used to retrieve the lost feature information.
[0018] The recursive Transformer consists of two normalized layers, a multi-head attention MHA, and a multi-layer perception MLP.
[0019] Finally, the outputs of the branches are weighted and summed to enhance the expressive power of feature information.
[0020] Step 4: Send the image to the reconstruction module for super-resolution reconstruction.
[0021] I SR =f build (F Rr )+f bicubic (I LR )
[0022] Among them, f build (.) indicates the upsampling module, f bicubic (.) represents the bicubic interpolation function.
[0023] Finally, the reconstructed super-resolution image is obtained.
[0024] Furthermore, the FEDAB module is designed with a multi-branch structure. Features are sent to two branches, and then concatenated with the input, forming a total of three branches, to obtain richer features. After concatenation again, features of different importance are obtained by weighted summation using channel attention and spatial attention. The final features will show stronger suppression of smooth regions in the input image.
[0025] Furthermore, during training, the L1 loss function is used for optimization, given the dataset. The loss function consists of two parts: one part is the L1 loss of the output SR and HR images; the other part is the upsampling of the branch output to obtain SR′, i.e.
[0026]
[0027] Among them, F branc It is the output of the branch, f build (.) represents an upsampling operation;
[0028] The overall loss function is:
[0029]
[0030] Here, α is a hyperparameter. The value of hyperparameter α is set to 0.08.
[0031] Beneficial technical effects of the present invention:
[0032] This invention proposes a lightweight dual-branch network (TBNet) for feature extraction (SR), achieving a better balance between model complexity and performance. A novel intermediate module, FEDAB, is designed for feature extraction. A combination of the backpropagation module BFM and a recursive Transformer is used to build a more efficient model. A novel attention mechanism (MA) is proposed. An additional branch structure is used to capture more feature information. Extensive experiments demonstrate that the method of this invention outperforms other currently available state-of-the-art methods, achieving a better balance between model complexity and performance. Attached Figure Description
[0033] Figure 1 This is a schematic diagram of the overall architecture of the convolutional neural network of this invention.
[0034] Figure 2 This is a schematic diagram of the LEEM structure of the local feature fusion module of the present invention.
[0035] Figure 3 This is a schematic diagram of the FEDAB feature extraction dual attention module of the present invention.
[0036] Figure 4 This is a schematic diagram of the recursive Transformer structure of the present invention.
[0037] Figure 5 The reconstruction results of the image (Mongolian pattern 1) are shown in the example. (a) is the LR image with a downsampling scale of X4, and (b) is the reconstructed SR image.
[0038] Figure 6 The reconstruction results of the image (Mongolian Pattern II) are shown in the example. (a) is the LR image with a downsampling scale of X4, and (b) is the reconstructed SR image. Detailed Implementation
[0039] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.
[0040] With the development of deep convolutional neural networks, the performance of image super-resolution reconstruction (SR) has been significantly improved. However, larger networks offer better performance, but their greater depth and size also require more computation and memory. To address this issue, this invention provides an image super-resolution reconstruction method based on a lightweight dual-branch network. The overall goal is to first downsample the high-resolution image (HR) using bicubic interpolation to generate a low-resolution image (LR), and then input the LR into the network to obtain the super-resolution reconstructed image (SR), ensuring that the generated SR image is as similar as possible to the original HR image. The method of this invention specifically includes the following steps:
[0041] Step 1: Place I LR I SR and I HR The image is defined as the input LR image, the reconstructed SR image, and the corresponding HR image.
[0042] The convolutional neural network structure used in this invention is as follows: Figure 1 As shown, at the beginning of the convolutional neural network, two 3x3 convolutional layers are applied for shallow feature extraction.
[0043] F1 = f conv (I LR )
[0044] Among them, f conv (.) represents a convolutional layer used to extract shallow features, and F1 represents the extracted shallow features.
[0045] Step 2: Input the data into the intermediate layer module for local feature extraction.
[0046] F cnn =f CNN，BFM (F1)+F1
[0047] Among them, f CNN，BFM (.) indicates that the features are extracted by the intermediate layer module and fused by the backward fusion BFM module, while the residual is connected to an initial feature to avoid gradient vanishing.
[0048] The middleware module consists of multiple LEEM modules (such as...) Figure 2 As shown), different levels of features are obtained. In the LEEM module, the FEDAB (Feature Extraction Dual-Attention Block) module is used, such as... Figure 3As shown, it has a stronger feature extraction capability. Residual connections are used between each FEDAB module to concatenate features to obtain richer features. At the same time, in order to keep the number of channels consistent, a 1x1 convolutional layer is used for dimensionality reduction before each FEDAB module. Finally, a multi-path attention (MA) layer is used to make the network focus on more important features.
[0049] The FEDAB module is designed with a multi-branch structure. Features are sent to two branches, and then concatenated with the input, forming a total of three branches, to obtain richer features. After concatenation again, features of different importance are obtained by weighted summation using channel attention and spatial attention. The final features will show stronger suppression of smooth regions in the input image.
[0050] The input is then fed into the backward fusion BFM module. Hierarchical information is crucial for SR; therefore, features from multiple levels are fused to obtain more contextual information. A reverse sequential concatenation strategy is used to fuse features from different receptive fields. The core operation of the feature fusion module is a 1×1 convolution followed by a ReLU activation function (omitted in the figure) to reduce the feature dimension by half. The output of the i-th FEDAB module is represented as F... i ,like Figure 1 As shown, there are four FRDABs. The features of each FRDAB module are first convolved through a 1×1 kernel, which reduces the dimensionality by half. Let F... i (i = 1, 2, 3, 4) represents the obtained features. These features, after being downsampled by 1×1 convolution, are then concatenated and convolved sequentially from right to left in the image. Through this backward sequential connection method, the feature fusion module can integrate rich features, which helps extract more layers of contextual information.
[0051] Step 3: Input into the recursive Transformer, and then overlay it with a branch to get:
[0052] F RT =f RT (F cnn )+F branc
[0053] Among them, f RT (.) denotes a recursive Transformer, which has the function of global information enhancement. branc It is an additional branch. In deeper network layers, the network will lose some high-level and low-level features. The branch is used to retrieve the lost feature information.
[0054] like Figure 4As shown, the recursive Transformer consists of two normalization layers, a multi-head attention (MHA) layer, and a multi-layer perceptron (MLP) layer. This improves network performance without increasing the number of parameters.
[0055] Finally, the outputs of the branches are weighted and summed to enhance the expressive power of feature information. The information obtained by the additional branch structure differs from the high-frequency information obtained by the main structure; it contains some low-frequency and mid-frequency information to increase the network's expressive power. The branch outputs are also regularized to enhance their ability to extract features.
[0056] Step 4: Send the image to the reconstruction module for SR image reconstruction.
[0057] I SR =f build (F RT )+f bicubic (I LR )
[0058] Among them, f build (.) indicates the upsampling module, f bicubic (.) represents the bicubic interpolation function.
[0059] Finally, the reconstructed SR image was obtained.
[0060] During training, the L1 loss function is used for optimization, given the dataset. The loss function consists of two parts: one part is the L1 loss of the output SR image and HR image; the other part is the upsampling of the branch output to obtain SR', i.e.
[0061]
[0062] Among them, F bra It is the output of the branch, f build (.) represents an upsampling operation;
[0063] The overall loss function is:
[0064]
[0065] Here, α is a hyperparameter. The value of hyperparameter α is set to 0.08.
[0066] Example:
[0067] The training set was div2k, and the test sets included set5, set14, etc. To obtain the LR images, bicubic interpolation was used to downsample the HR images, and the training data was augmented through random rotation and horizontal flipping. During training, 48×48 patches were randomly sampled from the training set as input. The initial learning rate was 2×10⁻⁶.-4 After cosine annealing, the final value decreased to 6.25 × 10⁻⁶. -6 The model was trained using the Adam optimizer. In the final model, each part had 32 input and output channels, using six (n=6) LFFMs, and the transformer module recursively executed once (S=1). The network had 0.78M parameters. The performance of TBNet and other lightweight SISR models at x4 was compared, as shown in Table 1. It can be seen that most of them achieved state-of-the-art results on lightweight SISR tasks.
[0068] Table 1 Model Performance Comparison
[0069]
[0070]
[0071] The reconstruction results of the two Mongolian patterns are as follows: Figure 5 and Figure 6 As shown, the reconstructed image has significantly improved resolution and sharpness compared to the original low-resolution image, resulting in a marked improvement in image quality.
Claims
1. A method for image super-resolution reconstruction based on a lightweight dual-branch network, characterized in that, Includes the following steps: Step 1: Put , and The image is defined as the input low-resolution image, the reconstructed image, and the corresponding high-resolution image; at the beginning of the convolutional neural network, two 3x3 convolutional layers are applied for shallow feature extraction; ； in, This represents a convolutional layer used to extract shallow features. This represents the extracted shallow features; Step 2: Apply shallow features The input is fed into the intermediate layer module for local feature extraction; + ； in, This indicates that the features are extracted by the intermediate layer module and fused by the reverse fusion module BFM. At the same time, a shallow feature is connected by a residual connection to avoid gradient vanishing. The intermediate layer module is a multi-local feature fusion module LEEM. In the LEEM module, the feature extraction dual attention module FEDAB is used. The features are concatenated with each FEDAB module by residual connection. A 1x1 convolutional layer is used for dimensionality reduction before each FEDAB module. Finally, it is passed through a multi-path attention MA, added to the FEDAB module input, and then output. The FEDAB module is designed with a multi-branch structure. The input features are sent to two branches containing different convolutional modules, and then together with the input, the three branches are concatenated. The three concatenated branches are added to the outputs of the two branches containing different convolutional modules, and then concatenated again. Channel attention and spatial attention are used to obtain features of different importance and perform weighted summation. The BFM module employs a reverse sequential cascade strategy to fuse features from different receptive fields, and the output of the i-th FEDAB module is represented as F. i The features of each FEDAB module are first convolved with a 1 × 1 kernel, and then the features downsampled from the 1 × 1 convolution are concatenated and convolved from right to left. Step 3: Put The input is recursively processed by the Transformer, with an additional branch. Superimposing them, we get: ； in, Indicates a recursive Transformer. The branch reduces the dimensionality of multiple convolutional layers and then passes it through a multi-path attention MA; in deeper network layers, the network will lose some high-level and low-level features, and the branch is used to retrieve the lost feature information; The recursive Transformer consists of two normalization layers, a multi-head attention MHA, and a multi-layer perception MLP. Step 4: The image is sent to the reconstruction module for super-resolution reconstruction. ； in, Indicates the upsampling module. This represents a bicubic interpolation function; Finally, the reconstructed super-resolution image is obtained. .
2. The image super-resolution reconstruction method based on a lightweight dual-branch network according to claim 1, characterized in that, During training, the L1 loss function is used for optimization, given the dataset. The loss function consists of two parts: one part is the L1 loss of the output super-resolution reconstructed image and the high-resolution image; the other part is the super-resolution reconstructed image of the additional branch output. and high-resolution images The L1 loss is: ； in, It is the output of the branch. It is an upsampling operation; The overall loss function is: ； in, It is a hyperparameter. For the reconstruction of super-resolution images.
3. The image super-resolution reconstruction method based on a lightweight dual-branch network according to claim 2, characterized in that, The hyperparameters The size is set to 0.08.