Lightweight super-resolution method based on structural reparameterization and linear large kernel attention

Abstract

The lightweight super-resolution method based on structure reparameterization and linear large kernel attention belongs to the technical field of image generation, in order to solve the problem that the existing lightweight network is difficult to balance the reconstruction accuracy and inference speed, the method constructs a shallow module based on a low-resolution image to convert to obtain a shallow feature; a deep extraction group is constructed based on the feature, and a deep feature is output through cascade block operation; a residual module is constructed to fuse the deep and shallow features; a reconstruction module is constructed to generate a super-resolution image; a loss module is constructed to obtain a hybrid loss; a model is constructed and trained; the final result is output by inputting the image to be processed, the present application can effectively perform super-resolution reconstruction on a low-resolution image, avoid the problem of insufficient long-range dependence modeling caused by the limited local receptive field of the traditional convolutional neural network, and the problem of the square level growth of the standard Transformer calculation overhead with the resolution.

Description

Technical Field

[0001] This invention relates to the field of image generation technology, specifically a lightweight super-resolution method based on structural reparameterization and linear big kernel attention. Background Technology

[0002] Image super-resolution reconstruction aims to restore low-resolution images to high-resolution images and is widely used in scenarios such as surveillance video enhancement, medical image processing, and mobile image optimization. Existing technologies are mainly divided into two categories: methods based on convolutional neural networks and methods based on Transformers.

[0003] Convolutional neural network-based methods have strong local feature extraction capabilities and translation invariance, but they are limited by the local receptive field, making it difficult to effectively model global context information and prone to blurring or artifacts in complex texture areas. Transformer-based methods achieve global modeling through self-attention mechanisms, but the computational complexity of standard self-attention increases quadratically with input resolution, resulting in extremely high computational overhead and memory usage, making it impossible to deploy in real time on resource-constrained platforms such as mobile devices and embedded devices.

[0004] While existing lightweight networks achieve a certain balance between performance and efficiency by reducing channels and depth, they still suffer from problems such as low operator execution efficiency and discontinuous memory access caused by multi-branch fusion in extreme scenarios such as ultra-high frame rate video streams and non-general-purpose computing platforms, making it difficult to balance reconstruction accuracy and inference speed.

[0005] To address the above issues, a lightweight super-resolution method based on structural reparameterization and linear big kernel attention is proposed. Summary of the Invention

[0006] The purpose of this invention is to provide a lightweight super-resolution method based on structural reparameterization and linear big kernel attention. By using this invention, the problem that existing lightweight networks cannot balance reconstruction accuracy and inference speed in the above-mentioned context is solved.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a lightweight super-resolution method based on structure reparameterization and linear big kernel attention, comprising the following steps: S1: Preprocess the image to obtain an image pair consisting of a high-resolution image and a low-resolution image; S2: Based on low-resolution images, a shallow feature extraction module is constructed; and spatial mapping and nonlinear transformation are performed on the low-resolution images to convert pixel-level information into high-dimensional feature representations, i.e., shallow features. S3: Based on high-dimensional feature representation, a deep reparameterized feature extraction group is constructed. This extraction group consists of N cascaded reparameterized large kernel attention blocks. Each reparameterized large kernel attention block sequentially performs structural reparameterized convolution, linear large kernel attention calibration and residual connection. The decomposition characteristics of depthwise separable dilated convolution are used to capture cross-region similarity and output deep features. S4: Based on shallow and deep features, a global residual connection module is constructed; it is used to sum the shallow and deep features element-wise and force the model to learn the residual mapping between low-resolution and high-resolution images to obtain fused features. S5: Based on the fusion features, a high-fidelity image reconstruction module is constructed. After smoothing and refining the fusion features, the feature map is recombined in the spatial dimension through sub-pixel convolution to generate a super-resolution image of the target multiple. S6: Based on super-resolution images and high-resolution images, a hybrid loss function module is constructed, which combines pixel robustness loss and edge structure constraint loss to obtain the hybrid loss; S7: Construct a lightweight super-resolution reconstruction model based on structure-re-parameterized convolution and linear large kernel attention, train the model using the hybrid loss, and update the model parameters; S8: Input the low-resolution image to be processed into the trained lightweight super-resolution reconstruction model, and output the final super-resolution image.

[0008] Furthermore, the shallow feature extraction module uses 3×3 standard convolution for spatial mapping and completes feature transformation through a nonlinear activation function with learnable parameters to obtain a shallow feature map.

[0009] Furthermore, the structure-reparameterized convolution adopts a multi-branch parallel topology during the training phase, including depth branches, convolution branches of different sizes, and identity branches, to enhance feature representation capabilities. During the inference phase, multiple branches are equivalently collapsed into a single standard convolution operator, eliminating redundant inference overhead.

[0010] Furthermore, the multi-branch parallel topology includes a serialization depth branch, a 3×3 convolution branch, a 1×3 convolution branch, a 3×1 convolution branch, a 1×1 convolution branch, and an identity branch, with each convolution branch connected to a batch normalization layer.

[0011] Furthermore, the linear large kernel attention block sequentially employs a 5×5 depthwise convolution, a 7×7 depthwise dilated convolution with an expansion rate of 3, and a 1×1 convolution to generate a global attention weight matrix, which is then element-wise multiplied with the deep input feature map fed into this attention block to achieve dynamic feature calibration.

[0012] Furthermore, each of the reparameterized large kernel attention blocks employs residual connections to add the structurally reparameterized convolution output features to the attention-calibrated features, thereby obtaining the current block output features.

[0013] Furthermore, the global residual connection directly sums the shallow and deep features element-wise, allowing the model to focus on learning the residuals between high and low resolution images, preserving low-frequency structural information and reducing training difficulty.

[0014] Furthermore, the high-fidelity image reconstruction module uses 3×3 refined convolution for feature smoothing, and then uses sub-pixel convolution layers to achieve upsampling by a specified factor to generate a high-fidelity super-resolution image.

[0015] Furthermore, the pixel robustness loss employs the Charbonnier loss function to maintain pixel-level fidelity between the super-resolution image and the true high-resolution image; the edge structure constraint loss is calculated using the difference between horizontal and vertical gradients.

[0016] Furthermore, the lightweight super-resolution reconstruction model completes model training by iteratively updating model parameters by minimizing the hybrid loss function.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention can effectively perform super-resolution reconstruction of low-resolution images, avoiding the problems of insufficient long-range dependency modeling caused by the limited local receptive field of traditional convolutional neural networks and the problem of the computational overhead of standard Transformer increasing quadratically with resolution. This invention uses structural reparameterization technology to effectively collapse the multi-branch feature enhancement capability in the training phase into a single-path standard convolution operator in the inference phase. It also combines a linear large kernel attention mechanism with the decomposition and combination of depthwise separable convolution and dilated convolution to achieve near-linear computational complexity. This significantly reduces the number of parameters and computational cost of the algorithm, resulting in lightweight characteristics while ensuring reconstruction performance. It is suitable for real-time deployment on mobile and embedded devices. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the super-resolution network structure of the present invention; Figure 2 This is a schematic diagram of the Rep-Conv structural reparameterization unit of the present invention; Figure 3 This is a schematic diagram of the linear large kernel attention mechanism (LKA) structure of the present invention; Figure 4 This is a schematic diagram of the method steps of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Example: Please refer to Figures 1-4 A lightweight super-resolution method based on structure reparameterization and linear big-kernel attention includes the following steps: S1: Preprocess the image to obtain image pairs consisting of a high-resolution image and a low-resolution image, as follows: Collect publicly available datasets or real-world scene images, and generate corresponding low-resolution images through preprocessing operations such as downsampling and noise addition. Construct training image pairs that correspond one-to-one between high-resolution and low-resolution images to provide a data foundation for model training.

[0021] S2: Based on low-resolution images, a shallow feature extraction module is constructed; and spatial mapping and nonlinear transformation are performed on the low-resolution images to convert pixel-level information into high-dimensional feature representations, i.e., shallow features. The shallow feature extraction module uses 3×3 standard convolution for spatial mapping and completes feature transformation through a non-linear activation function with learnable parameters to obtain a shallow feature map, as follows: Construct a shallow feature extraction module to extract the input low-resolution image. The conversion to high-dimensional shallow features is achieved using the following formula: ; ; in: Indicates the activation parameter; It is a learnable constant; These are the convolution weights; These are bias parameters; The superscript indicates the current floor number; The subscript indicates the size of the convolution kernel; In the superscript, the first parameter indicates the layer number of the current weight, and the second parameter indicates the position of the current weight in the convolution of the current layer. This is a convolution operation; To output a shallow feature map.

[0022] Processing procedure: Low-resolution image Spatial mapping is achieved through a standard 3×3 convolution, followed by activation function. Perform a nonlinear transformation to output a shallow feature map. This enables dimensional mapping from pixel space to feature space, providing a foundation for deep feature extraction.

[0023] S3: As Figures 1-3 As shown, based on high-dimensional feature representation, a deep reparameterized feature extraction group is constructed. This extraction group consists of N cascaded reparameterized large kernel attention blocks. Each reparameterized large kernel attention block sequentially performs structural reparameterized convolution, linear large kernel attention calibration, and residual connection. The decomposition characteristics of depthwise separable dilated convolution are used to capture cross-region similarity and output deep features. The structure-reparameterized convolution employs a multi-branch parallel topology during the training phase, including depth branches, convolution branches of different sizes, and identity branches, to enhance feature representation capabilities. During the inference phase, the multi-branch convolution is equivalently collapsed into a single standard convolution operator to eliminate redundant inference overhead. The multi-branch parallel topology includes a serialization depth branch, a 3×3 convolution branch, a 1×3 convolution branch, a 3×1 convolution branch, a 1×1 convolution branch, and an identity branch. Each convolution branch is followed by a batch normalization layer. Attention weights are generated by combining depthwise separable dilated convolutions. The specific execution process is as follows: a 5×5 depthwise convolution is used to capture local details, a 7×7 depthwise dilated convolution with a dilation rate of 3 is used to model long-range dependencies, and a 1×1 convolution is used to realize channel interaction. Finally, an attention weight matrix is ​​generated by using a sigmoid activation function and element-wise multiplication is performed with the input feature map to achieve dynamic feature calibration.

[0024] Each reparameterized large kernel attention block uses residual connections to add the structurally reparameterized convolution output features to the attention-calibrated features, thus obtaining the current block's output features.

[0025] Specifically: A deep reparameterized feature extraction group is constructed, consisting of N cascaded reparameterized large kernel attention blocks. Each block sequentially performs structural reparameterized convolution, linear large kernel attention calibration, and residual connection, outputting deep features. The deep reparameterized feature extraction group is performed based on the following formula: ; ; ; ; ; ; ; in, Represents a structure-reparameterized convolutional unit; This represents a linear large kernel attention module; This is an element-wise multiplication operation; express Input feature map of a heavily parameterized large kernel attention block; For the first Local structural features extracted from each block by a structure reparameterized convolutional unit; This represents the structural enhancement features after dynamic calibration using a linear large kernel attention mechanism; Indicates the first Output feature map of a heavily parameterized large kernel attention block; This indicates a deep branch of serialization, with non-linear depth enhanced by multi-level mapping; Indicates the kernel size as Convolution operations; This represents the batch normalized layer following the corresponding convolutional branch; Indicates the input feature map; This represents the characteristic response during the training phase; This represents the equivalent convolution weights after fusion; Indicates the bias after fusion; This represents a 5×5 depthwise convolution, responsible for local spatial detail perception; This represents a 7×7 depthwise dilated convolution with an expansion rate of 3, responsible for long-range dependency modeling; This represents a 1×1 convolution, which performs information exchange and linear mapping in the channel dimension; This represents the Sigmoid activation function, used to generate the attention weight matrix. .

[0026] The deep reparameterization feature extraction process is as follows: Input shallow features First, the input enters the first reparameterized large kernel attention block, followed by a structure-reparameterized convolutional unit. Local feature mining is performed to obtain feature maps. ; feature map Feed into the linear large kernel attention module Generate a global attention weight matrix and apply it to... Perform element-wise multiplication to obtain structural enhancement features. ; Use residual connections to integrate input features With structural enhancement features Add them together to obtain the output feature map of the first reparameterized large kernel attention block. ; Use the second to the third in sequence Each heavily parameterized large kernel attention block performs the above feature extraction and attention calibration operations on the output feature map of the previous block to obtain deep features. ; In this process, each heavily parameterized large kernel attention block adopts a five-branch parallel topology to enhance representation capabilities during the training phase. During the inference phase, the multi-branch structure is collapsed into a single standard convolution operator through mathematical equivalent transformation, thereby improving feature extraction accuracy without increasing redundant overhead at the inference end.

[0027] S4: Based on shallow and deep features, a global residual connection module is constructed; it is used to sum the shallow and deep features element-wise and force the model to learn the residual mapping between low-resolution and high-resolution images to obtain fused features. Specifically: Global residual connections directly sum the shallow and deep features element-wise, allowing the model to focus on learning the residuals between high- and low-resolution images, preserving low-frequency structural information and reducing training difficulty; shallow and deep features are fused, as shown in the following formula: ; in: This represents the fused feature map after global residual connections; Indicating the first in the deep reparameterized feature extraction group Deep feature maps output by heavily parameterized large kernel attention blocks; This represents the shallow feature map output by the shallow feature extraction module.

[0028] Processing procedure: Deep feature map With shallow feature map Direct element-wise summation forces the model to learn only the residual mapping between high- and low-resolution images, preserving low-frequency structural information, reducing training difficulty, and outputting a fused feature map. .

[0029] S5: Based on the fusion features, a high-fidelity image reconstruction module is constructed. After smoothing and refining the fusion features, the feature map is recombined in the spatial dimension through sub-pixel convolution to generate a super-resolution image of the target multiple. The high-fidelity image reconstruction module uses 3×3 refined convolution for feature smoothing, and then uses sub-pixel convolution layers to achieve upsampling by a specified factor to generate a high-fidelity super-resolution image.

[0030] Specifically, the high-fidelity image reconstruction module is executed based on the following formula: ; ; in This represents the activation function; Indicates weight; Indicates the bias parameter; The superscript indicates the current floor number; The subscript indicates the size of the convolution kernel; In the superscript, the first parameter indicates the layer number of the current weight, and the second parameter indicates the position of the current weight in the convolution of the current layer. This represents the subpixel convolution upsampling operator; This represents the fused feature map output by the global residual connection module; This represents the feature map after smoothing by the refined convolutional layer. This represents the generated super-resolution image.

[0031] The high-fidelity image reconstruction process is as follows: Fusion Feature Map First, a 3×3 refined convolutional layer is used for feature smoothing to obtain the feature map. ; feature map Feed into subpixel convolutional layer By rearranging channel-dimensional information into spatial dimensions, an r-fold upsampling mapping is achieved with extremely low parameter overhead, generating high-fidelity super-resolution images. .

[0032] S6: Based on super-resolution images and high-resolution images, a hybrid loss function module is constructed, which combines pixel robustness loss and edge structure constraint loss to obtain the hybrid loss; The pixel robustness loss uses the Charbonnier loss function to maintain the pixel-level fidelity between the super-resolution image and the real high-resolution image; the edge structure constraint loss is calculated by the difference between the horizontal and vertical gradients to enhance the high-frequency edge reconstruction capability.

[0033] The hybrid loss function module is executed based on the following formula: ; ; ; in, Indicates pixel robustness loss; This represents the edge structure constraint loss; Represents the total loss function; This represents the generated super-resolution image; This represents the corresponding true high-resolution image; and These represent the gradient operators of the image in the horizontal and vertical directions, respectively; Represents the L1 norm; It is the numerical stability constant; These are the edge loss weighting coefficients.

[0034] The process of handling the hybrid loss function is as follows: Calculate and generate image With real images pixel robustness loss The Charbonnier loss function is used to maintain pixel-level fidelity; the generated images are calculated respectively. With real images The edge structure constraint loss is obtained by calculating the gradient difference in the horizontal and vertical directions. Enhance the model's ability to reconstruct high-frequency structures; incorporate pixel robustness loss. Edge structure constraint loss By weight Perform a weighted summation to obtain the total loss function. While maintaining the overall brightness consistency of the image, the sharpness of the structural edges is maximized.

[0035] S7: Construct a lightweight super-resolution reconstruction model based on structure-re-parameterized convolution and linear large kernel attention, train the model using hybrid loss, and update the model parameters. The lightweight super-resolution reconstruction model completes model training by iteratively updating model parameters after minimizing the mixture loss function.

[0036] The super-resolution reconstruction model is as follows: ; in, This represents the super-resolution network model proposed in this paper; This represents the parameters in the super-resolution model presented in this paper, with the ultimate goal of updating the parameters. This makes the total loss function Minimize, that is: .

[0037] S8: Input the low-resolution image to be processed into the trained lightweight super-resolution reconstruction model and output the final super-resolution image.

[0038] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0039] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A lightweight super-resolution method based on structure reparameterization and linear big-kernel attention, characterized in that, Includes the following steps: S1: Preprocess the image to obtain an image pair consisting of a high-resolution image and a low-resolution image; S2: Construct a shallow feature extraction module based on low-resolution images; It also performs spatial mapping and nonlinear transformation on low-resolution images to convert pixel-level information into high-dimensional feature representations, i.e., shallow features. S3: Based on high-dimensional feature representation, a deep reparameterized feature extraction group is constructed. This extraction group consists of N cascaded reparameterized large kernel attention blocks. Each reparameterized large kernel attention block sequentially performs structural reparameterized convolution, linear large kernel attention calibration and residual connection. The decomposition characteristics of depthwise separable dilated convolution are used to capture cross-region similarity and output deep features. S4: Construct a global residual connection module based on shallow and deep features; This is used to sum the shallow and deep features element-wise and force the model to learn the residual mapping between low-resolution and high-resolution images to obtain fused features. S5: Based on the fusion features, a high-fidelity image reconstruction module is constructed. After smoothing and refining the fusion features, the feature map is recombined in the spatial dimension through sub-pixel convolution to generate a super-resolution image of the target multiple. S6: Based on super-resolution images and high-resolution images, a hybrid loss function module is constructed, which combines pixel robustness loss and edge structure constraint loss to obtain the hybrid loss; S7: Construct a lightweight super-resolution reconstruction model based on structure-re-parameterized convolution and linear large kernel attention, train the model using the hybrid loss, and update the model parameters; S8: Input the low-resolution image to be processed into the trained lightweight super-resolution reconstruction model, and output the final super-resolution image.

2. The lightweight super-resolution method based on structural reparameterization and linear big kernel attention as described in claim 1, characterized in that: The shallow feature extraction module uses 3×3 standard convolution for spatial mapping and completes feature transformation through a non-linear activation function with learnable parameters to obtain a shallow feature map.

3. The lightweight super-resolution method based on structural reparameterization and linear big kernel attention as described in claim 2, characterized in that: The reparameterized convolutional structure employs a multi-branch parallel topology during the training phase, including depth branches, convolutional branches of different sizes, and identity branches, to enhance feature representation capabilities. During the inference phase, multiple branches are equivalently collapsed into a single standard convolution operator, eliminating redundant inference overhead.

4. The lightweight super-resolution method based on structural reparameterization and linear big-kernel attention as described in claim 3, characterized in that: The multi-branch parallel topology includes a serialization depth branch, a 3×3 convolution branch, a 1×3 convolution branch, a 3×1 convolution branch, a 1×1 convolution branch, and an identity branch, with each convolution branch connected to a batch normalization layer.

5. The lightweight super-resolution method based on structural reparameterization and linear big kernel attention as described in claim 4, characterized in that: The linear large kernel attention block sequentially employs a 5×5 depthwise convolution, a 7×7 depthwise dilated convolution with an expansion rate of 3, and a 1×1 convolution to generate a global attention weight matrix. This matrix is ​​then element-wise multiplied with the deep input feature map fed into the attention block to achieve dynamic feature calibration.

6. The lightweight super-resolution method based on structural reparameterization and linear big-kernel attention as described in claim 5, characterized in that: Each of the reparameterized large kernel attention blocks employs residual connections to add the structurally reparameterized convolution output features to the attention-calibrated features, thus obtaining the current block's output features.

7. The lightweight super-resolution method based on structural reparameterization and linear big kernel attention as described in claim 6, characterized in that: The global residual connection directly sums the shallow and deep features element-wise, allowing the model to focus on learning the residuals between high and low resolution images, preserving low-frequency structural information and reducing training difficulty.

8. The lightweight super-resolution method based on structural reparameterization and linear big-kernel attention as described in claim 7, characterized in that: The high-fidelity image reconstruction module uses 3×3 refined convolution for feature smoothing, and then uses sub-pixel convolution layers to achieve upsampling by a specified factor to generate a high-fidelity super-resolution image.

9. The lightweight super-resolution method based on structural reparameterization and linear big-kernel attention as described in claim 8, characterized in that: The pixel robustness loss uses the Charbonnier loss function to maintain pixel-level fidelity between the super-resolution image and the real high-resolution image; the edge structure constraint loss is calculated by the difference between the horizontal and vertical gradients.

10. The lightweight super-resolution method based on structural reparameterization and linear big-kernel attention according to claim 9, characterized in that: The lightweight super-resolution reconstruction model completes model training by iteratively updating model parameters after minimizing the hybrid loss function.

Images