An image super-resolution reconstruction method

Abstract

This invention discloses an image super-resolution reconstruction method, comprising: preprocessing a low-resolution image; performing shallow feature extraction on the preprocessed low-resolution image to obtain a shallow feature map; inputting the shallow feature map into a deep feature extraction module to obtain a deep feature map; performing global residual fusion of the deep feature map and the shallow feature map; the deep feature extraction module includes several cascaded residual groups; each residual group includes sequentially connected local self-attention units (L-SA) and spatial-channel reconstruction convolutional units (SCConv); and inputting the fused features into an image reconstruction module to output the final high-resolution image. The image super-resolution reconstruction method provided by this invention not only has the advantages of strong and stable local textures, but also possesses excellent global modeling capabilities, while also exhibiting advantages such as low computational complexity, low memory usage, and fast computation speed.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning technology, specifically to an image super-resolution reconstruction method. Background Technology

[0002] Image super-resolution reconstruction aims to recover high-resolution details from low-resolution images and is an important technology in fields such as video enhancement, remote sensing, medical imaging, and security monitoring. Currently, the mainstream methods are based on deep learning. These mainly include super-resolution methods based on convolutional neural networks, such as EDSR and ESRGAN. Their main advantages are strong and stable local texture, but they lack the ability to model global information. Later, Transformer-based super-resolution methods emerged, with representative models such as SwinIR, OmniSR, and RGT. Their advantages are the ability to capture long-range dependencies in features, but their disadvantages are high computational complexity, high memory consumption, and slow inference speed. Among these, the network model structure represented by the Recursive Generalized Transformer (RGT) reduces the global modeling cost by setting recursive global self-attention, but it still relies on complex cloud structures such as multiple matrix projections and cross-feature attention, resulting in significant training and inference resource consumption. On the other hand, convolutional networks have good local modeling capabilities and are hardware-friendly, but they lack the ability to integrate cross-regional information, making it difficult to ensure reconstruction quality in scenarios with strong global consistency.

[0003] Therefore, there is an urgent need for a super-resolution network structure that combines the advantages of convolution efficiency with global modeling capabilities, and has lower computational complexity. Summary of the Invention

[0004] To overcome the problems existing in the prior art, this invention provides an image super-resolution reconstruction method. For the first time, the SCConv module is used for the SR (Super-Resolution) task. A residual group structure with alternating convolution and attention stacking is designed. The image super-resolution reconstruction method based on this structure not only has the advantages of strong local texture and stability, but also has excellent global modeling capabilities. At the same time, it also has the advantages of low computational complexity, low memory usage, and fast computation speed.

[0005] To achieve the above objectives, the present invention provides an image super-resolution reconstruction method, comprising the following steps:

[0006] Step S1: Preprocess the low-resolution image;

[0007] Step S2: Perform shallow feature extraction on the preprocessed low-resolution image, and map the image spatial information to a high-dimensional feature space to obtain a shallow feature map;

[0008] Step S3: Input the shallow feature map into the deep feature extraction module to extract deep features and obtain a deep feature map; perform global residual fusion between the deep feature map and the shallow feature map;

[0009] The deep feature extraction module includes several cascaded residual groups; each residual group includes a sequentially connected Local Self-Attention (L-SA) module and a Spatial-Channel Reconstruction Convolution (SCConv) module; the L-SA module is used to capture long-distance dependencies between pixels within a local window; the SCConv module is used to perform dual spatial and channel reconstruction of features, which reduces feature redundancy while enhancing the spatial and channel representation capabilities of features.

[0010] Step S4: Input the fused features into the image reconstruction module to reconstruct the image and output the final high-resolution image.

[0011] Preferably, step S2 includes:

[0012] Step S21: Use convolutional layers to extract features and expand channels in the preprocessed low-resolution image; expand the RGB three channels into a high-dimensional feature space through convolution operations, and output the expanded feature tensor;

[0013] Step S22: Input the expanded feature tensor into the space-channel reconstruction convolution SCConv module to reconstruct and optimize the output shallow feature map.

[0014] Preferably, step S3 includes:

[0015] Step 31: Convert the shallow feature map into a sequence format, then apply layer normalization to output the normalized features in sequence format;

[0016] Step 32: Normalized features of sequence format. Input several residual groups, process them, and output the deep features of the sequence format.

[0017] Step 33: Apply layer normalization to the deep features of the sequence format, then convert the format back to image format, and output the normalized deep features;

[0018] Step 34: Use a post-processing convolutional layer to refine the normalized deep features, and then perform residual connections with the shallow feature maps to achieve the fusion of shallow and deep features, and output the final fused features.

[0019] Preferably, in step S32, the processing steps for each residual group include:

[0020] Step 321: Save the input features of the current residual group;

[0021] Step 322: Input the input features of the current residual group into the local self-attention L-SA module, and obtain the same dimension output by capturing the long-distance dependencies between pixels within the local window;

[0022] Step 323: Convert the format of the features processed by the L-SA module into an image format;

[0023] Step 324: Input the image format features into the spatial-channel reconstruction convolution SCConv module for spatial and channel reconstruction processing;

[0024] Step 325: Perform feature fusion on the features processed by the SCConv module using convolution;

[0025] Step 326: Convert the image format features fused in step 325 back to sequence format, and then add the input features saved in step 321 back to the features converted back to sequence format to achieve identity mapping and gradient flow.

[0026] Preferably, the spatial-channel reconstruction convolution SCConv module includes a spatial reconstruction unit (SRU) and a channel reconstruction unit (CRU).

[0027] Preferably, the processing steps of the Spatial Reconfiguration Unit (SRU) include:

[0028] Group normalization: The input features are grouped by channel dimension, and the features within each group are normalized; the normalized features are scaled and translated using a learnable scaling parameter γ and an offset parameter β.

[0029] Weighted importance assessment: Calculate the normalized weights of the scaling parameter γ; multiply the normalized features by the normalized weights to obtain the weighted feature representation, and then map the weighted features to the [0, 1] interval through the Sigmoid activation function to obtain the reweighted values;

[0030] Adaptive gating mechanism: Set a gating threshold. When the reweighted value is greater than or equal to the gating threshold, generate an information mask to identify information-rich feature regions; when the reweighted value is less than the gating threshold, generate a non-information mask to identify sparse feature regions.

[0031] Feature separation: Information-rich features are obtained by using information masks to extract features from information-rich feature regions; information-sparse features are obtained by using non-information masks to extract features from sparse feature regions.

[0032] Cross-reconstruction fusion: The information-rich features are divided into two halves along the channel dimension; the information-sparse features are divided into two halves along the channel dimension; the divided information-rich features and information-sparse features are cross-added to obtain the feature tensor after spatial reconstruction;

[0033] The CRU (Channel Reconfiguration Unit) processing steps include:

[0034] Channel dimension segmentation: The feature tensor after spatial reconstruction is divided into an upper half and a lower half along the channel dimension;

[0035] Channel compression: Channel compression is performed on the upper and lower halves using convolutions respectively;

[0036] Multi-path feature transformation: Parallel processing of the compressed upper half features: Path 1: Group convolution transformation; Path 2: Point convolution transformation; The outputs of Path 1 and Path 2 are then fused; Lower half processing: Point convolution expansion is performed on the compressed lower half features, and the expanded features are concatenated with the original compressed lower half features.

[0037] Channel splicing: splices the processing results of the upper and lower parts along the channel dimension;

[0038] Channel attention weighting: Global average pooling is performed on the concatenated features, and then the Softmax function is used to generate channel attention weights; the channel attention weights are multiplied by the concatenated features to achieve adaptive weighting;

[0039] Feature fusion output: The weighted features are divided into two parts along the channel dimension, and then the addition and fusion are performed to output the feature tensor after channel reconstruction.

[0040] Preferably, the image reconstruction module performs upsampling through subpixel convolution operations, and then reconstructs and outputs the final high-resolution image through convolutional layers.

[0041] Preferably, step S4 includes:

[0042] Step 41: Use convolutional layers to reduce the feature dimension of the fused features;

[0043] Step 42: Upsampling Processing: Perform the corresponding upsampling operation according to the upsampling factor; for power-of-2 upsampling: use a progressive PixelShuffle upsampling strategy to expand the spatial size, upsampling by 2 times each time, and performing n iterations; for 3x upsampling: use a single PixelShuffle operation to expand the spatial size by 3 times; output the upsampled features;

[0044] Step 43: The upsampled features are used to generate the original super-resolution image using the final convolutional layer;

[0045] Step 44: Adjust the pixel values ​​of the original super-resolution image to the original range and output the final high-resolution image.

[0046] Preferably, the convolutional layer in step S21 is configured with 3 input channels, 180 output channels, a kernel size of 3×3, a stride of 1, and padding of 1.

[0047] Preferably, the preprocessing procedure in step S1 includes:

[0048] Step S11: Normalize the pixel values ​​of the low-resolution image;

[0049] Step S12: Then, center the normalized low-resolution image by subtracting the mean of the RGB three channels.

[0050] Compared with the prior art, the beneficial effects of the present invention are:

[0051] 1. The SCConv module is used for the first time in the SR (Super-Resolution) task, and a residual group structure with alternating convolution and attention is designed. The image super-resolution reconstruction method based on this structure not only has the advantages of strong local texture and stability, but also has excellent global modeling ability. At the same time, it also has the advantages of low computational complexity, low memory usage and fast computation speed.

[0052] To address the challenges of feature level mismatch and cross-layer fusion, this invention employs a fixed alternation structure of L-SA and SCConv. SCConv performs spatial compression and channel reorganization before entering the next attention module, achieving implicit feature alignment. This solves the semantic misalignment problem in RGT, resulting in a more continuous and stable information flow between modules, eliminating the need for an additional HAI (Hybrid-Adaptive-Integration) fusion structure.

[0053] 2. Addressing the issue of SCConv's incompatibility with super-resolution tasks: The original SCConv paper only covered classification and detection testing, not super-resolution tasks. This invention innovatively solves the problem of its application in the field of super-resolution tasks. To address its local dependency matching fusion with L-SA, the synthesis mode of SCConv and L-SA designed in this invention is not a simple concatenation, but rather a consistency information routing and residual fusion design based on the characteristics of "local texture + global consistency". For example, in step 323, data format conversion is performed to match the input data format of each module; in step 326, residual connection is adopted to perform residual connection between the input and output of the module, etc.

[0054] 3. Addressing the high complexity issue arising from global modeling's reliance on attention: Traditional Transformers like RGT require recursive generalization of attention and cross-attention to obtain the global context, involving multiple Q, K, V projections and matrix multiplications, resulting in high computational and memory overhead, significantly burdening high-resolution SR scenes. The SCConv module (SRU+CRU) replaces the RG-SA module, employing a GN-γ-guided spatial saliency filtering and channel reconstruction mechanism as the global feature modeling path. It utilizes spatial-channel reconstruction convolution to achieve "global information approximation," replacing the Q / K / V cross-attention mechanism. By combining SCConv's SRU (spatial reconstruction) and CRU (channel reconstruction) to replace the Q / K / V matrix calculations and cross-attention mechanism in remote attention, the complex Cross-Attention module is eliminated, significantly reducing FLOPs and memory costs, thus achieving low-cost global feature extraction.

[0055] 4. Solve the problem of high training and inference costs in high-resolution super-resolution tasks: Replace the global attention computation path with a convolution-attention hybrid path. The SCConvSR convolution operation can make full use of GPU (Graphic-Process-Unit) acceleration, with higher inference throughput, reduced peak memory usage, and is more suitable for edge deployment and real-time super-resolution.

[0056] 5. The model of the method of the present invention is a lightweight super-resolution high-efficiency model, which can reduce FLOPs and GPU memory and improve deployment efficiency.

[0057] 6. In experiments: Compared with the traditional GRT model, the advantages of this invention are: halved parameters, from 4.33 million to 2.2 million; halved computational cost, from 17G to 9G; inference acceleration, ideally about 1.9 times; and memory saving, with storage space reduced by about 50%. Compared with current mainstream methods such as RCAN, SAN, HAN, NLSA, and SwinIR, this invention achieves performance improvements on multiple datasets, especially demonstrating stronger recovery and structure preservation capabilities on images with complex textures and rich details (such as Urban100 and Manga109). Attached Figure Description

[0058] Figure 1 The overall flowchart of the algorithm model provided in the embodiments of the present invention;

[0059] Figure 2 for Figure 1 Schematic diagram of L-SA Block in the diagram;

[0060] Figure 3 for Figure 1 A schematic diagram of SCConv in the diagram. Detailed Implementation

[0061] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] Recently, a paper titled SCConv: Spatial and Channel Reconstruction Convolution for Feature Redundancy, published at CVPR 2023, made a core contribution by proposing a two-step convolutional compression strategy, SCConv, which simultaneously reduces spatial and channel redundancy. SCConv consists of SRU (Spatial Reconstruction Unit) and CRU (Channel Reconstruction Unit) connected in series. It can be used to replace standard convolutions in a plug-and-play manner, enhancing feature representation while reducing parameters and FLOPs.

[0063] To improve the reconstruction capability of image super-resolution models, this invention proposes for the first time to replace the recursive global attention module RG-SA with the SCConv module in the Transformer backbone of image super-resolution, and to alternately stack it with the local self-attention module L-SA in a fixed order to construct residual sets, such as... Figure 1 As shown. The image super-resolution reconstruction method based on this structure not only has the advantages of strong and stable local texture, but also excellent global modeling capabilities, as well as advantages such as low computational complexity, low memory usage, and fast computation speed. The specific steps are as follows:

[0064] Step S1 (Input Image Processing): Preprocess the low-resolution image to better match the image data with the network model for model training and experimentation.

[0065] Step S2 (Shallow Feature Mapping): Obtain the preprocessed low-resolution image Shallow feature extraction is performed on the image, mapping the image spatial information to a high-dimensional feature space to obtain a shallow feature map. .

[0066] Step S3 (Deep Feature Extraction): Extract the shallow feature map The input is fed into the deep feature extraction module, which consists of several cascaded residual groups. Within each residual group, the feature flow is alternately processed by local self-attention units (L-SA) and spatial-channel reconstruction convolutional units (ScConv). The spatial-channel reconstruction convolutional units reduce feature redundancy while enhancing the spatial and channel representation capabilities of the features. The local self-attention units capture long-distance dependencies between pixels within a local window. Finally, a deep feature map containing high-frequency detail information is output. .

[0067] Step S4 (Global Fusion and Reconstruction): The fused features are input into the image reconstruction module, upsampled through sub-pixel convolution (PixelShuffle) operation to enlarge the spatial resolution of the feature map to the target size, and the final high-resolution image is reconstructed through convolutional layers.

[0068] Step 1: Input Image Preprocessing

[0069] Input: Low-resolution image The dimensions are (B, 3, H, W), where B is the batch size, 3 is the RGB three-channel image, and H and W are the height and width of the image, respectively.

[0070] Processing procedure: The image pixel value range is [0,255]. The input image pixel value range [0,255] is converted to [0,1] by dividing by 255; then the image is centered by subtracting the mean of the RGB three channels ( = (0.4488, 0.4371, 0.4040); Convert the processed image tensor to a floating-point tensor to ensure that the data type is consistent with the model parameters.

[0071] Output: Preprocessed image tensor The dimensions are (B, 3, H, W).

[0072] Trials and optimizations during the research and development process: In the early stages of the experiment, we tried inputting the raw image data (0-255 pixel values) into the convolutional network.

[0073] Failure phenomenon: Experiments showed that the model converged very slowly, and gradient explosion was prone to occur in deep networks, leading to training interruption.

[0074] Cause analysis: The range of values ​​in the input data is too large, which leads to unstable weight updates.

[0075] Improved solution: The current preprocessing solution was finally determined, which is to first normalize the pixel values ​​to the range of [0,1] and then subtract the mean values ​​of the RGB channels (0.4488, 0.4371, 0.4040). This standardization operation improved the stability of subsequent feature extraction by about 30%.

[0076] Step 2: Shallow Feature Extraction and Initial Structure Reconstruction

[0077] Input: The preprocessed image tensor x_preprocessed, with dimensions (B, 3, H, W).

[0078] Step 2.1: Initial Feature Expansion

[0079] Processing procedure: The first convolutional layer is used to extract features and expand channels of the input image; Convolutional layer configuration: 3 input channels, 180 output channels (default), 3x3 kernel size, 1 stride, and 1 padding; The RGB three channels are expanded into an embedd_dim dimensional feature space through convolution operation.

[0080] Output: The expanded feature tensor x_shallow, with dimensions (B, embed_dim, H, W).

[0081] Step 2.2: Spatial-channel reconstruction convolution optimization (first application of SCConv)

[0082] Processing procedure: The expanded shallow features are subjected to initial feature reconstruction and optimization using the Spatial-Channel Reconstruction Convolution (SCConv) module, such as... Figure 3 As shown.

[0083] SCConv module internal processing flow:

[0084] Step 2.2.1: Spatial Reconfiguration Unit (SRU) Processing

[0085] Group normalization: The input feature tensor x_shallow is grouped by channel dimension into group_num groups (16 groups by default); the features within each group are normalized: the mean and standard deviation of each group are calculated, and then a standardization operation is performed; the normalized features are scaled and translated using a learnable scaling parameter γ and an offset parameter β.

[0086] Weighted importance assessment: Calculate the normalized weights of the scaling parameter γ of the normalized layer: w_gamma = γ / Σγ; Multiply the normalized features by the weights to obtain the weighted feature representation; Map the weighted features to the [0, 1] interval using the Sigmoid activation function to obtain the reweighted values.

[0087] Adaptive gating mechanism: Set the gating threshold gate_treshold (default 0.5); Based on the relationship between the reweights and the threshold, generate two binary masks: information mask: info_mask = (reweights ≥ gate_treshold), which identifies information-rich feature regions; non-information mask: noninfo_mask = (reweights < gate_treshold), which identifies information-sparse feature regions.

[0088] Feature separation: Use an information mask to extract information-rich features: x_info = info_mask × x_shallow; Use a non-information mask to extract sparse features: x_noninfo = noninfo_mask × x_shallow.

[0089] Cross-reconstruction fusion: The information-rich feature x_info is divided into two halves along the channel dimension: x_info_1, x_info_2 = split(x_info); The information-sparse feature x_noninfo is divided into two halves along the channel dimension: x_noninfo_1, x_noninfo_2 = split(x_noninfo); Perform cross-addition operation: x_sru_out = concat([x_info_1 +x_noninfo_2, x_info_2 + x_noninfo_1]).

[0090] SRU output: spatially reconstructed feature tensor x_sru, with dimensions (B, embed_dim, H, W).

[0091] Step 2.2.2: Channel Reconfiguration Unit (CRU) Processing

[0092] Channel dimension segmentation: The feature tensor output by SRU is divided into two parts along the channel dimension: upper part: up = x_sru[:, :up_channel, :, :], where up_channel = α × embed_dim (default α=0.5); lower part: low = x_sru[:, up_channel:, :, :], the number of channels is embed_dim - up_channel.

[0093] Channel compression: Apply 1×1 convolution to the upper part for channel compression: up_compressed = Conv1×1(up), with a compression ratio of squeeze_radio (default 2); apply 1×1 convolution to the lower part for channel compression: low_compressed = Conv1×1(low), with a compression ratio of squeeze_radio.

[0094] Multi-path feature transformation: Upper half parallel processing: Path 1: Group Convolution Transform (GWC): Y1_gwc = GroupConv(up_compressed, kernel=3×3, groups=1); Path 2: Point Convolution Transform (PWC1): Y1_pwc = Conv1×1(up_compressed); Fusion: Y1 = Y1_gwc + Y1_pwc, output channel number is embed_dim. Lower half processing: Point convolution expansion of the compressed features: low_expanded = Conv1×1(low_compressed), expanded channel number is embed_dim - low_compressed_channels; Concatenation with the original compressed features: Y2 = concat([low_expanded, low_compressed]), total channel number is embed_dim.

[0095] Channel concatenation: Concatenate the processing results of the upper and lower parts along the channel dimension: x_cru_inter = concat([Y1,Y2]), the number of channels is 2×embed_dim.

[0096] Channel attention weighting: Global average pooling is performed on the concatenated features to obtain the channel descriptor: channel_desc = AdaptiveAvgPool2d(x_cru_inter), with size (B, 2×embed_dim, 1, 1); the channel attention weights are generated using the Softmax function: attention_weights = Softmax(channel_desc); the attention weights are multiplied by the features to achieve adaptive weighting: x_cru_weighted = attention_weights × x_cru_inter.

[0097] Feature fusion output: The weighted features are divided into two parts along the channel dimension: out1, out2 = split(x_cru_weighted); perform addition fusion: x_cru_out = out1 + out2, and the number of output channels is embed_dim.

[0098] CRU output: The feature tensor x_cru after channel reconstruction, with dimensions (B, embed_dim, H, W).

[0099] The final output of the SCConv module is the feature tensor x_sc_conv_shallow after spatial and channel reconstruction, with dimensions (B, embed_dim, H, W).

[0100] Output: The feature tensor F_shallow = x_sc_conv_shallow after shallow feature extraction and reconstruction, with dimensions (B, embed_dim, H, W).

[0101] Step 3: Deep Feature Extraction

[0102] Input: Shallow features F_shallow, with dimensions (B, embed_dim, H, W).

[0103] Step 3.1: Feature format conversion and normalization

[0104] Processing steps: Convert the feature tensor from image format to sequence format: flatten the spatial dimension H×W into one dimension to obtain (B, H×W, embed_dim); apply layer normalization (LayerNorm) to the features in sequence format: calculate the mean and variance of each feature vector and perform standardization.

[0105] Output: Normalized features x_seq in sequence format, with dimensions (B, H×W, embed_dim).

[0106] Step 3.2: Multi-level residual group processing

[0107] Processing procedure: The network contains N (4 by default) residual groups. Each residual group uses the ScConvResidualGroup structure to replace the original ResidualGroup based on recursive-generalized self-attention (RG-SA).

[0108] Perform the following operations for each residual group:

[0109] Step 3.2.1: Save residual connection input: Save the input features of the current residual group: x_residual = x_seq, for subsequent residual connections.

[0110] Step 3.2.2: L-SA Input Processing: The input features are processed by the L-SA module to obtain outputs of the same dimension. The input and output data format is (B, HxW, embed_dim). The L-SA module is a module in the traditional RGT model (RECURSIVE GENERALIZATION TRANSFORMER FOR IMAGE SUPER-RESOLUTION). Figure 2 This is a schematic diagram of an L-SA Block.

[0111] Step 3.2.3: Data format conversion: Convert the sequence format to the image format: x_image = Reshape(x_seq), from (B, H×W, embed_dim) to (B, embed_dim, H, W).

[0112] Step 3.2.3: Spatial-Channel Reconstruction Convolution Processing: Apply the SCConv module to the image format features, performing the same spatial and channel reconstruction processing as in Step 2.2. Output: x_sc_conv_deep, with dimensions (B, embed_dim, H, W).

[0113] Step 3.2.5: Residual Convolutional Connection: Use a 3x3 convolution to fuse features from the SCConv output. Output: x_conv = Conv2d(embed_dim→embed_dim, kernel=3×3)(x_sc_conv_deep).

[0114] Step 3.2.6: Data Format Conversion and Residual Connection: Convert the image format back to sequence format: x_seq_processed = Reshape(x_conv), from (B, embed_dim, H, W) to (B, H×W, embed_dim); Perform residual connection: x_seq = x_residual + x_seq_processed, adding the original input with the processed features to achieve identity mapping and gradient flow.

[0115] Output: The feature x_seq_deep after processing all residual groups, with dimensions (B, H×W, embed_dim).

[0116] Step 3.3: Deep Feature Normalization and Format Restoration: Apply layer normalization to the deep features in sequence format: x_seq_deep_norm = LayerNorm(x_seq_deep); Convert the sequence format back to image format: restore the flattened H×W dimensions to spatial dimensions H and W, obtaining (B, embed_dim, H, W). Output: Normalized deep feature F_deep, with dimensions (B, embed_dim, H, W).

[0117] Step 3.4: Deep Feature Post-processing and Residual Fusion: Deep features are refined using a post-processing convolutional layer (conv_after_body). Convolution configuration: Both input and output channels are embed_dim, kernel size is 3×3, stride is 1, and padding is 1. Perform the convolution operation: F_deep_processed = Conv2d(embed_dim→embed_dim,kernel=3×3)(F_deep). Perform a residual connection with shallow features: F_final = F_deep_processed + F_shallow, fusing shallow and deep features. Output: The final fused feature F_final, with dimensions (B, embed_dim, H, W).

[0118] Failures and attempts during the R&D process: This step is the core of model design, and the architecture underwent multiple adjustments during the R&D process.

[0119] (1) Regarding the selection of core modules:

[0120] Early attempts: Ultimately, the "Recursive Generalized Self-Attention (RG-SA) module" from the original RGT model was adopted.

[0121] Problems encountered: Although RG-SA can extract global information, its computational complexity is too high (FLOPs reach 16.97G) and its memory consumption is huge, making it unsuitable for real-time inference on edge devices. It was also found that RG-SA is a coarse-grained design, which easily leads to the loss of high-frequency texture details.

[0122] Failed attempt: An attempt was made to directly reduce the number of layers in RG-SA, which resulted in a significant decrease in super-resolution performance (PSNR).

[0123] Final solution: Spatial-channel reconstruction convolution (ScConv) was introduced to replace RG-SA. Experiments show that ScConv removes redundant features through SRU and CRU units, and while reducing the number of parameters by 49.3%, it significantly improves the reconstruction quality due to its denoising effect.

[0124] (2) Challenges regarding dimensional alignment:

[0125] The problem encountered: The Transformer module (L-SA) outputs sequence-format features (B, N, C), while the convolution module (ScConv) requires image-format features (B, C, H, W). Direct concatenation caused a dimension mismatch error.

[0126] Failed attempt: Initial attempts to modify the internal code of ScConv to support sequential inputs, but this undermined the parallel computational advantages of convolutions, resulting in slower inference speeds.

[0127] Solution: An explicit dimension transformation layer (Reshape / View operations) was designed to wrap the ScConv module. Although this added a few operations, it ensured efficient collaboration between the two heterogeneous operators and successfully solved the problem of "gradient flow breakage".

[0128] Step 4: Image Reconstruction and Upsampling

[0129] Input: The fused features F_final, with dimensions (B, embed_dim, H, W).

[0130] Step 4.1: Feature Preparation: Reduce the feature dimension from `embed_dim` to `num_feat` (default 64) using convolutional layers: Convolution configuration: input channels are `embed_dim`, output channels are `num_feat`, kernel size is 3×3, stride is 1, padding is 1; Perform convolution operation: `x_feat = Conv2d(embed_dim→num_feat, kernel=3×3)(F_final)`. Apply the LeakyReLU activation function: `x_feat_act = LeakyReLU(x_feat)` to enhance non-linear expressiveness. Output: The ready feature `x_feat_act`, with dimensions (B, num_feat, H, W).

[0131] Step 4.2: Upsampling: Perform the corresponding upsampling operation according to the upsampling factor s (usually 2, 3 or 4).

[0132] For power-of-2 upsampling: a progressive PixelShuffle upsampling strategy is adopted, upsampling by a factor of 2 each time, for a total of n iterations; each iteration includes: convolution expansion: using a 3×3 convolution to expand the number of channels by a factor of 4: x_expand = Conv2d(num_feat→4×num_feat, kernel=3×3)(x_input); PixelShuffle operation: rearranging the expanded features to achieve a factor of 2 upsampling: x_upsampled = PixelShuffle(x_expand), increasing the spatial size by a factor of 2, and restoring the number of channels to num_feat. After n iterations, the spatial size expands from (H, W) to (s×H, s×W).

[0133] For 3x upsampling: using a single PixelShuffle operation: convolutional expansion: x_expand = Conv2d(num_feat→9×num_feat, kernel=3×3)(x_feat_act); PixelShuffle operation: x_upsampled= PixelShuffle(x_expand, scale=3), increasing the spatial size by a factor of 3. Output: the upsampled features x_upsampled, with dimensions (B, num_feat, s×H, s×W).

[0134] Step 4.3: Final Image Reconstruction

[0135] Processing procedure: Generate RGB output using the final convolutional layer: Convolution configuration: input channels are num_feat, output channels are 3 (RGB), kernel size is 3×3, stride is 1, padding is 1; Perform convolution operation: I_SR_raw = Conv2d(num_feat→3, kernel=3×3)(x_upsampled). Output: Original super-resolution image I_SR_raw, with dimensions (B, 3, s×H, s×W).

[0136] Step 4.4: Post-processing and Output: Processing steps: Pixel values ​​are reduced back to their original range: The image is divided by the `img_range` parameter to restore the original pixel value range; the RGB mean is added back, adding the mean of the RGB three channels back to the image to achieve de-centering. Output: The final super-resolution image I_SR has dimensions (B, 3, sxH, sxW).

[0137] Trials and optimizations during the R&D process: In the image upsampling stage, we compared different reconstruction strategies:

[0138] Early attempts: Initially, deconvolution / transposed convolution layers were used to directly perform upsampling.

[0139] Failure phenomenon: The output high-resolution image shows obvious "checkerboard effect" in flat areas, which seriously affects visual perception.

[0140] Improved solution: Abandon deconvolution and adopt a "convolution + subpixel shuffling" strategy instead.

[0141] Technical effect: This strategy expands the spatial dimension by rearranging the channel dimension, which not only completely eliminates the checkerboard artifact, but also makes the edge reconstruction sharper by utilizing more channel information.

[0142] Table 1 compares the number of parameters and floating-point computations of the RG module and the replaced ScConv module in the RGT model.

[0143] Table 1: Comparison of RGT models using the RG module and those using the ScConv module

[0144]

[0145] The advantages of ScConv compared to the RG module are: half the number of parameters, from 4.33 million to 2.2 million; half the computational load, from 17G to 9G; inference speed up, ideally by about 1.9 times; and memory saving, with storage space reduced by about 50%.

[0146] 5. Quantitative Indicator Results

[0147] The improved model uses the DIV2K training dataset (containing 800 images and their corresponding downsampled images), and the test sets include Set5, Set14, BSD100, Urban100, and Manga109. PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Score) are used as quantitative metrics to measure model performance.

[0148] Table 2: Quantization comparison of this invention with other models on different test sets

[0149]

[0150] In the experimental setup, all methods were compared under a 2x upsampling task. As shown in Table 2, the improved model achieved excellent performance on all test sets. On the Set5 dataset, the model achieved a PSNR of 38.32 dB and an SSIM of 0.9617; on Set14, the PSNR was 34.15 and the SSIM was 0.9229; on BSD100, the PSNR was 32.43 and the SSIM was 0.9027; on Urban100, the PSNR was 33.61 and the SSIM was 0.9406; and on Manga109, the PSNR was 39.36 and the SSIM was 0.9783. Compared with current mainstream methods such as RCAN, SAN, HAN, NLSA, and SwinIR, this invention achieves performance improvements on multiple datasets, especially demonstrating stronger recovery and structure preservation capabilities on images with complex textures and rich details (such as Urban100 and Manga109).

[0151] 6. Experimental Environment and Details

[0152] The algorithm model of this invention is developed and implemented based on the open-source super-resolution benchmark framework BasicSR. The specific hardware and software configurations are as follows:

[0153] (1) Hardware environment

[0154] Computing Core (GPU): Equipped with an NVIDIA GeForce RTX 3090 graphics card

[0155] Memory capacity: 24GB GDDR6X, providing ample memory support for the deep attention architecture in large-scale image processing and high-frequency feature extraction.

[0156] (2) Software environment

[0157] To ensure the reproducibility of experimental results and the efficient execution of the algorithm, the software environment is configured as follows:

[0158] Development Framework: Custom development is based on the BasicSR open-source framework. This framework integrates the implementation logic of various classic super-resolution algorithms, providing a standard training, verification, and evaluation process for this invention.

[0159] Operating system: Ubuntu 20.04 LTS.

[0160] Virtual environment: Conda is used for environment isolation, effectively managing complex third-party dependency libraries.

[0161] Core library version: Python 3.8; deep learning framework: PyTorch 1.10, which utilizes the dynamic computation graph feature to achieve heterogeneous collaboration between ScConv and L-SA modules; CUDA acceleration: CUDA 11.x and cuDNN environment, which significantly optimizes convolution operation efficiency through parallel computing.

[0162] Data preprocessing: Image normalization (0-1 interval) and centering (subtracting the mean of the RGB three channels) were performed using NumPy and OpenCV, which significantly improved the stability of model training.

[0163] (3) Implementation instructions

[0164] The model follows the standard BasicSR process during training, performing upsampling through a PixelShuffle strategy. Compared to traditional deconvolution methods, this approach more effectively eliminates the checkerboard effect on an RTX 3090 environment, while maintaining high-performance output with a reduction of approximately 49.3% in parameters.

[0165] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. An image super-resolution reconstruction method, characterized in that, Includes the following steps: Step S1: Preprocess the low-resolution image; Step S2: Perform shallow feature extraction on the preprocessed low-resolution image, and map the image spatial information to a high-dimensional feature space to obtain a shallow feature map; Step S3: Input the shallow feature map into the deep feature extraction module to extract deep features and obtain a deep feature map; perform global residual fusion between the deep feature map and the shallow feature map; The deep feature extraction module includes several cascaded residual groups; each residual group includes a sequentially connected Local Self-Attention (L-SA) module and a Spatial-Channel Reconstruction Convolution (SCConv) module; the L-SA module is used to capture long-distance dependencies between pixels within a local window; the SCConv module is used to perform dual spatial and channel reconstruction of features, reducing feature redundancy while enhancing the spatial and channel representation capabilities of features; Step S4: Input the fused features into the image reconstruction module to reconstruct the image and output the final high-resolution image.

2. The image super-resolution reconstruction method according to claim 1, characterized in that, Step S2 includes: Step S21: Use convolutional layers to extract features and expand channels in the preprocessed low-resolution image; expand the RGB three channels into a high-dimensional feature space through convolution operations, and output the expanded feature tensor; Step S22: Input the expanded feature tensor into the spatial-channel reconstruction convolution SCConv module to reconstruct features and optimize the output shallow feature map.

3. The image super-resolution reconstruction method according to claim 2, characterized in that, Step S3 includes: Step 31: Convert the shallow feature map into a sequence format, then apply layer normalization processing to output the normalized features in sequence format; Step 32: The normalized features of the sequence format are input into several residual groups for processing to output the deep features of the sequence format; Step 33: Apply layer normalization to the deep features of the sequence format, then convert the format back to image format, and output the normalized deep features; Step 34: Use a post-processing convolutional layer to refine the normalized deep features, and then perform a residual connection with the shallow feature map to achieve the fusion of shallow and deep features, and output the final fused features.

4. The image super-resolution reconstruction method according to claim 3, characterized in that, In step S32, the processing steps for each residual group include: Step 321: Save the input features of the current residual group; Step 322: Input the input features of the current residual group into the Local Self-Attention (L-SA) module, and obtain the same dimension output by capturing the long-distance dependencies between pixels within the local window; Step 323: Convert the format of the features processed by the L-SA module into an image format; Step 324: Input the image format features into the spatial-channel reconstruction convolution SCConv module for spatial and channel reconstruction processing; Step 325: Perform feature fusion on the features processed by the SCConv module using convolution; Step 326: Convert the image format features fused in step 325 back to sequence format, and then add the input features saved in step 321 back to the features converted back to sequence format to achieve identity mapping and gradient flow.

5. An image super-resolution reconstruction method according to any one of claims 1-4, characterized in that: The spatial-channel reconstruction convolution SCConv module includes a spatial reconstruction unit (SRU) and a channel reconstruction unit (CRU).

6. The image super-resolution reconstruction method according to claim 5, characterized in that: The processing steps of the spatial reconstruction unit (SRU) include: Group normalization: The input features are grouped by channel dimension, and the features within each group are normalized; the normalized features are scaled and translated using a learnable scaling parameter γ and an offset parameter β. Weighted importance assessment: Calculate the normalized weights of the scaling parameter γ; multiply the normalized features by the normalized weights to obtain the weighted feature representation, and then map the weighted features to the [0, 1] interval through the Sigmoid activation function to obtain the reweighted value; Adaptive gating mechanism: Set a gating threshold. When the reweighted value is greater than or equal to the gating threshold, generate an information mask to identify information-rich feature regions; when the reweighted value is less than the gating threshold, generate a non-information mask to identify information-sparse feature regions. Feature separation: Information-rich features are obtained by using information masks to extract features from information-rich feature regions; information-sparse features are obtained by using non-information masks to extract features from sparse feature regions. Cross-reconstruction fusion: The information-rich features are divided into two halves along the channel dimension; the information-sparse features are divided into two halves along the channel dimension; the divided information-rich features and information-sparse features are cross-added to obtain the feature tensor after spatial reconstruction; The channel reconstruction unit (CRU) processing steps include: Channel dimension segmentation: The feature tensor after spatial reconstruction is divided into an upper half and a lower half along the channel dimension; Channel compression: Channel compression is performed on the upper and lower halves using convolution respectively; Multi-path feature transformation: Parallel processing of the compressed upper half features: Path 1: Group convolution transformation; Path 2: Point convolution transformation; The outputs of Path 1 and Path 2 are then fused; Lower half processing: Point convolution expansion is performed on the compressed lower half features, and the expanded features are concatenated with the original compressed lower half features. Channel splicing: splices the processing results of the upper and lower parts along the channel dimension; Channel attention weighting: Global average pooling is performed on the concatenated features, and then the Softmax function is used to generate channel attention weights; the channel attention weights are multiplied by the concatenated features to achieve adaptive weighting; Feature fusion output: The weighted features are divided into two parts along the channel dimension, and then the addition and fusion are performed to output the feature tensor after channel reconstruction.

7. The image super-resolution reconstruction method according to claim 1, characterized in that: The image reconstruction module performs upsampling through subpixel convolution operations, and then reconstructs and outputs the final high-resolution image through convolutional layers.

8. The image super-resolution reconstruction method according to claim 7, characterized in that, Step S4 includes: Step 41: Use convolutional layers to reduce the feature dimension of the fused features; Step 42: Upsampling Processing: Perform the corresponding upsampling operation according to the upsampling factor; for power-of-2 upsampling: use a progressive PixelShuffle upsampling strategy to expand the spatial size, upsampling by 2 times each time, and performing n iterations; for 3x upsampling: use a single PixelShuffle operation to expand the spatial size by 3 times; output the upsampled features; Step 43: The upsampled features are used to generate the original super-resolution image using the final convolutional layer; Step 44: Adjust the pixel values ​​of the original super-resolution image to the original range and output the final high-resolution image.

9. The image super-resolution reconstruction method according to claim 2, characterized in that: The convolutional layer in step S21 is configured with 3 input channels, 180 output channels, a kernel size of 3×3, a stride of 1, and padding of 1.

10. The image super-resolution reconstruction method according to claim 1, characterized in that, The preprocessing procedure in step S1 includes: Step S11: Normalize the pixel values ​​of the low-resolution image; Step S12: Then, center the normalized low-resolution image by subtracting the mean of the RGB three channels.

Images