A lightweight image super-resolution method based on superpixel guidance

Abstract

This invention proposes a lightweight image super-resolution method based on superpixel guidance. The method includes: processing a low-resolution image using SLIC to obtain a superpixel label map; performing a convolutional mapping on the low-resolution image using lightweight convolution to obtain a shallow feature map; using the shallow feature map as input to a feature enhancement block to obtain the output feature map of an intermediate feature enhancement block; using the output feature map of the intermediate feature enhancement block as input to the next feature enhancement block, and stacking a predetermined number of feature enhancement blocks to obtain a deep feature map; using the deep feature map and the shallow feature map to obtain a final feature map; and obtaining a high-resolution image using the final feature map and the low-resolution image. This invention unifies "region-level context," "structure-sensitive sparse aggregation," and "fine-grained texture modeling" into a single lightweight module, achieving both structure preservation and detail recovery at a lower cost.

Description

Technical Field

[0001] This invention relates to the field of digital image processing technology, and in particular to a lightweight image super-resolution method based on superpixel guidance. Background Technology

[0002] Single-image super-resolution aims to reconstruct high-resolution images from low-resolution images to restore sharp edges and fine-grained textures. This task is significant in scenarios such as mobile photography, medical imaging, security surveillance, and streaming media transmission: it can improve visual quality and enhance the input quality of downstream perception tasks at a lower cost when imaging hardware, bandwidth, or storage are limited. Image super-resolution relies not only on local texture refinement but also on cross-regional structural consistency and redundant repetitive textures.

[0003] Traditional convolutional neural networks primarily rely on local convolutions to progressively expand the receptive field, often requiring deeper and wider networks to capture long-range dependencies. This leads to a significant increase in computational and memory requirements, making them difficult to deploy on mobile devices or real-time systems. In recent years, deep learning networks based on global image attention have provided a more direct approach to long-range dependency modeling, with representative works validating the effectiveness of windowed attention in super-resolution and image restoration tasks. However, in lightweight scenarios, self-attention is still limited by resolution-driven increases in computational and memory overhead; simultaneously, fixed window partitioning independent of content weakens the ability to model cross-regional similar textures. Correspondingly, while lightweight attention design can reduce complexity, it still generally faces a key bottleneck: the partitioning of interaction units is mostly content-independent, making it difficult for the model to stably focus computation on truly relevant regional relationships under low-cost conditions, resulting in redundant interactions or irrelevant interference. Summary of the Invention

[0004] In view of the above, the main objective of this invention is to propose a lightweight image super-resolution method based on superpixel guidance to solve the aforementioned technical problems.

[0005] This invention proposes a lightweight image super-resolution method based on superpixel guidance, the method comprising the following steps:

[0006] Step 1: Process the low-resolution image using SLIC to obtain a superpixel label map;

[0007] Step 2: Use lightweight convolution to perform convolution mapping on the low-resolution image to obtain a shallow feature map;

[0008] Step 3: The shallow feature map is used as the input feature map and input into the feature enhancement block composed of the superpixel guided fusion block and the high-frequency attention fusion block. The input feature map is then reconstructed layer by layer using the superpixel label map and optional sample-level sparse budget parameters to obtain the output feature map of the intermediate feature enhancement block.

[0009] Step 4: Use the output feature map of the intermediate feature enhancement block as the input feature map of the next feature enhancement block and perform layer-by-layer reconstruction again. After stacking a predetermined number of feature enhancement blocks, the output feature map of the final feature enhancement block is obtained, and the output feature map of the final feature enhancement block is used as the deep feature map.

[0010] Step 5: Perform cross-layer residual processing on the deep feature map and the shallow feature map to obtain the final feature map;

[0011] Step 6: Add the upsampling residual branches of the final feature map and the low-resolution image to obtain the high-resolution image.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0013] 1. This invention performs superpixel segmentation only once and reuses the group index within the network. It introduces structural priors without introducing the overhead of iterative clustering during inference, thereby improving the effectiveness of long-range interactions and enhancing the quality of super-resolution.

[0014] 2. This invention unifies “region-level context”, “structure-sensitive sparse aggregation” and “fine-grained texture modeling” into a single lightweight module, achieving both structure preservation and detail restoration at a lower cost.

[0015] 3. High-frequency enhancement branches are introduced to strengthen texture and edge response; at the same time, the intermediate activation scale is reduced through superpixel compression and sparse interaction to alleviate the memory pressure under the same input size and improve the feasibility of lightweight deployment. Attached Figure Description

[0016] Figure 1 This is a flowchart of the lightweight image super-resolution method based on superpixel guidance proposed in this invention;

[0017] Figure 2 This is the main network framework diagram of the lightweight image super-resolution method based on superpixel guidance proposed in this invention;

[0018] Figure 3 This is a schematic diagram comparing the performance of the lightweight image super-resolution method based on superpixel guidance proposed in this invention with existing image super-resolution methods.

[0019] Figure 4This diagram illustrates a comparison between the proposed lightweight image super-resolution method based on superpixel guidance and existing methods in terms of running speed, peak memory usage, computational load, and parameter count. Detailed Implementation

[0020] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0021] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0022] Please see Figure 1 and Figure 2 This invention proposes a lightweight image super-resolution method based on superpixel guidance, which includes the following steps:

[0023] Step 1: Process the low-resolution image using SLIC to obtain a superpixel label map;

[0024] In this step, the superpixel label map is generated once by SLIC and reused within the network as a structural prior.

[0025] Step 2: Use lightweight convolution to perform convolution mapping on the low-resolution image to obtain a shallow feature map;

[0026] In step 2, lightweight convolution is used to perform convolution mapping on the low-resolution image to obtain a shallow feature map. The corresponding process has the following relationship:

[0027] ;

[0028] in, This represents a shallow feature map. Indicates a low-resolution image. This indicates convolutional mapping processing.

[0029] Step 3: The shallow feature map is used as the input feature map and input into the feature enhancement block composed of the superpixel guided fusion block and the high-frequency attention fusion block. The input feature map is then reconstructed layer by layer using the superpixel label map and optional sample-level sparse budget parameters to obtain the output feature map of the intermediate feature enhancement block.

[0030] In step 3, the shallow feature map is used as the input feature map and input into the feature enhancement block composed of the superpixel guided fusion block and the high-frequency attention fusion block. The input feature map is then reconstructed layer by layer using the superpixel label map and optional sample-level sparse budget parameters to obtain the output feature map of the intermediate feature enhancement block. The specific steps include the following:

[0031] The input feature map is flattened to obtain the feature sequence;

[0032] Using the feature sequence, the query matrix, key matrix, and value matrix are obtained through linear projection;

[0033] In each superpixel location partitioned by the superpixel label map, mean aggregation is performed using the key matrix and value matrix to obtain the superpixel-level key vector and superpixel-level value vector, respectively.

[0034] Attention is calculated using superpixel-level key vectors and query matrices to obtain the global relevance distribution matrix;

[0035] The global correlation distribution matrix is ​​used as the attention weight, and weighted aggregation is performed in combination with the superpixel-level value vector to obtain compressed branch features;

[0036] The compressed branch features of all pixel locations are stacked to obtain the total compressed branch features;

[0037] Using the number of candidate superpixels retained at each pixel position by the optional sample-level sparse budget parameter, the index with the highest relevance is selected from the superpixel dimension at each pixel position and in each attention head to obtain the superpixel index set;

[0038] Weights corresponding to a subset of the superpixel index set are extracted from the global correlation distribution matrix to obtain subset normalized weights;

[0039] Extract the set corresponding to the superpixel index set from the superpixel-level value vector to obtain the value vector submatrix;

[0040] Sparse aggregation of value vectors in the value vector submatrix is ​​performed using subset normalized weights to obtain the selection branch features;

[0041] Stack the selection branch features of all pixel locations and all attention heads to obtain the total selection branch feature;

[0042] At each pixel location, a local window of fixed size is divided with the pixel as the center. Within the local window, a set of key vectors and a set of value vectors are extracted and obtained. The attention weights within the local window are obtained using the set of key vectors and the query vector. The value vector set is then weighted using the attention weights within the local window to obtain the local window branch features.

[0043] The local branch features at each pixel location are stacked to obtain the total local window branch features;

[0044] The compressed branch gating weight, the selected branch gating weight, and the local window branch gating weight are obtained using the feature sequences, respectively.

[0045] The fused features are obtained by using the compressed branch gating weight, the selected branch gating weight, the local window branch gating weight, the total compressed branch feature, the total selected branch feature, and the total local window branch feature;

[0046] By utilizing fusion features, the output of the superpixel guided fusion block is obtained through linear projection processing;

[0047] After performing lightweight channel projection processing on the output of the superpixel guided fusion block, it is integrated with the feature sequence through residual form to obtain the output feature sequence of the superpixel guided fusion block.

[0048] The output feature sequence of the superpixel guided fusion block is reconstructed in the form of a two-dimensional tensor to obtain the output feature map of the superpixel guided fusion block;

[0049] The output feature map of the superpixel guided fusion block is divided into a predetermined number of local blocks to obtain local block features;

[0050] The local block features are processed by high-frequency enhancement branch and standard multi-head self-attention, respectively, to obtain the output of high-frequency enhancement branch and the output of standard multi-head self-attention. The local block features, the output of high-frequency enhancement branch and the output of standard multi-head self-attention are then superimposed with residuals to obtain the updated local block output features.

[0051] All updated local block output features are recombined to obtain the output feature map of the high-frequency attention fusion block;

[0052] The output feature map of the high-frequency attention fusion block is subjected to channel mixing and nonlinear enhancement through lightweight separable convolution, and then residual processing is performed with the output feature map of the high-frequency attention fusion block to obtain the output feature map of the intermediate feature enhancement block.

[0053] The feature enhancement block consists of a superpixel guided fusion block and a high-frequency attention fusion block, and the superpixel guided fusion block and the high-frequency attention fusion block are combined using residual processing to maintain training stability.

[0054] It should be noted that lightweight single-image super-resolution faces a typical contradiction: "ensuring structural consistency and cross-regional texture propagation requires effective long-range interaction of the network"; however, the computation and memory overhead of standard global self-attention increases significantly with resolution, making it difficult to meet the requirements of a lightweight budget.

[0055] To address this, the present invention proposes a superpixel guided fusion block, which replaces the pixel-to-pixel fully connected attention with a pixel-to-superpixel interaction and combines sparse selection and local compensation to simultaneously model the global structure and local texture within a limited computational space.

[0056] Specifically, in the process of flattening the input feature map to obtain the feature sequence, the following relationship exists:

[0057] ;

[0058] in, Represents the characteristic sequence, This indicates an operation to flatten a two-dimensional feature map. The dimensional space representing a tensor This indicates the total number of spatial locations after flattening. Indicates the height of the input feature map. This indicates the width of the input feature map. Indicates the size of the batch processing. Indicates the number of channels;

[0059] The superpixel guided fusion block consists of three complementary branches: a compression branch, a selection branch, and a local window branch. All three branches share the same set of linear projections and perform gated fusion at the end.

[0060] In the process of obtaining the query matrix, key matrix, and value matrix through linear projection using feature sequences, the following relationship exists:

[0061] ;

[0062] in; This represents the query matrix, used to characterize the information retrieval needs of the current location for other locations; The key matrix represents the feature descriptions that can be matched at each position; The value matrix is ​​used to characterize the content features involved in the weighted aggregation. This represents the learnable linear projection matrix that maps the input feature map to the query space. This represents the learnable linear projection matrix that maps the input feature map to the key space. This represents the learnable linear projection matrix that maps the input feature map to the value space;

[0063] Compressed branches:

[0064] In the process of performing mean aggregation on the key matrix and value matrix at each superpixel location partitioned by the superpixel label map to obtain the superpixel-level key vector and superpixel-level value vector respectively, the following relationship exists:

[0065] ;

[0066] in, Indicates by the first The superpixel-level key vector is obtained by aggregating the features of all pixels within a superpixel using the mean. Indicates by the first The superpixel-level value vector obtained by aggregating the features of all pixels within a superpixel using the mean. Indicates the first A superpixel contains a set of pixels. Indicates the first The number of pixels within the set of pixels contained in a superpixel. Indicates the first The key vector corresponding to each pixel position. Indicates the first The value vector corresponding to each pixel position;

[0067] In the process of using superpixel-level key vectors and query matrices to perform attention calculations to obtain the global relevance distribution matrix, the following relationship exists:

[0068] ;

[0069] in, This represents the global correlation distribution matrix between pixel queries and superpixel-level key vectors. Represents a superpixel-level key vector; This represents the channel dimension of the key vector in each attention head, used to scale the dot product results to stabilize the training process; Indicates the transpose symbol. This represents mapping a real number vector to a probability distribution;

[0070] In the process of using the global correlation distribution matrix as attention weights and combining it with superpixel-level value vectors for weighted aggregation to obtain compressed branch features, the following relationship exists:

[0071] ;

[0072] in, Indicates the first The compressed branch features are obtained by weighted convergence of all superpixel level value vectors at each pixel location. Indicates the first The position of the pixel is related to the _th pixel Attention weights for each superpixel;

[0073] Select a branch:

[0074] This branch transforms the pixel-to-superpixel correlation distribution obtained from the compressed branch into sparse interaction. That is, for each pixel location, it no longer interacts with all superpixel features, but only selects the k superpixel features with the highest correlation for weighted aggregation, thereby significantly reducing the redundant computation of subsequent interactions.

[0075] In the process of selecting the most relevant index from the superpixel dimension at each pixel location and within each attention head to obtain the superpixel index set, the following relationship exists:

[0076] ;

[0077] in, Indicates the first The pixel position The set of Top-k (default is 3 in this invention) superpixel indices selected under each attention head. express The Middle The pixel position The entire row of relevance vectors corresponding to each attention head. This indicates the number of candidate superpixels retained at each pixel location. This means selecting the largest value from the input. The index corresponding to each element;

[0078] In the process of extracting weights corresponding to a subset of the superpixel index set from the global correlation distribution matrix to obtain the subset-normalized weights, the following relationship exists:

[0079] ;

[0080] in, Indicates the first The first one in the attention. The position of the pixel is related to the _th pixel Normalized weights for a subset of candidate superpixels express The Middle The first one in the attention. The position of the pixel is related to the _ ... The original correlation scores between candidate superpixels express The Middle The first one in the attention. The position of the pixel is related to the _ ... The original correlation scores between candidate superpixels This indicates processing using an exponential function with the natural constant as the base;

[0081] In the process of extracting the set corresponding to the superpixel index set from the superpixel-level value vector to obtain the value vector submatrix, the following relationship exists:

[0082] ;

[0083] in, Indicates the first All superpixel-level value vectors in each attention head Indicates by following from Extracted from The value vector submatrix composed of candidate superpixel value vectors This indicates an index extraction operation. This represents the dimension of the value vector submatrix. This represents the channel dimension of the value vector in each attention head;

[0084] In the process of sparsely aggregating the value vectors in the value vector submatrix using subset normalized weights to obtain the branch selection features, the following relationship exists:

[0085] ;

[0086] in, Indicates the first The first thing to pay attention to. The selected branch feature is obtained by aggregating coefficient superpixels at each pixel location. Indicates the first The first one in the attention. The superpixel-level value vector corresponding to each candidate superpixel. This indicates that the output dimension is ;

[0087] Local window branch:

[0088] This branch was introduced to improve the ability to model insufficient textures and fine-grained details, focusing only on the local neighborhood of a pixel location. Internal attention interaction;

[0089] In the process of obtaining attention weights within a local window using the key vector set and query vector, and then weighting the value vector set using these attention weights to obtain the local window branch features, the following relationship exists:

[0090] ;

[0091] in, Indicates the first Attention weights for a local window centered at a pixel location. Indicates the first A query vector for each pixel position. Indicates the first The set of key vectors corresponding to all positions within a local window centered at a given pixel location. Indicates the first Local window branch features within a local window centered at a pixel location. Indicates the first The set of value vectors corresponding to all positions within a local window centered at a pixel location;

[0092] In local window branches, the window size is fixed (3×3 in this invention). The constant value makes the complexity of local window branching grow approximately linearly with the input image resolution, thereby enhancing the expression of local texture and detail with lower overhead;

[0093] Lightweight gating fusion:

[0094] To adaptively fuse the three complementary branches, this invention introduces pixel-by-pixel lightweight gating to dynamically weight the outputs of the three branches based on content relevance.

[0095] In the process of obtaining the compressed branch gating weight, the selected branch gating weight, and the local window branch gating weight using the feature sequence, the following relationship exists:

[0096] ;

[0097] in, This represents the three-way gating coefficient tensor generated by the gating module. This indicates that the process has been handled by the Sigmoid function. This represents the learnable linear mapping parameters of the gating module. express Size, This indicates the gating weight corresponding to the compression branch. This indicates the gating weight corresponding to the selected branch. This represents the gating weight corresponding to the local window branch. This indicates that along the last dimension They are divided into three sub-weights;

[0098] In the process of obtaining the fused features using compressed branch gating weights, selected branch gating weights, local window branch gating weights, total compressed branch features, total selected branch features, and total local window branch features, the following relationship exists:

[0099] ;

[0100] in, This represents the fused features of the three branches after pixel-by-pixel gated modulation. This represents element-wise multiplication. Indicates the overall compression branch characteristics. Indicates the overall selection branch feature, Indicates the overall local window branching characteristics;

[0101] In the process of obtaining the output of the superpixel guided fusion block through linear projection using fusion features, the following relationship exists:

[0102] ;

[0103] in, Indicates the output projection matrix. This indicates the output of the superpixel guided fusion block. This indicates superpixel-guided fusion attention processing. Presentation layer normalization processing, This represents a superpixel label image. Indicates optional sample-level sparse budget parameters;

[0104] After performing lightweight channel projection processing on the output of the superpixel guided fusion block and integrating it with the feature sequence through residual form to obtain the output feature sequence of the superpixel guided fusion block, the following relationship exists:

[0105]

[0106] in, Represents the intermediate feature sequence. This indicates lightweight channel projection processing. This represents the output feature sequence of the superpixel guided fusion block. This indicates feedforward mapping processing with convolutional mixing;

[0107] Superpixel-guided fusion blocks achieve structural consistency and cross-regional information transfer within a limited budget through superpixel-guided compression, sparse selection, and window interaction. However, under lightweight settings, high-gradient details such as textures and edges may still exhibit insufficient responsiveness, resulting in textures becoming "blurred" or edges not being sharp enough.

[0108] To this end, the present invention further introduces a high-frequency attention fusion block in each feature enhancement block as a local high-frequency compensation module after the superpixel guided fusion block, so as to enhance fine-grained texture and edge expression with less additional overhead.

[0109] In the process of dividing the output feature map of the superpixel guided fusion block into a predetermined number of local blocks to obtain local block features, the following relationship exists:

[0110] ;

[0111] in, This represents the length of the local block's side. This represents the total number of local blocks. Indicates the index of a local block. This indicates that the feature map will be processed by a non-overlapping local block segmentation operator. Indicates the first Local block features, This represents the output feature map of the superpixel guided fusion block;

[0112] For each local block, standard multi-head self-attention and high-frequency branch computation are performed simultaneously. Standard multi-head self-attention is used to supplement the local relationship modeling that is difficult to cover by high-frequency branches. At the same time, the high-frequency attention fusion block does not introduce additional gating or complex weight prediction, but adopts a simple and stable three-way residual superposition.

[0113] In the process of residual superposition of local block features, the output of high-frequency enhancement branches, and the output of standard multi-head self-attention processing to obtain updated local block output features, the following relationship exists:

[0114] ;

[0115] in, This represents the updated local block output features. This indicates that the process has undergone standard multi-head self-attention processing. This indicates high-frequency enhanced branching processing after text design. This represents a learnable linear mapping implemented by pointwise convolutions for channel recovery. This represents a learnable linear mapping implemented by pointwise convolution for channel compression. This indicates that the process has been performed using a non-linear activation function. This indicates that the data has undergone channel-wise convolution processing within depthwise separable convolution.

[0116] In the process of reorganizing all updated local block output features to obtain the output feature map of the high-frequency attention fusion block, the following relationship exists:

[0117] ;

[0118] in, This represents the output feature map of the high-frequency attention fusion block, i.e.; This represents the inverse operation of processing the feature map according to the non-overlapping local block segmentation operator;

[0119] In the process of performing channel blending and nonlinear enhancement on the output feature map of the high-frequency attention fusion block using lightweight separable convolution, and then performing residual processing with the output feature map of the high-frequency attention fusion block to obtain the output feature map of the intermediate feature enhancement block, the following relationship exists:

[0120] ;

[0121] in, This represents the output feature map of the intermediate feature enhancement block.

[0122] It should be noted that the high-frequency attention fusion block aims to supplement the superpixel-guided fusion block's responsiveness at textures and edges with explicit high-frequency enhancement, while maintaining a lightweight design; this invention employs a structure of "depth-separable convolution + channel compression / restore" to construct local high-frequency mapping:

[0123] Overall, the high-frequency attention fusion block provides explicit local high-frequency compensation through high-frequency enhancement branches and supplements it with standard multi-head attention within the block as a necessary local relation modeling, enabling the network to simultaneously take into account both structure and texture reconstruction with a lightweight setup.

[0124] Step 4: Use the output feature map of the intermediate feature enhancement block as the input feature map of the next feature enhancement block and perform layer-by-layer reconstruction again. After stacking a predetermined number of feature enhancement blocks, the output feature map of the final feature enhancement block is obtained, and the output feature map of the final feature enhancement block is used as the deep feature map.

[0125] Specifically, the present invention uses the output feature map of the previous feature enhancement block (the output feature map of the intermediate feature enhancement block) as the input feature map of the next feature enhancement block for layer-by-layer reconstruction.

[0126] After repeating L-1 times, the output feature map of the Lth feature enhancement block (the final output feature map of the feature enhancement block) is obtained, and the output feature map of the Lth feature enhancement block is used as the deep feature map.

[0127] It should be noted that the value of L in this invention is 8.

[0128] Step 5: Perform cross-layer residual processing on the deep feature map and the shallow feature map to obtain the final feature map;

[0129] In step 5, cross-layer residual processing is performed on the deep feature map and the shallow feature map to obtain the final feature map. The corresponding process has the following relationship:

[0130] ;

[0131] in, This represents the final feature map. Represents deep feature maps. Indicates cross-layer residual processing

[0132] Step 6: Add the upsampling residual branches of the final feature map and the low-resolution image to obtain the high-resolution image;

[0133] In step 6, the final feature map and the low-resolution image are summed using upsampling residual branches to obtain the high-resolution image. The corresponding process has the following relationship:

[0134] ;

[0135] in, This indicates that the process is performed using the bilinear interpolation upsampling operator. This indicates processing via an image reconstruction function. This indicates a high-resolution image.

[0136] This invention uses DIV2K as the training dataset, which contains 800 training images. It also uses five benchmarks: Set5, Set14, BSD100, Urban100, and Manga109, with magnification factors of ×2, ×3, and ×4, respectively. The images are transformed to the YCbCr color space, and the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) in the Y channel are calculated to evaluate the quality of the reconstructed image.

[0137] During training (performed on an NVIDIA GeForce RTX 4090 graphics processor with a batch size of 32), the training data angles were rotated [0, 90, 180, 270] for data augmentation. Furthermore, the original image was cropped into 64×64 blocks for network training. The model was trained using the L1 loss function, and ADAM[] was used in… =0.9 and Optimization is performed when the value is 0.99, where, and denoted as the exponential decay rates of the first-moment estimate and the second-moment estimate, respectively;

[0138] The initial learning rate is set to 2×10⁻⁴, and the learning rate is halved when the training iterations reach a certain milestone.

[0139] This invention uses pre-trained weights from a 2× model to initialize the weights of 3× and 4×, and halves the learning rate and total number of training iterations to reduce training time. During training, the corresponding superpixel labels are obtained from the input training set, and only the average size of the superpixels is set at the beginning of training.

[0140] Please see Figure 3 , Figure 3 The paper presents metrics for testing the image restoration level of this network at 2x magnification on five widely used benchmark datasets, and compares the invention with existing lightweight image super-resolution models, including CARN, IMDN, OSFFNet, ELAN-light, IPG-Tiny, SPIN, catanet, SRFormer, mambaIR, SwinIR-light, SMFANet, HiT-SNG, Omni-SR, LAMNET, and CRAFT. Detailed information on the model parameters and the number of training datasets for each model is provided.

[0141] like Figure 3 As shown, in the ×2 super-resolution reconstruction task, the SGFA method of this invention achieved superior reconstruction results on multiple public test sets, demonstrating excellent overall performance. Specifically, the peak signal-to-noise ratios of this invention on the Set5, Set14, and Urban100 datasets reached 38.29 dB, 34.05 dB, and 33.09 dB, respectively, and the structural similarities reached 0.9617, 0.9219, and 0.9372, respectively, all of which are the best results in the table. On the B100 dataset, this invention achieved results of 32.36 dB and 0.9022, which are second-best, with only a very small difference from the best results. On the Manga109 dataset, the structural similarity of this invention reached 0.9787, which is the best result, and the peak signal-to-noise ratio reached 39.34 dB, which is only 0.05 dB different from the best value of 39.39 dB.

[0142] It can be seen that the method of the present invention not only has stable recovery ability on different types of datasets, but also performs outstandingly in texture detail reconstruction, structural information preservation and model generalization performance.

[0143] It should be noted that, Figure 3 The red values ​​represent the best results compared to existing technologies, while the blue values ​​represent the second-best results.

[0144] Please see Figure 4 ,pass Figure 4As shown, under the condition of input image size (3, 256, 256), the method of this invention achieves a good balance between restoration effect and computational resource consumption in the Set14 (×2) restoration task. Specifically, the parameter quantity of this invention is 577K, the computation quantity is 39.7G, the average running time is 78 ms, and the peak memory usage is 648 MB. Among these, the computation quantity, average running time, and peak memory usage are all better than the comparative methods, indicating that this invention has significant advantages in inference efficiency and resource consumption. In terms of restoration effect, this method achieves a peak signal-to-noise ratio of 34.05 dB and a structural similarity of 0.9219, both of which are the best results in the table. In particular, compared with CATANet, this method further improves the peak signal-to-noise ratio and structural similarity while reducing the computation quantity from 46.8G to 39.7G, the average running time from 86 ms to 78 ms, and the peak memory usage from 2180 MB to 648 MB.

[0145] This demonstrates that the method of the present invention can not only guarantee high image restoration quality, but also significantly reduce the computation and memory overhead during model deployment, making it more suitable for application in real-world resource-constrained scenarios.

[0146] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0147] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0148] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A lightweight image super-resolution method based on superpixel guidance, characterized in that, The method includes the following steps: Step 1: Process the low-resolution image using SLIC to obtain a superpixel label map; Step 2: Use lightweight convolution to perform convolution mapping on the low-resolution image to obtain a shallow feature map; Step 3: The shallow feature map is used as the input feature map and input into the feature enhancement block composed of the superpixel guided fusion block and the high-frequency attention fusion block. The input feature map is then reconstructed layer by layer using the superpixel label map and optional sample-level sparse budget parameters to obtain the output feature map of the intermediate feature enhancement block. The specific steps include the following: The input feature map is flattened to obtain the feature sequence; Using the feature sequence, the query matrix, key matrix, and value matrix are obtained through linear projection; In each superpixel location partitioned by the superpixel label map, mean aggregation is performed using the key matrix and value matrix to obtain the superpixel-level key vector and superpixel-level value vector, respectively. Attention is calculated using superpixel-level key vectors and query matrices to obtain the global relevance distribution matrix; The global correlation distribution matrix is ​​used as the attention weight, and weighted aggregation is performed in combination with the superpixel-level value vector to obtain compressed branch features; The compressed branch features of all pixel locations are stacked to obtain the total compressed branch features; Using the number of candidate superpixels retained at each pixel position by the optional sample-level sparse budget parameter, the index with the highest relevance is selected from the superpixel dimension at each pixel position and in each attention head to obtain the superpixel index set; Weights corresponding to a subset of the superpixel index set are extracted from the global correlation distribution matrix to obtain subset normalized weights; Extract the set corresponding to the superpixel index set from the superpixel-level value vector to obtain the value vector submatrix; Sparse aggregation of value vectors in the value vector submatrix is ​​performed using subset normalized weights to obtain the selection branch features; Stack the selection branch features of all pixel locations and all attention heads to obtain the total selection branch feature; At each pixel location, a local window of fixed size is divided with the pixel as the center. Within the local window, a set of key vectors and a set of value vectors are extracted and obtained. The attention weights within the local window are obtained using the set of key vectors and the query vector. The value vector set is then weighted using the attention weights within the local window to obtain the local window branch features. The local branch features at each pixel location are stacked to obtain the total local window branch features; The compressed branch gating weight, the selected branch gating weight, and the local window branch gating weight are obtained using the feature sequences, respectively. The fused features are obtained by using the compressed branch gating weight, the selected branch gating weight, the local window branch gating weight, the total compressed branch feature, the total selected branch feature, and the total local window branch feature; By utilizing fusion features, the output of the superpixel guided fusion block is obtained through linear projection processing; After performing lightweight channel projection processing on the output of the superpixel guided fusion block, it is integrated with the feature sequence through residual form to obtain the output feature sequence of the superpixel guided fusion block. The output feature sequence of the superpixel guided fusion block is reconstructed in the form of a two-dimensional tensor to obtain the output feature map of the superpixel guided fusion block; The output feature map of the superpixel guided fusion block is divided into a predetermined number of local blocks to obtain local block features; The local block features are processed by high-frequency enhancement branch and standard multi-head self-attention, respectively, to obtain the output of high-frequency enhancement branch and the output of standard multi-head self-attention. The local block features, the output of high-frequency enhancement branch and the output of standard multi-head self-attention are then superimposed with residuals to obtain the updated local block output features. All updated local block output features are recombined to obtain the output feature map of the high-frequency attention fusion block; The output feature map of the high-frequency attention fusion block is subjected to channel mixing and nonlinear enhancement through lightweight separable convolution, and then residual processing is performed with the output feature map of the high-frequency attention fusion block to obtain the output feature map of the intermediate feature enhancement block. Step 4: Use the output feature map of the intermediate feature enhancement block as the input feature map of the next feature enhancement block and perform layer-by-layer reconstruction again. After stacking a predetermined number of feature enhancement blocks, the output feature map of the final feature enhancement block is obtained, and the output feature map of the final feature enhancement block is used as the deep feature map. Step 5: Perform cross-layer residual processing on the deep feature map and the shallow feature map to obtain the final feature map; Step 6: Add the upsampling residual branches of the final feature map and the low-resolution image to obtain the high-resolution image.

2. The lightweight image super-resolution method based on superpixel guidance according to claim 1, characterized in that, In step 2, lightweight convolution is used to perform convolution mapping on the low-resolution image to obtain a shallow feature map. The corresponding process has the following relationship: ； in, This represents a shallow feature map. Indicates a low-resolution image. This indicates convolutional mapping processing.

3. The lightweight image super-resolution method based on superpixel guidance according to claim 2, characterized in that, In the process of flattening the input feature map to obtain the feature sequence, the following relationship exists: ； in, Represents the characteristic sequence, This indicates an operation to flatten a two-dimensional feature map. The dimensional space representing a tensor This indicates the total number of spatial locations after flattening. Indicates the height of the input feature map. This indicates the width of the input feature map. Indicates the size of the batch processing. Indicates the number of channels; In the process of obtaining the query matrix, key matrix, and value matrix through linear projection using feature sequences, the following relationship exists: ； in, Represents the query matrix. Represents the key matrix, Represents a value matrix, This represents the learnable linear projection matrix that maps the input feature map to the query space. This represents the learnable linear projection matrix that maps the input feature map to the key space. This represents the learnable linear projection matrix that maps the input feature map to the value space; In the process of performing mean aggregation on the key matrix and value matrix at each superpixel location partitioned by the superpixel label map to obtain the superpixel-level key vector and superpixel-level value vector respectively, the following relationship exists: ； in, Indicates by the first The superpixel-level key vector is obtained by aggregating the features of all pixels within a superpixel using the mean. Indicates by the first The superpixel-level value vector obtained by aggregating the features of all pixels within a superpixel using the mean. Indicates the first A superpixel contains a set of pixels. Indicates the first The number of pixels within the set of pixels contained in a superpixel. Indicates the first The key vector corresponding to each pixel position. Indicates the first The value vector corresponding to each pixel position.

4. The lightweight image super-resolution method based on superpixel guidance according to claim 3, characterized in that, In the process of using superpixel-level key vectors and query matrices to perform attention calculations to obtain the global relevance distribution matrix, the following relationship exists: ； in, This represents the global correlation distribution matrix between pixel queries and superpixel-level key vectors. Represents a superpixel-level key vector. This represents the channel dimension of the key vector in each attention head. Indicates the transpose symbol. This represents mapping a real number vector to a probability distribution; In the process of using the global correlation distribution matrix as attention weights and combining it with superpixel-level value vectors for weighted aggregation to obtain compressed branch features, the following relationship exists: ； in, Indicates the first The compressed branch features are obtained by weighted convergence of all superpixel level value vectors at each pixel location. Indicates the first The position of the pixel is related to the _th pixel Attention weights for each superpixel; In the process of selecting the most relevant index from the superpixel dimension at each pixel location and within each attention head to obtain the superpixel index set, the following relationship exists: ； in, Indicates the first The pixel position The set of Top-k superpixel indices selected under each attention head. express The Middle The pixel position The entire row of relevance vectors corresponding to each attention head. This indicates the number of candidate superpixels retained at each pixel location. This means selecting the largest value from the input. The index corresponding to each element.

5. The lightweight image super-resolution method based on superpixel guidance according to claim 4, characterized in that, In the process of extracting weights corresponding to a subset of the superpixel index set from the global correlation distribution matrix to obtain the subset-normalized weights, the following relationship exists: ； in, Indicates the first The first one in the attention. The position of the pixel is related to the _th pixel Normalized weights for a subset of candidate superpixels express The Middle The first one in the attention. The position of the pixel is related to the _ ... The original correlation scores between candidate superpixels express The Middle The first one in the attention. The position of the pixel is related to the _ ... The original correlation scores between candidate superpixels This indicates processing using an exponential function with the natural constant as the base; In the process of extracting the set corresponding to the superpixel index set from the superpixel-level value vector to obtain the value vector submatrix, the following relationship exists: ； in, Indicates the first All superpixel-level value vectors in each attention head Indicates by following from Extracted from The value vector submatrix composed of candidate superpixel value vectors This indicates an index extraction operation. This represents the dimension of the value vector submatrix. This represents the channel dimension of the value vector in each attention head; In the process of sparsely aggregating the value vectors in the value vector submatrix using subset normalized weights to obtain the branch selection features, the following relationship exists: ； in, Indicates the first The first thing to pay attention to. The selected branch feature is obtained by aggregating coefficient superpixels at each pixel location. Indicates the first The first one in the attention. The superpixel-level value vector corresponding to each candidate superpixel. This indicates that the output dimension is ; In the process of obtaining attention weights within a local window using the key vector set and query vector, and then weighting the value vector set using these attention weights to obtain the local window branch features, the following relationship exists: ； in, Indicates the first Attention weights for a local window centered at a pixel location. Indicates the first A query vector for each pixel position. Indicates the first The set of key vectors corresponding to all positions within a local window centered at a given pixel location. Indicates the first Local window branch features within a local window centered at a pixel location. Indicates the first The set of value vectors corresponding to all positions within a local window centered at a pixel location.

6. The lightweight image super-resolution method based on superpixel guidance according to claim 5, characterized in that, In the process of obtaining the compressed branch gating weight, the selected branch gating weight, and the local window branch gating weight using the feature sequence, the following relationship exists: ； in, This represents the three-way gating coefficient tensor generated by the gating module. This indicates that the signal has been processed by the Sigmoid function. This represents the learnable linear mapping parameters of the gating module. express Size, This indicates the gating weight corresponding to the compression branch. This indicates the gating weight corresponding to the selected branch. This represents the gating weight corresponding to the local window branch. Indicates along the last dimension They are divided into three sub-weights; In the process of obtaining the fused features using compressed branch gating weights, selected branch gating weights, local window branch gating weights, total compressed branch features, total selected branch features, and total local window branch features, the following relationship exists: ； in, This represents the fused features of the three branches after pixel-by-pixel gated modulation. This represents element-wise multiplication. Indicates the overall compression branch characteristics. Indicates the overall selection branch feature, Indicates the overall local window branching characteristics; In the process of obtaining the output of the superpixel guided fusion block through linear projection using fusion features, the following relationship exists: ； in, Indicates the output projection matrix. This indicates the output of the superpixel guided fusion block. This indicates superpixel-guided fusion attention processing. Presentation layer normalization processing, This represents a superpixel label image. This represents the optional sample-level sparse budget parameter.

7. The lightweight image super-resolution method based on superpixel guidance according to claim 6, characterized in that, After performing lightweight channel projection processing on the output of the superpixel guided fusion block and integrating it with the feature sequence through residual form to obtain the output feature sequence of the superpixel guided fusion block, the following relationship exists: ； in, Represents the intermediate feature sequence. This indicates lightweight channel projection processing. This represents the output feature sequence of the superpixel guided fusion block. This indicates feedforward mapping processing with convolutional mixing; In the process of dividing the output feature map of the superpixel guided fusion block into a predetermined number of local blocks to obtain local block features, the following relationship exists: ； in, This represents the length of the local block's side. This represents the total number of local blocks. Indicates the index of a local block. This indicates that the feature map will be processed by a non-overlapping local block segmentation operator. Indicates the first Local block features, This represents the output feature map of the superpixel guided fusion block; In the process of residual superposition of local block features, the output of high-frequency enhancement branches, and the output of standard multi-head self-attention processing to obtain updated local block output features, the following relationship exists: ； in, This represents the updated local block output features. This indicates that the process has undergone standard multi-head self-attention processing. This indicates high-frequency enhanced branching processing after text design. This represents a learnable linear mapping implemented by pointwise convolutions for channel recovery. This represents a learnable linear mapping implemented by pointwise convolution for channel compression. This indicates that the process has been performed using a non-linear activation function. This indicates that the data has undergone channel-wise convolution processing within depthwise separable convolution. In the process of reorganizing all updated local block output features to obtain the output feature map of the high-frequency attention fusion block, the following relationship exists: ； in, This represents the output feature map of the high-frequency attention fusion block. This represents the inverse operation of processing the feature map according to the non-overlapping local block segmentation operator; In the process of performing channel blending and nonlinear enhancement on the output feature map of the high-frequency attention fusion block using lightweight separable convolution, and then performing residual processing with the output feature map of the high-frequency attention fusion block to obtain the output feature map of the intermediate feature enhancement block, the following relationship exists: ； in, This represents the output feature map of the intermediate feature enhancement block.

8. The lightweight image super-resolution method based on superpixel guidance according to claim 7, characterized in that, In step 5, cross-layer residual processing is performed on the deep feature map and the shallow feature map to obtain the final feature map. The corresponding process has the following relationship: ； in, This represents the final feature map. Represents deep feature maps. This indicates the handling of cross-layer residuals.

9. The lightweight image super-resolution method based on superpixel guidance according to claim 8, characterized in that, In step 6, the final feature map and the low-resolution image are summed by upsampling residual branches to obtain the high-resolution image. The corresponding process has the following relationship: ； in, This indicates that the process is performed using the bilinear interpolation upsampling operator. This indicates processing via an image reconstruction function. This indicates a high-resolution image.

Images