Lightweight image super-resolution method and device based on adaptive token selection

Abstract

The application discloses a lightweight image super-resolution method and device based on adaptive token selection, relates to the technical field of image super-resolution reconstruction, and comprises the following steps: S1, a lightweight image super-resolution network is constructed; after shallow layer features of an input low-resolution image are extracted through a first convolutional layer, the low-resolution image is input into a plurality of adaptive token selection blocks connected in sequence for deep layer feature extraction, and then the low-resolution image is input into a second convolutional layer; the output of the second convolutional layer is added to the output of the shallow layer features, and then the added output is input into an up-sampling module for up-sampling to obtain a reconstructed image; S2, the network is trained; and S3, a trained network is used to perform super-resolution reconstruction on a low-resolution single-frame image. In the adaptive token selection block, sparse space self-attention and channel self-attention based on adaptive token selection are introduced, so that the redundant correlation interference between tokens is reduced, the redundant information interference of the channel dimension is reduced, and key features are enhanced.

Description

Technical Field

[0001] This invention relates to the field of image super-resolution reconstruction technology, and in particular to a lightweight image super-resolution method and apparatus based on adaptive token selection. Background Technology

[0002] Single-frame super-resolution (SISR) is widely used in computer vision, such as in medical imaging, video surveillance, remote sensing, and video transmission. SISR generates a corresponding high-resolution (HR) image from an existing low-resolution (LR) image through software processing. With the development of deep learning, methods based on convolutional neural networks (CNNs) have far surpassed traditional interpolation algorithms and can learn more accurate mapping relationships between HR-LR image patches, resulting in higher-quality reconstructed HR images. Therefore, CNN-based methods are currently the main approach in single-frame super-resolution research.

[0003] Deep learning-based super-resolution methods can be broadly categorized into two types. The first type is based on generative adversarial networks (GANs). These methods optimize perceptual loss to make the generated high-resolution (HR) images more consistent with human subjective visual perception. However, the HR images reconstructed by these algorithms have lower PSNR and SSIM scores, and their detail texture differs significantly from the original images, thus exhibiting noticeable shortcomings in practical applications.

[0004] The second type places greater emphasis on the details and texture features of the reconstructed image, and its objective metrics are generally higher than the first method. However, this type of method still has some problems. First, to improve the image reconstruction quality, the model often needs to stack a large number of modules to increase the network depth, which makes model training more difficult and time-consuming. Second, because some super-resolution algorithms lack consideration and research on feature extraction modules, the extracted depth features are weak. In addition, these models lack the ability to adaptively distinguish between important and redundant features, treating all feature information in the image equally. Therefore, insufficient feature extraction can cause problems such as loss of detail and blurred edges in the reconstructed image. Summary of the Invention

[0005] To address the above problems, this invention proposes a lightweight image super-resolution method and apparatus based on adaptive token selection. It introduces an adaptive token selection mechanism to reduce redundant correlation interference between tokens and extract more representative enhanced features, overcoming the problem of overly redundant feature information extracted by the original classic super-resolution model. Furthermore, by enhancing the representativeness of the captured features through token selection channel self-attention and sparse space self-attention, it effectively improves the reconstruction effect of clear image details.

[0006] On the one hand, a lightweight image super-resolution method based on adaptive token selection has the following specific steps:

[0007] S1, Construct a lightweight image super-resolution network; the lightweight image super-resolution network includes a first convolutional layer, several adaptive token selection blocks, a second convolutional layer, and an upsampling module;

[0008] After the low-resolution image is processed by the first convolutional layer to extract shallow features, it is input into several sequentially connected adaptive token selection blocks for deep feature extraction. Then it is input into the second convolutional layer. The shallow features and the output of the second convolutional layer are added together and then input into the upsampling module for upsampling to obtain the reconstructed image.

[0009] The adaptive token selection block includes several ATS Transformer layers and a third convolutional layer. The input of the adaptive token selection block is processed through a series of interconnected ATS Transformer layers for deep feature extraction. The output of the last ATS Transformer layer is then added to the input of the adaptive token selection block after passing through the third convolutional block to obtain the output of the adaptive token selection block. The Transformer layer includes a sparse spatial self-attention module, a channel self-attention module, a shift-window-based multi-head self-attention module, and an FFN layer. The sparse spatial self-attention module performs spatial redundancy removal on the input shallow features to obtain spatially deredundant features. The sparse spatial self-attention module uses a prior dictionary in the sparse spatial self-attention calculation process. Based on the learnable parameters of the prior dictionary, a prior dictionary for the sparse spatial self-attention module in the next ATSTransformer layer is constructed. The channel self-attention module performs channel redundancy removal on the input shallow features to obtain channel-deredundant features. The shift-window-based multi-head self-attention module enhances the input shallow features to obtain multi-head attention-enhanced features. The spatially deredundant features, channel-deredundant features, and multi-head attention-enhanced features are summed, added to the input of the ATS Transformer layer, and then input into the FFN layer feedforward network for feature enhancement. The output is used as the output of the ATS Transformer layer. S2: The lightweight image super-resolution network is trained to obtain a trained lightweight image super-resolution network.

[0010] S3 uses a trained lightweight image super-resolution network to perform super-resolution reconstruction on a low-resolution single-frame image, resulting in a reconstructed image.

[0011] Preferably, the sparse space self-attention module is as follows:

[0012] Input features are transformed by linear transformation Mapped to query token matrix Where t represents the parameter for reducing the dimension of the feature map to reduce computational cost; C represents the dimension of the token; and N represents the number of tokens.

[0013] Introducing a priori dictionary A linear transformation maps the prior dictionary to dictionary keys (tokens). and dictionary value token Where M represents the number of tokens in the dictionary;

[0014] Will and Perform matrix multiplication and redundancy removal to obtain the redundancy-removed self-attention map, represented as:

[0015] ;

[0016] in, express and The self-attention map obtained after redundancy removal from the matching; Indicates to Squaring the result; The weight coefficients of the sparse processing branches are learnable weight parameters. This represents the weight coefficients of the traditional, normalized branches, and is a learnable weight parameter. express function;

[0017] The self-attention map after redundancy removal With dictionary value token Multiplying them together yields the spatially redundant features.

[0018] Preferably, the channel self-attention module includes several parallel branch networks; the input of all branch networks is the input of the channel self-attention module; each branch network is specifically as follows:

[0019] Branch networks map inputs to queries, keys, and values. , , And transpose them respectively to obtain , and ;in, Indicates the number of tokens; Indicates the number of channels;

[0020] Will and Perform matrix multiplication to obtain the self-attention score matrix. ;

[0021] Set a scaling factor K, and based on the descending order of the relevance scores of each element in the attention score matrix, keep the first K elements of the self-attention score matrix unchanged, while setting the other scores to zero, to obtain... ;

[0022] Will and Matrix multiplication is used to obtain features ; indicates as:

[0023] ;

[0024] in, express function; Indicates the first A branch network;

[0025] Will Transpose to obtain the output of the branch network ;

[0026] The outputs of all branch networks with different scaling factor K values ​​are summed to obtain the channel redundancy-free features.

[0027] Preferably, the number of parallel branch networks is 4.

[0028] Preferably, the upsampling module is a subpixel convolutional layer.

[0029] On the other hand, a lightweight image super-resolution device based on adaptive token selection includes the following:

[0030] A network construction module is used to construct a lightweight image super-resolution network; the lightweight image super-resolution network includes a first convolutional layer, several adaptive token selection blocks, a second convolutional layer, and an upsampling module;

[0031] After the low-resolution image is processed by the first convolutional layer to extract shallow features, it is input into several sequentially connected adaptive token selection blocks for deep feature extraction. Then it is input into the second convolutional layer. The shallow features and the output of the second convolutional layer are added together and then input into the upsampling module for upsampling to obtain the reconstructed image.

[0032] The adaptive token selection block includes several ATS Transformer layers and a third convolutional layer. The input of the adaptive token selection block is processed through a series of interconnected ATS Transformer layers for deep feature extraction. The output of the last ATS Transformer layer is then added to the input of the adaptive token selection block after passing through the third convolutional block to obtain the output of the adaptive token selection block. The Transformer layer includes a sparse spatial self-attention module, a channel self-attention module, a shift-window-based multi-head self-attention module, and an FFN layer. The sparse spatial self-attention module performs spatial redundancy removal on the input shallow features to obtain spatially de-redundant features. The sparse spatial self-attention module uses a prior dictionary in the sparse spatial self-attention calculation process. Based on the learnable parameters of the prior dictionary, a prior dictionary for the sparse spatial self-attention module in the next ATSTransformer layer is constructed. The channel self-attention module performs channel redundancy removal on the input shallow features to obtain channel-de-redundant features. The shift-window-based multi-head self-attention module enhances the input shallow features to obtain multi-head attention-enhanced features. The spatially de-redundant features, the channel-de-redundant features, and the multi-head attention-enhanced features are summed, added to the input of the ATS Transformer layer, and then input into the FFN layer feedforward network for feature enhancement. The output is then used as the output of the ATS Transformer layer.

[0033] The network training module is used to train the lightweight image super-resolution network to obtain the trained lightweight image super-resolution network.

[0034] The image reconstruction module is used to reconstruct low-resolution single-frame images using a trained lightweight image super-resolution network to obtain reconstructed images.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] (1) This invention provides an adaptive token selection mechanism to reduce redundant correlation interference between tokens and extract more representative enhanced features by using token-based channel self-attention, sparse space self-attention with the introduction of prior dictionaries and multi-head self-attention mechanism. Furthermore, by enhancing the representativeness of the captured features through token-based channel self-attention and sparse space self-attention, the reconstruction effect of clear details in the image is effectively improved. While eliminating redundant features to make the model lighter, it enhances the model's ability to extract important features and enables the network to adaptively identify features and assign different weights to features of different importance. It overcomes the problem of excessively redundant feature information extracted by the original classic super-resolution model. Compared with the original classic single-frame super-resolution method, it can greatly improve the reconstruction performance of the network model, further restore the texture details of the reconstructed image, and effectively reduce the number of model parameters.

[0037] (2) The channel self-attention module of the present invention selects the top-k channels in the channel dimension. The channel dimension is much smaller than the pixel space dimension, which saves a lot of computation. Furthermore, selecting the top-k channels is more semantically meaningful. Attached Figure Description

[0038] The present invention will now be described in further detail with reference to the accompanying drawings;

[0039] Figure 1 This is a flowchart of a lightweight image super-resolution method based on adaptive token selection according to an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the network framework of the lightweight image super-resolution method based on adaptive token selection according to an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the sparse space self-attention module of the lightweight image super-resolution method based on adaptive token selection in an embodiment of the present invention.

[0042] Figure 4 This is a schematic diagram of the channel self-attention module based on token selection in the lightweight image super-resolution method based on adaptive token selection according to an embodiment of the present invention.

[0043] Figure 5 This is a structural block diagram of a lightweight image super-resolution device based on adaptive token selection according to an embodiment of the present invention. Detailed Implementation

[0044] The present invention will be further described below through specific embodiments.

[0045] like Figure 1As shown, the lightweight image super-resolution method based on adaptive token selection has the following specific steps:

[0046] S1, constructing a lightweight image super-resolution network.

[0047] An adaptive token selection block is introduced into a feature-enhanced single-frame image super-resolution network, constructing a single-frame image super-resolution network (or lightweight image super-resolution network) based on adaptive token selection and feature enhancement. Figure 2 As shown, the network consists of two 3×3 convolutional layers, multiple Adaptive Token Selected Blocks (ATS), and an upsampling module based on subpixel convolution. The input low-resolution image is processed by a 3×3 convolutional layer to extract shallow features. These shallow features are then fed into a series of Adaptive Token Selected Blocks, where each ATS block contains multiple ATS Transformer layers. Sparse spatial self-attention, token selection channel self-attention, and conventional multi-head self-attention based on shift windows are combined to form the Transformer layers. It should be noted that the upsampling module can employ upsampling methods other than subpixel convolution; the specific configuration is determined as needed, and this embodiment does not impose any restrictions.

[0048] In the token selection block, three self-attention modules work in parallel: SparseSpatial Self-Attention (SSA) provides spatially de-redundant features, Token-SelectedChannel Self-Attention (SCA) provides channel-redundant features, and Window Self-Attention (WSA) provides features before redundancy removal to supplement lost details. These features are then combined with shallow features extracted from convolutional layers through summation, fully fusing spatial and channel features. Feature enhancement is achieved through the Feed Forward Network (FFN) module, followed by further deep feature extraction in the Transformer Layer. Adding these features to lower-level features preserves more local details, contributing to finer texture generation. Furthermore, the prior dictionary, a learnable parameter, participates in the SSA calculation process of each Transformer layer and is continuously updated to better adapt to the input features of the next layer. After the ATS block, additional convolutional and sub-pixel convolutional layers are used to upsample the extracted shallow and deep features to generate the final SR image.

[0049] The proposed SSA module first maps the introduced prior dictionary into keys (K) and values ​​(V), thus integrating the external prior into the self-attention analysis. This also helps to enhance the token representation by leveraging the similarity prior in the dictionary, making it unrestricted by the receptive field of window partitioning.

[0050] Specifically, the SSA module increases the sparsity of the self-attention map by setting lower low-level self-attention scores and higher high-level self-attention scores in the spatial dimension, thereby reducing noise from irrelevant tokens. By collaboratively using SCA and SSA to remove token redundancy in both the channel and spatial dimensions, and combining this with conventional shift-window-based multi-head self-attention in parallel, an adaptive token selection block is formed. This allows ATS to capture the most representative augmentation features while maintaining extremely low computational burden.

[0051] The Channel Self-Attention (SCA) module implements a token selection mechanism along the channel attention dimension based on the top-k idea. This module obtains a self-attention score matrix through self-attention, thereby determining the correlation strength between token pairs. It selects the top-k most correlated token pairs to avoid introducing noise into feature generation from weakly correlated token pairs. However, unlike the traditional top-k selection in pixel space, it selects the top-k along the channel dimension. The channel dimension is much smaller than the pixel space dimension, saving significant computational resources. Furthermore, selecting the top-k channels is more semantically meaningful. In pixel space, it is very difficult to determine the correlation strength between each pixel and why to exclude them.

[0052] like Figure 3 As shown, for each input First, map them to , , Then, after transposing, we get , , Used for self-attention computation along the channel dimension. Specifically, SCA first... and Perform matrix multiplication to obtain the self-attention score matrix. The matrix contains varying scores. High scores indicate strong correlation between token pairs, which is helpful for reconstructing image details. Low scores indicate weak correlation between token pairs, which may introduce noise. Therefore, a scaling factor K is set, and the top K largest correlation scores are selected to obtain the matrix. .

[0053] The selection method is... The top K largest scores remain unchanged, while the other scores are set to 0, so as not to affect the next step of the calculation. Next, and Matrix multiplication is used to obtain features that mitigate the interference of redundant correlations between tokens. Then, after transposing, we get In SCA, to preserve key information as much as possible, four different scaling factors K1, K2, K3, and K4 are set and processed in parallel as independent branches to obtain Y1S, Y2S, Y3S, and Y4S. Finally, they are added together to obtain the final module output. .

[0054] ;

[0055] ;

[0056] ;

[0057] ;

[0058] ;

[0059] in, express The i-th row and j-th column, express If the correlation score in the i-th row and j-th column is among the top K largest, then it remains unchanged. If it is not among the top K largest, then set it to 0.

[0060] like Figure 4 As shown, the Sparse Space Self-Attention Module (SSA) is as follows:

[0061] First, input features Mapped to query token Matrix, and introduce a prior dictionary Map it to a dictionary key token and dictionary value token :

[0062] ;

[0063] in, , , These are linear transformations of the query token, dictionary key token, and dictionary value token, respectively. 't' is a parameter used to reduce the computational cost and decrease the feature map dimension, and 'C' is the dimension of the token. It can be observed that... Dimensions and They are different; this is also to reduce the amount of computation, taking... , where N is the number of tokens.

[0064] Next, by and Matrix multiplication is performed to obtain the self-attention map. To filter out low similarity coefficients in the self-attention map, ReLU2 is applied. and The resulting self-attention map increases the original high similarity coefficients and decreases the low similarity coefficients, exacerbating the correlation differences and causing the network to focus more on strongly correlated parts. However, to avoid excessively sparse self-attention, a branch using conventional softmax processing is added in parallel to retain more complete information. By adding the self-attention maps from both branches, redundant information can be removed while key features are preserved. Finally, this de-redundant self-attention map is then compared with... Multiplying them together yields the dictionary-enhanced features.

[0065] ;

[0066] ;

[0067] in, represent and The self-attention map obtained through matching, Indicates to Squaring the result The weight coefficients representing the sparse processing branches are learnable parameters. These represent the weight coefficients of the traditional normalized branches, which are also learnable parameters. Through these two learnable weight parameters, the best balance between sparse and dense processing can be adaptively achieved. This indicates the enhanced and deredundant output characteristics of the SSA module.

[0068] The prior dictionary consists of learnable parameters that participate in the SSA calculation process of each Transformer layer and are continuously updated to better adapt to the input features of the next layer. Specifically, first, the prior dictionary D is initialized with a set of learnable parameters (D0, D1, ..., D...). M Then, it participates in the SSA calculation process of each Transformer layer during the training phase, and continuously updates the weights (i.e., parameters D) during training to better adapt them to the input features of the next layer. During the inference phase, the external prior of the learned prior dictionary is used to precisely guide the attention distribution to enhance the token representation.

[0069] S2 uses a lightweight image super-resolution network to reconstruct the low-resolution image, resulting in a reconstructed image.

[0070] S3 uses a trained lightweight image super-resolution network to perform super-resolution reconstruction on a low-resolution single-frame image, resulting in a reconstructed image.

[0071] like Figure 5 As shown, the present invention also discloses a lightweight image super-resolution device based on adaptive token selection, comprising:

[0072] The network construction module 501 is used to construct a lightweight image super-resolution network; the lightweight image super-resolution network includes a first convolutional layer, several adaptive token selection blocks, a second convolutional layer, and an upsampling module;

[0073] After the low-resolution image is processed by the first convolutional layer to extract shallow features, it is input into several sequentially connected adaptive token selection blocks for deep feature extraction. Then it is input into the second convolutional layer. The shallow features and the output of the second convolutional layer are added together and then input into the upsampling module for upsampling to obtain the reconstructed image.

[0074] The adaptive token selection block includes several ATS Transformer layers and a third convolutional layer. The input of the adaptive token selection block is processed through a series of interconnected ATS Transformer layers for deep feature extraction. The output of the last ATS Transformer layer is then added to the input of the adaptive token selection block after passing through the third convolutional block to obtain the output of the adaptive token selection block. The Transformer layer includes a sparse spatial self-attention module, a channel self-attention module, a shift-window-based multi-head self-attention module, and an FFN layer. The sparse spatial self-attention module performs spatial redundancy removal on the input shallow features to obtain spatially de-redundant features. The sparse spatial self-attention module uses a prior dictionary in the sparse spatial self-attention calculation process. Based on the learnable parameters of the prior dictionary, a prior dictionary for the sparse spatial self-attention module in the next ATSTransformer layer is constructed. The channel self-attention module performs channel redundancy removal on the input shallow features to obtain channel-de-redundant features. The shift-window-based multi-head self-attention module enhances the input shallow features to obtain multi-head attention-enhanced features. The spatially de-redundant features, the channel-de-redundant features, and the multi-head attention-enhanced features are summed, added to the input of the ATS Transformer layer, and then input into the FFN layer feedforward network for feature enhancement. The output is then used as the output of the ATS Transformer layer.

[0075] The network training module 502 is used to train the lightweight image super-resolution network to obtain the trained lightweight image super-resolution network.

[0076] Image reconstruction module 503 is used to reconstruct low-resolution single-frame images using a trained lightweight image super-resolution network to obtain reconstructed images.

[0077] The specific implementation of the lightweight image super-resolution device based on adaptive token selection is the same as the lightweight image super-resolution method based on adaptive token selection, and will not be described again in this embodiment.

[0078] The above are merely specific embodiments of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial modifications made to the present invention using this concept shall be considered as infringing upon the protection scope of the present invention.

Claims

1. A lightweight image super-resolution method based on adaptive token selection, characterized in that, Includes the following steps: S1, Construct a lightweight image super-resolution network; the lightweight image super-resolution network includes a first convolutional layer, several adaptive token selection blocks, a second convolutional layer, and an upsampling module; After the low-resolution image is processed by the first convolutional layer to extract shallow features, it is input into several sequentially connected adaptive token selection blocks for deep feature extraction. Then it is input into the second convolutional layer. The shallow features and the output of the second convolutional layer are added together and then input into the upsampling module for upsampling to obtain the reconstructed image. The adaptive token selection block includes several ATS Transformer layers and a third convolutional layer. The input of the adaptive token selection block is processed through a series of interconnected ATS Transformer layers for deep feature extraction. The output of the last ATS Transformer layer is then added to the input of the adaptive token selection block after passing through the third convolutional block to obtain the output of the adaptive token selection block. The ATS Transformer layer includes a sparse spatial self-attention module, a channel self-attention module, a multi-head self-attention module based on a shift window, and an FFN layer. The sparse spatial self-attention module performs spatial redundancy removal on the shallow features of the input to obtain the spatially deredundant features. The sparse space self-attention module uses a prior dictionary to participate in the sparse space self-attention calculation process; the prior dictionary of the sparse space self-attention module in the next ATSTransformer layer is constructed based on the learnable parameters of the prior dictionary; the channel self-attention module performs channel redundancy removal on the input shallow features to obtain the channel redundancy removal features. The multi-head self-attention module based on the shift window enhances the input shallow features to obtain multi-head attention enhanced features. The features after spatial redundancy removal, the features after channel redundancy removal, and the features enhanced by multi-head attention are summed, added to the input of the ATS Transformer layer, and then fed into the FFN layer feedforward network for feature enhancement. The output is then used as the output of the ATS Transformer layer. S2, train the lightweight image super-resolution network to obtain the trained lightweight image super-resolution network; S3 uses a trained lightweight image super-resolution network to perform super-resolution reconstruction on a low-resolution single-frame image, resulting in a reconstructed image.

2. The lightweight image super-resolution method based on adaptive token selection according to claim 1, characterized in that, The sparse space self-attention module is as follows: Input features are transformed through linear transformation Mapped to query token matrix Where t represents the parameter for reducing the dimension of the feature map to reduce computational cost; C represents the dimension of the token; and N represents the number of tokens. Introducing a priori dictionary A linear transformation maps the prior dictionary to dictionary keys (tokens). and dictionary value token Where M represents the number of tokens in the dictionary; Will and Perform matrix multiplication and redundancy removal to obtain the redundancy-removed self-attention map, represented as: ； in, express and The self-attention map obtained after redundancy removal through matching; Indicates to Squaring the result; The weight coefficients of the sparse processing branches are learnable weight parameters. This represents the weight coefficients of the traditional, normalized branches, and is a learnable weight parameter. express function; The self-attention map after redundancy removal With dictionary value token Multiplying them together yields the spatially redundant features.

3. The lightweight image super-resolution method based on adaptive token selection according to claim 1, characterized in that, The channel self-attention module includes several parallel branch networks; the input of all branch networks is the input of the channel self-attention module; each branch network is as follows: Branch networks map inputs to queries, keys, and values. , , And transpose them respectively to obtain , and ;in, Indicates the number of tokens; Indicates the number of channels; Will and Perform matrix multiplication to obtain the self-attention score matrix. ; Set a scaling factor K, and based on the descending order of the relevance scores of each element in the attention score matrix, keep the first K elements of the self-attention score matrix unchanged, while setting the other scores to zero, to obtain... ; Will and Matrix multiplication is used to obtain features ; indicates as: ； in, express function; Indicates the first A branch network; Will Transpose to obtain the output of the branch network ; The outputs of all branch networks with different scaling factor K values ​​are summed to obtain the channel redundancy-free features.

4. The lightweight image super-resolution method based on adaptive token selection according to claim 3, characterized in that, The number of parallel branch networks is 4.

5. The lightweight image super-resolution method based on adaptive token selection according to claim 1, characterized in that, The upsampling module is a subpixel convolutional layer.

6. A lightweight image super-resolution device based on adaptive token selection, comprising the following: A network construction module is used to construct a lightweight image super-resolution network; the lightweight image super-resolution network includes a first convolutional layer, several adaptive token selection blocks, a second convolutional layer, and an upsampling module; After the low-resolution image is processed by the first convolutional layer to extract shallow features, it is input into several sequentially connected adaptive token selection blocks for deep feature extraction. Then it is input into the second convolutional layer. The shallow features and the output of the second convolutional layer are added together and then input into the upsampling module for upsampling to obtain the reconstructed image. The adaptive token selection block includes several ATS Transformer layers and a third convolutional layer. The input of the adaptive token selection block is processed through a series of interconnected ATS Transformer layers for deep feature extraction. The output of the last ATS Transformer layer is then added to the input of the adaptive token selection block after passing through the third convolutional block to obtain the output of the adaptive token selection block. The ATS Transformer layer includes a sparse spatial self-attention module, a channel self-attention module, a multi-head self-attention module based on a shift window, and an FFN layer. The sparse spatial self-attention module performs spatial redundancy removal on the shallow features of the input to obtain the spatially deredundant features. The sparse space self-attention module uses a prior dictionary to participate in the sparse space self-attention calculation process; the prior dictionary of the sparse space self-attention module in the next ATSTransformer layer is constructed based on the learnable parameters of the prior dictionary; the channel self-attention module performs channel redundancy removal on the input shallow features to obtain the channel redundancy removal features. The multi-head self-attention module based on the shift window enhances the input shallow features to obtain multi-head attention enhanced features. The features after spatial redundancy removal, the features after channel redundancy removal, and the features enhanced by multi-head attention are summed, added to the input of the ATS Transformer layer, and then fed into the FFN layer feedforward network for feature enhancement. The output is then used as the output of the ATS Transformer layer. The network training module is used to train the lightweight image super-resolution network to obtain the trained lightweight image super-resolution network. The image reconstruction module is used to reconstruct low-resolution single-frame images using a trained lightweight image super-resolution network to obtain reconstructed images.

Images