A lightweight image super-resolution reconstruction method based on an attention mechanism

Abstract

The application discloses a light-weight image super-resolution reconstruction method based on an attention mechanism and relates to the technical field of computer vision and image processing. The method constructs a network model comprising a shallow feature extraction module, a deep feature extraction module and a reconstruction module, the deep feature extraction module is composed of multiple prior sparse self-attention blocks, each block counts the texture complexity in a local window to determine the target sparsity, and according to the similarity with the reference key, a corresponding number of tokens are reserved to participate in the calculation before attention calculation, and compensation information is generated based on the statistical characteristics of the discarded tokens and is superimposed on the attention output. The application realizes the adaptive allocation of the self-attention calculation amount with the texture complexity of the area without generating additional calculation overhead.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, specifically to a lightweight image super-resolution reconstruction method based on an attention mechanism, which is particularly suitable for real-time image super-resolution reconstruction on edge devices with limited computing resources. Background Technology

[0002] Super-resolution, aiming to recover a high-resolution, sharp image from a low-resolution image, is a fundamental task in computer vision. In recent years, self-attention mechanisms based on the Transformer architecture have achieved significantly better performance than traditional methods when applied to this task. However, their computational complexity increases quadratically with image size, severely limiting their deployment in resource-constrained scenarios.

[0003] To reduce computational cost, existing technologies mainly focus on two directions. One approach restricts self-attention to a fixed local window, but the window size is independent of the image content, leading to wasted computation in flat areas and insufficient detail in textured areas, resulting in a lack of adaptability. The other approach selects important tokens for attention computation by calculating attention scores or similarity scores, but the selection decisions of this type of method rely on additional computations such as clustering, ranking, or learnable routes. These decision-making overheads may partially or even completely offset the efficiency gains brought by sparsity.

[0004] Furthermore, while existing solutions utilize local variance to modulate attention weights, their role is to "adjust weights after attention calculation," which is a post-processing weighting. Therefore, how to adaptively allocate the computational cost of self-attention based on image texture complexity without incurring additional decision-making overhead remains an unsolved problem in current technologies. Summary of the Invention

[0005] This invention addresses the problems of existing self-attention computation lacking content adaptability and the efficiency improvement being limited by the reliance on additional decision computation in adaptive schemes. It provides a lightweight image super-resolution reconstruction method based on an attention mechanism, aiming to achieve adaptive allocation of self-attention computation based on the region of image texture complexity without incurring additional computational overhead.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A lightweight image super-resolution reconstruction method based on an attention mechanism includes the following steps:

[0008] Step S1: Construct the training dataset.

[0009] A high-resolution image is acquired, and the high-resolution image is downsampled to obtain a corresponding low-resolution image. The high-resolution image and the low-resolution image are then used to form a training sample pair.

[0010] Step S2: Construct a lightweight image super-resolution network model.

[0011] The network model includes a shallow feature extraction module, a deep feature extraction module, and a reconstruction module.

[0012] The shallow feature extraction module is used to perform preliminary feature extraction on the input low-resolution image. In a preferred embodiment, the shallow feature extraction module consists of a convolutional kernel with a size of [missing information]. It consists of convolutional layers with 3 input channels, 180 output channels, a stride of 1, and padding of 1.

[0013] The deep feature extraction module consists of multiple cascaded prior sparse self-attention blocks. Each prior sparse self-attention block includes a zero-overhead texture complexity measurement module, a prior sparse self-attention module, and a minimal information compensation module.

[0014] The reconstruction module is used to reconstruct a high-resolution image from deep features. In a preferred embodiment, the reconstruction module consists of a convolutional layer and a sub-pixel convolutional layer connected in sequence. The convolutional layer has 180 input channels and 180 output channels. ,in As an upsampling factor, the subpixel convolutional layer is used to amplify the spatial resolution. times.

[0015] The following provides a detailed description of the core technical solution of the present invention, namely the specific workflow of each of the prior sparse self-attention blocks.

[0016] (a) Zero-overhead texture complexity measurement module

[0017] This module is used to: divide the input feature map into several local windows in the spatial dimension, calculate the texture complexity index of each window, and determine the target sparsity of the window based on the index.

[0018] This invention uses local variance as an indicator of texture complexity. In image processing, local variance reflects the degree of variation in pixel values ​​within an image region—the larger the variance, the more high-frequency information such as edges, corners, or textures the region contains, requiring more computational resources to recover these details in super-resolution reconstruction; the smaller the variance, the flatter the region tends to be, with high information redundancy, requiring less computational resources to achieve acceptable reconstruction results. This physical characteristic of local variance is naturally suitable as a basis for guiding the allocation of computational resources.

[0019] Specifically, the input feature map Divide into several non-overlapping spatial dimensions. A local window. Among them, and These represent the height and width of the feature map, respectively. The number of feature channels, This refers to the window size.

[0020] like or Cannot be Divisible by, then The right and bottom boundaries are zero-filled, resulting in a height after filling. ,width .

[0021] After division, we get A local window. For the first A local window ( ), which contains a total of eigenvalues. .

[0022] No. Local variance of a local window Calculate using the following formula:

[0023]

[0024] in, For the first in this window 1 eigenvalue, This is the mean of all feature values ​​within the window. .

[0025] The calculation of local variance only involves reading, accumulating, multiplying, and dividing the existing values ​​in the input feature map. It does not require operations such as convolution, matrix multiplication, nonlinear activation, or sorting, and therefore does not generate additional computational overhead. This is the key to the "zero-overhead" content awareness of this invention—the decision information (local variance) is obtained directly from the inherent statistics of the input feature map, rather than being generated through additional computational modules.

[0026] Obtain the local variance of all local windows Then, the local variance of each window is normalized to eliminate the scale effect caused by differences in overall brightness or contrast between different input images. In a preferred embodiment, Z-score normalization is used:

[0027]

[0028] in, The mean of the local variances of all windows. Standard deviation To prevent the removal of the zero constant.

[0029] The normalized local variance is mapped to a preset interval. The mapping result is the target sparsity. In a preferred embodiment, the mapping is performed using the Sigmoid function:

[0030]

[0031] in, For the first The target sparsity of each window For the Sigmoid function, , The monotonicity of the Sigmoid function guarantees that the higher the normalization complexity, the closer the target sparsity is to the target value. The more tokens retained, the closer to the target value; conversely, the fewer tokens retained, the closer to the target value. This mapping method transforms continuously varying texture complexity into continuously varying sparsity, allowing the allocation of computing resources to change smoothly with texture complexity, thus avoiding the discontinuities caused by hard thresholding.

[0032] No. The number of tokens that need to be retained in each window is:

[0033]

[0034] in, This represents the total number of tokens within the window, where each token is a feature vector corresponding to a single spatial location on the feature map. This is the floor function.

[0035] The module outputs This directly determines the number of tokens involved in subsequent self-attention calculations, transforming texture complexity—an image content attribute—into a quantifiable computational control parameter, thus enabling region-adaptive pre-allocation of computational resources.

[0036] (ii) Prior sparse self-attention module

[0037] This module is used to: within each local window, based on the similarity between the keys of each token and the reference keys, retain the keys and values ​​of the several tokens with the highest similarity to participate in the self-attention calculation. The number of tokens retained is determined by the aforementioned target sparsity. It is confirmed that the keys and values ​​of the remaining tokens will not participate in the self-attention calculation.

[0038] In self-attention mechanisms, keys represent token features, queries match keys, and values ​​are aggregated information. The attention score is essentially the inner product similarity between the query and the key, and its computational complexity increases quadratically with the number of tokens. If important tokens can be pre-selected before self-attention computation, without relying on the attention score itself, attention computation for unimportant tokens can be fundamentally avoided.

[0039] Based on the above ideas, this invention proposes: Select a reference position within the window; tokens most similar to this reference position are more likely to be important information carriers within the window. This is because within the same local window, structurally similar tokens often belong to the same texture region, and the reference position, selected at the center of the window, can represent the overall feature distribution of the window. Therefore, by filtering tokens most similar to the reference key, a set of tokens worthy of participating in attention calculation can be pre-determined without calculating attention scores.

[0040] For the A partial window, which contains Each token corresponds to one 3D feature vector ( ).

[0041] Each token's feature vector is mapped to a query, key, and value through three independent linear transformations:

[0042]

[0043] in, It is a learnable linear transformation matrix. The dimension for each attention head. In a preferred embodiment, a multi-head attention mechanism is employed, where the number of heads... Dimensions of each head .

[0044] The reference key is the key of the token corresponding to a preset reference position within the local window. In a preferred embodiment, the preset reference position is the geometric center of the local window. The center position has the shortest average distance to all other positions within the window, and using the center as a reference can reflect the relative relationship between each token and the whole more evenly. If a corner or edge position is selected, the distance deviation will cause the filtering results to be biased to one side, which is not fair.

[0045] The similarity between each token's key and its reference key is based on Euclidean distance, and the similarity is negatively correlated with the Euclidean distance. The key of each token With reference key The Euclidean distance between them is:

[0046]

[0047] in, Indicator key The Each component.

[0048] Euclidean distance measures the proximity of two key vectors in the feature space. A smaller distance indicates greater similarity between the key and the reference token, making it more likely to receive higher attention weights in self-attention. Therefore, using Euclidean distance instead of attention score as a selection criterion is reasonable and efficient—the former only requires calculation... Secondary vector distance, while the latter requires calculation Secondary vector inner product.

[0049] All in the window Each token according to Sort by size from smallest to largest, and keep the smallest distance. Each token's key and value are used to discard the rest. Because this filtering occurs before the dot product calculation of self-attention, and the filtering criteria are completely independent of attention scores, the filtering itself does not introduce any additional computational dependencies—this is the most fundamental difference between this invention and existing technologies. In existing technologies, filtering based on attention scores must be completed first. and Only matrix multiplication can be performed, and this matrix multiplication is the core overhead of self-attention computation, meaning that the "filtering" itself has already consumed the computational resources that should have been saved.

[0050] Stack the remaining Keys and Values ​​into matrices. and The query stack of all tokens is Then, calculate the sparse self-attention using the following formula:

[0051]

[0052] in, This is a row-wise softmax normalization. Output .

[0053] Under the multi-head attention mechanism, The outputs of each size are concatenated along the channel dimension:

[0054]

[0055] in For the first Output of individual heads.

[0056] By outputting the projection matrix Perform linear transformation fusion:

[0057]

[0058] That is, the first Sparse self-attention output for each window.

[0059] This module reduces the size of the attention score matrix from Reduced to ,because The average is approximately The computational workload is reduced by about half. More importantly, the screening process is based on... Vector distance calculation replaced Matrix multiplication fundamentally eliminates the computational overhead of the sparsification decision itself.

[0060] (III) Minimalist Information Compensation Module

[0061] This module is used to generate compensation information based on the statistical features of the keys of the discarded tokens, and to overlay the compensation information onto the self-attention output.

[0062] Although discarded tokens do not participate in fine-grained attention calculations, they still carry global statistical information about the window. Completely discarding them would cause a shift in the window's global features—for example, discarded tokens from flat areas might carry overall brightness information for that area, potentially causing a deviation in the window's overall average brightness. Therefore, this invention compensates for this by retaining the statistical information of these discarded tokens at minimal cost.

[0063] For the Discarded within a window There are tokens, and the set of indices for their keys is . , Extract the mean of these discarded keys along the channel dimension as a compensation vector:

[0064]

[0065] like Then let .

[0066] The compensation vector is broadcast and superimposed onto each spatial location of the self-attention output:

[0067]

[0068] in, For broadcast operations, used to broadcast Extend to Same space dimensions.

[0069] This compensation module contains no learnable parameters and only involves summation, averaging, and broadcasting operations. Its design principle is based on the fact that the channel mean of the keys of discarded tokens reflects the overall distribution center of these tokens across various feature dimensions. Superimposing this as a compensation signal onto all output positions can compensate for the global statistical information lost due to token discarding, making the output features closer to the state before token discarding in terms of overall distribution. Since this compensation does not require training any parameters, the computational cost is negligible.

[0070] (iv) Window splicing and residual connection

[0071] All Each window The complete feature map is reconstructed by piecing together the original window spatial positions, resulting in... If filling was applied previously, the filled area will be cropped, restoring the original state. .

[0072] Finally, the input and the compensated output are added together via residual connections:

[0073]

[0074] in and These are the input and output feature maps of the prior sparse self-attention block, respectively.

[0075] Residual connections allow gradients to propagate directly, bypassing intra-block processing, ensuring the effectiveness of deep network training. Simultaneously, the input feature map is added element-wise to the sparsed feature map, ensuring that the block output retains both the original low-frequency contour information and the high-frequency detail information enhanced by sparse self-attention. This complementary approach guarantees the integrity of the feature representation.

[0076] Step S3: Train the network model.

[0077] Using the training sample pairs constructed in step S1, and with the high-resolution image as the supervision signal, the network model constructed in step S2 is trained to obtain the trained network model.

[0078] In a preferred embodiment, training is performed using the L1 loss function, parameter optimization is performed using the Adam optimizer, and the initial learning rate is [missing information]. The cosine annealing learning rate scheduling strategy is adopted, and the total number of training rounds is 1000.

[0079] Step S4: Image super-resolution reconstruction.

[0080] The low-resolution image to be reconstructed is input into the trained network model. The shallow feature extraction module performs preliminary feature extraction on the low-resolution image to be reconstructed and inputs the extracted shallow feature map into the deep feature extraction module. The multiple prior sparse self-attention blocks of the deep feature extraction module sequentially perform internal processing on the input feature map and output deep features. The reconstruction module upsamples and reconstructs the deep features to output a high-resolution image.

[0081] The technical effects and advantages of this invention are as follows:

[0082] First, this invention uses local variance as a texture complexity metric. This metric is directly read and calculated from the input feature map, involving only basic arithmetic operations such as accumulation, multiplication, and division, without involving additional calculations such as convolution, fully connected layers, sorting, or clustering, thus achieving truly zero-overhead content-adaptive sparsity. This effect stems directly from the design of the "zero-overhead texture complexity measurement module" in this invention. The decision basis of this module comes entirely from the inherent statistics of the input feature map, rather than any derived features that require additional calculations to obtain.

[0083] Second, the sparse decision-making in this invention occurs entirely before the dot product calculation of self-attention, and the decision-making basis does not depend on any attention scores that need to be pre-calculated. This is achieved through a "prior sparse self-attention module"—which uses the Euclidean distance with the reference key to perform the filtering. Vector distance calculation replaced The matrix multiplication is completely decoupled from attention computation, eliminating the additional computational dependency brought about by the posterior mode of "computing first and then filtering" in the prior art.

[0084] Third, this invention allocates different sparsity rates based on the texture complexity of different regions—more tokens are retained in texture-dense regions to ensure reconstruction quality, while more tokens are discarded in texture-flat regions to save computational resources, thus achieving region-adaptive allocation of computational resources. This is the result of the synergistic effect of the "zero-overhead texture complexity measurement module" and the "prior sparse self-attention module": the former provides a region-adaptive sparsity rate, and the latter performs differential filtering based on this sparsity rate; together, they achieve a fine allocation of computational resources among different regions of the image.

[0085] Fourth, the compensation module of this invention does not contain any learnable parameters, but only involves summation, averaging, and broadcasting operations, thus preserving the global statistical information of discarded tokens with extremely low computational cost. This "minimalist information compensation module" effectively compensates for the information loss caused by sparsity, enabling this invention to maintain reconstruction quality while reducing computational load. Attached Figure Description

[0086] Figure 1 This is a flowchart of the method of the present invention;

[0087] Figure 2 This is the single-head processing flow of the prior sparse self-attention module of the present invention;

[0088] Figure 3 The following is the processing flow of the simplified information compensation module of the present invention;

[0089] Figure 4 This describes the training process for the network model of the present invention. Detailed Implementation

[0090] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0091] Example 1

[0092] As attached Figures 1 to 4 The lightweight image super-resolution reconstruction method based on an attention mechanism is shown below, and its specific implementation is as follows:

[0093] I. Construction of the training dataset

[0094] This embodiment uses the DIV2K dataset as the training data source. The DIV2K dataset contains 800 high-resolution training images and 100 high-resolution validation images. The image format is PNG, and the resolution ranges from... Pixel to The pixel count varies and covers a variety of categories, including people, animals, buildings, natural scenes, and objects.

[0095] For each high-resolution image, bicubic interpolation is used for downsampling to generate a corresponding low-resolution image. The downsampling factor is denoted as . Its value is determined based on the target super-resolution task. In this embodiment, it is... Let's take an example to illustrate. For The task will High-resolution image downsampling Low-resolution images, here and These represent the height and width of the low-resolution image, which are the spatial dimensions of the feature maps within the network model.

[0096] Randomly crop the high-resolution image to The image patch of pixels is used as the training label, and the corresponding low-resolution image cropping size is [value missing]. Pixel. When At that time, low-resolution image patches are pixels. If the high-resolution image size is smaller than [number of pixels]. For pixels, the data is enlarged to this size using bicubic interpolation before being cropped to ensure that all training samples have a uniform input and output size.

[0097] Online data augmentation is performed on the cropped image patches, including: random horizontal flipping (probability 0.5), random... Rotation (probability 0.5), random color jitter (brightness adjustment range) Contrast adjustment range The data augmentation operations described above are randomly re-executed during each training round. This means the same sample may be input into the network with different augmentation methods in different training rounds, thereby increasing the diversity of the training samples. The batch size during training is denoted as... In this embodiment, it is set to 16, meaning that 16 samples are processed simultaneously each time the parameters are updated.

[0098] II. Construction of Network Model

[0099] The network model includes a shallow feature extraction module, a deep feature extraction module, and a reconstruction module. The input low-resolution image is denoted as... Its dimensions are ,in and These are the height and width of the image, respectively. It has three color channels: RGB.

[0100] (I) Shallow Feature Extraction Module

[0101] The shallow feature extraction module consists of a convolutional kernel with a size of It consists of convolutional layers. The number of input channels in this convolutional layer is... The number of output channels is 180, and the step size is... Fill with The step size is... And fill with This ensures that the convolutional layer does not change the spatial resolution of the feature map, i.e., the input is... The output is also This convolutional layer performs a convolutional transformation on the input image, mapping the RGB image from pixel space to a high-dimensional feature space, enabling subsequent deep feature extraction modules to perform richer feature representation. The shallow feature map output by this convolutional layer is denoted as... Its dimensions are Where 180 is the number of feature channels, denoted as ,Right now .

[0102] (II) Deep Feature Extraction Module

[0103] The deep feature extraction module consists of It consists of a series of cascaded prior sparse self-attention blocks. Each prior sparse self-attention block (hereinafter referred to as "block") includes three sub-modules, which are in the following order of processing: zero-overhead texture complexity measurement module, prior sparse self-attention module, and minimal information compensation module.

[0104] The term "concatenation" refers to the sequential flow of data through each block: the input to the first block is a shallow feature map. The output of the first block serves as the input to the second block, and so on, with the output of the last block being the final output of the deep feature extraction module. . The dimension is The number of blocks is This is the optimal value after comprehensively considering model capacity and computational efficiency.

[0105] Each block uses a residual connection structure. For the first... blocks ( ), its input is denoted as After the cascading processing of the above three sub-modules, the processing result is combined with... The output of this block is obtained by adding each element one by one. The expression for the residual join is: ,in Indicates the first Cascade transformation of three sub-modules within a block.

[0106] This residual structure allows the network to learn only the difference between the input and the output, reducing the learning difficulty and facilitating the propagation of gradients in deep networks, thus alleviating the gradient vanishing problem.

[0107] (III) Reconstruction Module

[0108] The reconstruction module consists of a convolutional layer and a sub-pixel convolutional layer connected in sequence. The kernel size of the convolutional layer is [size missing]. The number of input channels is 180, and the number of output channels is... .when The number of output channels is 48, and the step size is... Fill with The function of this convolutional layer is to transform deep features from 180 channels to... Each channel prepares for subpixel upsampling. Subpixel convolutional layers amplify spatial resolution. The specific operation is as follows: multiply the channel dimension. The values ​​are rearranged into spatial dimensions according to specific rules, resulting in an increase in both height and width. The number of channels is reduced to times. Output High-resolution images Subpixel convolutional layers do not introduce additional learnable parameters; they only rearrange the shape of tensors, resulting in a small number of parameters and high computational efficiency.

[0109] III. Internal Structure of Prior Sparse Self-Attention Blocks

[0110] The following describes in detail the internal processing flow of each prior sparse self-attention block. The input to each block is denoted as... , dimension The output is denoted as The dimensions are the same. Intra-block processing is executed sequentially according to the order of the three sub-modules, i.e., input... First, the output of the zero-overhead texture complexity measurement module is used by the prior sparse self-attention module. The output of this module then enters the minimal information compensation module, and finally, the result is obtained through residual connections. .

[0111] (a) Zero-overhead texture complexity measurement module

[0112] The core function of this module is to quantitatively evaluate the texture complexity of each local region of the input feature map without incurring additional computational overhead, and to convert the complexity into the target sparsity for subsequent self-attention computation.

[0113] Step 1: Divide the window

[0114] This module first takes the input feature map Divide into several non-overlapping spatial dimensions. A partial window; in this embodiment, the window size is... Window size It is the optimal value verified by experiments—the window is too small (e.g.) This can lead to each window containing too few tokens, making the statistical variance sensitive to noise and resulting in insufficient reliability; an excessively large window (e.g.) If the grain size is too coarse, it will be impossible to distinguish between flat areas and complex texture areas.

[0115] like or Cannot be If divisible, zeros are padded to the right and bottom boundaries of the feature map so that the padded height and width can be divided. Divisible. Specifically, right-hand filler. Column, bottom supplement Okay, among them , The values ​​at the zero-padding positions are all 0. Zero padding is used to ensure that all windows have a uniform size. This facilitates subsequent batch parallel processing, and these zero-padding regions will be pruned in the final output.

[0116] After division, we get A local window, Each window contains There are 1 spatial location, each spatial location corresponds to 1 token, and each token is a... The feature vector is dimensional, therefore each window contains There are 1 characteristic values, and the total number of tokens in the window is denoted as . , .when hour, Each window contains Each feature value.

[0117] Step 2: Calculate local variance

[0118] For the windows ( ), calculate the variance of all eigenvalues ​​within the window. This serves as an indicator of the texture complexity of the window. The specific calculation process is as follows: first, calculate the average of all feature values ​​within the window. Then calculate the average of the squared deviations of each eigenvalue from the mean, that is:

[0119]

[0120] in This represents the total number of feature values ​​within the window. ; For the first in this window eigenvalues ​​( ); This is the average of all feature values ​​within the window.

[0121] Local variance It has a clear physical meaning: the larger the variance value, the greater the difference in values ​​of each spatial location in different channels within the window, that is, the richer the high-frequency information such as edges and textures contained in the region; the smaller the variance value, the more consistent the values ​​within the window, that is, the region is a flat region with high information redundancy, and less computing resources can be allocated.

[0122] Step 3: Normalization

[0123] Get all Variance of each window Then, these variances are Z-score normalized. The purpose of normalization is to eliminate the scale effect caused by differences in overall brightness or contrast between different input images. For example, an image that is generally bright and an image that is generally dark may have systematic differences in the absolute values ​​of their local variances, but their relative texture complexity distribution characteristics should be similar. Normalization can eliminate this absolute scale difference.

[0124] Normalization specifically involves: first calculating the mean of the variances of all windows. and standard deviation ,in , Then, subtract the mean from the variance of each window and divide by the standard deviation to obtain the normalized value. :

[0125]

[0126] in To prevent division by zero constant, When all window variances are exactly equal (i.e. When the constant is normalized, it guarantees that the denominator is not zero. After normalization, all... The mean is variance is .

[0127] Step 4: Map to target sparsity

[0128] Normalized values Mapping to intervals via the Sigmoid function Within this window, the mapping result is the target sparsity. :

[0129]

[0130] in For the Sigmoid function, In this embodiment , ,therefore The monotonically increasing property of the Sigmoid function ensures that: the higher the normalization value (i.e., the more complex the texture of the window relative to the whole image), the closer the target sparsity is to 0.9 (i.e., more tokens are retained); the lower the normalization value (i.e., the flatter the texture of the window relative to the whole image), the closer the target sparsity is to 0.2 (i.e., more tokens are discarded).

[0131] and The value is determined based on experience: This ensures that even the flattest areas retain at least [a certain percentage of the original text]. The token is used to prevent complete information loss, which could lead to reconstruction failure. It ensures that even the most complex areas are discarded. The token is used to maintain overall computational efficiency.

[0132] Step 5: Determine the number of tokens to retain.

[0133] Finally, the target sparsity Multiply by the total number of tokens in the window ( Then, round up to get the number of tokens that the window needs to retain. :

[0134]

[0135] when hour, , The range of values ​​is to That is, 13 to 58. Rounding up is to ensure... The value is a positive integer to facilitate subsequent indexing operations. At this point, the module has completed a quantitative assessment of the computational cost for each window—windows with rich textures retain more tokens, while flat windows retain fewer tokens.

[0136] Explanation of zero overhead: Of the five steps above, steps 2 to 4 only involve basic arithmetic operations such as reading, accumulation, multiplication, and division. Step 1 only involves the segmentation of the spatial location of the feature map, and does not involve operations such as convolution, matrix multiplication, or sorting. The computational essence of this module is the extraction of statistical quantities of existing values ​​in the input feature map, rather than superimposing new learnable transformations or complex operations on the feature map. Therefore, it does not generate additional computational overhead in the traditional sense.

[0137] (ii) Prior sparse self-attention module

[0138] The input to this module is the first... Within a window tokens ( Each token is a 3D feature vector, denoted as ( ), corresponding to that spatial location on the feature map Each channel value. The goal of this module is to retain only the channel values. The most important token participates in the self-attention calculation, the rest... One token was discarded.

[0139] Step 1: Generate queries, keys, and values

[0140] This module employs a multi-head attention mechanism, with the number of heads denoted as . In this embodiment The dimension of each head is denoted as... , Multi-head attention mechanisms enable the model to capture different feature correlations from multiple subspaces, improving the model's expressive power. (Number of heads) It can be divided by 180 feature channels, ensuring that each head has an even distribution of channels and that the calculation is regular.

[0141] For the Size ( The feature vector of each token is mapped to the query value using three independent linear transformation matrices. ,key Sum :

[0142]

[0143] in For the first A set of learnable linear transformation matrices, which are the network's learnable parameters, updated during training via backpropagation. After the linear transformation, That is, each head will The input features of dimension are compressed to Attention computation is performed in a dimensional subspace.

[0144] Step 2: Determine the reference key

[0145] A reference position is preset within the window; in this embodiment, the geometric center of the window is chosen as the reference point. For a window ( ), spatial location number is ,in , The geometric center is located at the first... Line 1 Column (0-based index). When hour, That is, the geometric center is located at Use the key of the token corresponding to that position as the reference key, denoted as... ,in Is the token inside the window? Linear index in a token .

[0146] The reason for choosing the geometric center as the reference point is that the center has the shortest average distance to all other positions within the window. Using the center as a reference point can reflect the relative relationship between each token within the window and the whole more evenly. If a corner position were chosen as the reference point, tokens farther from the reference point might be systematically underestimated in importance due to the large distance, which is unfair.

[0147] Step 3: Calculate similarity and filter

[0148] For the The size, calculates the key of each token within the window. With reference key Euclidean distance between :

[0149]

[0150] in express The Each component ( The smaller the Euclidean distance, the more similar the key of the token is to the reference key, and the more likely the token is to carry important information within the window. This is because tokens similar to the central reference point often belong to the same texture region or have similar structural features.

[0151] Will Each token according to Sort by size from smallest to largest, and keep the smallest distance. Each token's key and value are used to discard the rest. Each token has a key and value. Discarded keys and values ​​do not participate in subsequent self-attention calculations. This filtering process is performed independently for each head, meaning different heads may retain different token sets due to the different linear transformation matrices of each head. Since each head focuses on a different feature subspace, the "important tokens" selected by each head may naturally differ. This independent selection method maintains the diversity advantage of multi-head attention.

[0152] Step 4: Calculate sparse self-attention

[0153] The first The keys and values ​​retained from each element are stacked into a key matrix. Sum matrix All The query stack of each token forms a query matrix. Here, "stacking" refers to arranging multiple vectors in rows to form a two-dimensional matrix.

[0154] Then, sparse self-attention is calculated according to the standard self-attention calculation formula:

[0155]

[0156] in For the first Output of size This is a softmax normalization operation performed row by row.

[0157] The dimensional changes of each part in the formula are as follows: The dimension is , The dimension is Multiplying the two together gives The attention score matrix; divided by Scaling is applied to prevent the dot product value from becoming too large, which could cause the softmax gradient to vanish; the gradient remains the same after softmax. multiplied by (dimension) After that, I got The output of .

[0158] It is worth noting that the dimension of the attention score matrix in the above formula is [missing information]. The dimension of this matrix in full self-attention is... .because The average is approximately The computational cost (number of dot products) is approximately one-third that of full self-attention. The flatter the texture, the better. The smaller the value, the more computational effort is saved.

[0159] Step 5: Multi-head fusion

[0160] Will The outputs of each size are concatenated along the channel dimension:

[0161]

[0162] Then through an output projection matrix Perform linear transformation fusion:

[0163]

[0164] in express The transpose of . This linear transformation will The features after splicing are integrated and fused, and the output dimension is restored to [value]. The channel, consistent with the input dimension, facilitates subsequent residual connections. That is, the first Sparse self-attention output for each window.

[0165] (III) Minimalist Information Compensation Module

[0166] Although discarded tokens do not participate in self-attention calculations, they still carry global statistical information about the window (such as the overall mean and variance of each channel). Completely discarding them would cause a shift in the global features of the window, affecting reconstruction quality. This module retains the statistical information of these discarded tokens at extremely low computational cost to compensate for the loss of information.

[0167] For the The set of indices of the keys of the discarded tokens is denoted as . Its size is Extract the first The average value of the discarded keys in the head along the channel dimension is used as the compensation vector for that head. :

[0168]

[0169] If no token is discarded (i.e.) If ), then the compensation vector is the zero vector.

[0170] Compensation vectors for all heads By concatenating the vectors along the channel dimension, a complete compensation vector is obtained. :

[0171]

[0172] Then broadcast the compensation vector to all At each spatial location, and with sparse self-attention output Element-by-element addition:

[0173]

[0174] The meaning of broadcast operation is: to broadcast The compensation vector of dimension is copied in the spatial dimension. One copy for each spatial location, forming The tensor, and then with Element-wise addition. This compensation module contains no learnable parameters and only involves summation, averaging, concatenation, and broadcasting operations; the computational overhead is negligible.

[0175] (iv) Window splicing and residual connection

[0176] All Each window The complete feature map is reconstructed by piecing together the original window's spatial position, resulting in... Concatenation is the inverse operation of partitioning: it combines each... The window block is then placed back into its corresponding position in the original feature map. If zero-padding was previously applied at the boundaries, this step clips the zero-padding area, restoring it to its original size. .

[0177] Finally, the input of the block is connected via residual connections. With the processed output Adding each element one by one, we get:

[0178]

[0179] Residual joins require that both dimensions be exactly the same, which are both present here. This concludes the processing flow for a priori sparse self-attention block.

[0180] IV. Training the Network Model

[0181] (a) Loss Function

[0182] Using the L1 loss function as the supervision signal, the average absolute error of all pixels between the reconstructed image and the high-resolution labeled image is calculated:

[0183]

[0184] in For batch size, For sample index within a batch, For pixel position and channel index, and The first in each batch The reconstructed image and high-resolution labeled image of each sample. The L1 loss penalizes larger errors linearly, thus it is robust to outliers and can generate images with sharper edges, making it suitable for super-resolution tasks.

[0185] (ii) Optimizer

[0186] The Adam optimizer is used for parameter optimization. Adam is an adaptive learning rate optimization algorithm that combines the advantages of momentum and RMSProp, and can adaptively adjust the learning rate for each parameter. The initial learning rate is set to... The remaining hyperparameters use the default settings of the Adam optimizer: first-order moment decay rate. Second-order moment decay rate Numerical stability constant We do not use weight decay to avoid unnecessary constraints on model capacity.

[0187] (III) Learning Rate Scheduling

[0188] A cosine annealing learning rate scheduling strategy is adopted. The learning rate is set at the beginning of training. Then it smoothly descends according to the cosine curve to :

[0189]

[0190] in This represents the current training epoch. In this embodiment, the total number of training rounds is [number]. Cosine annealing, compared to step descent, allows for a smoother adjustment of the learning rate, which helps the model converge more precisely in the later stages of training.

[0191] (iv) Training process

[0192] The training will be conducted in the following steps:

[0193] Step 1: Initialize all network parameters. The weights of the convolutional and linear transform layers are initialized using Kaiming uniform initialization, i.e., from a uniform distribution. Mid-sampling, of which The current layer has the number of input channels, and the bias is initialized to... .

[0194] Step 2: Randomly sample from the training dataset A pair of low-resolution and high-resolution images ( ( ), forming a batch.

[0195] Step 3: Input the low-resolution images from the batch into the network, and sequentially perform shallow feature extraction and deep feature extraction. Process each block sequentially and reconstruct it, outputting a reconstructed image. .

[0196] Step 4: Calculate the reconstructed image With high-resolution labels L1 loss value between .

[0197] Step 5: Calculate the gradient of the loss with respect to each learnable parameter through backpropagation.

[0198] Step 6: Update the network parameters using the Adam optimizer based on the calculated gradients.

[0199] Step 7: Repeat steps 2 through 6. After each epoch (i.e., after one traversal of all training samples), update the learning rate according to the cosine annealing schedule.

[0200] Step 8: Training until the desired level is reached. The training process stops after a certain number of epochs. During training, an evaluation is performed on the validation set every 50 epochs, the PSNR metric on the validation set is calculated, and the model parameter with the highest PSNR is saved as the final training result.

[0201] As an alternative implementation, an early stopping strategy can also be adopted: if the PSNR on the validation set no longer increases for 50 consecutive epochs, training is terminated early to prevent overfitting.

[0202] This embodiment is in Training is performed on NVIDIA A100 GPUs using a data-parallel strategy, with each GPU processing [data / processing data]. For each sample, the gradient is synchronized across all GPUs.

[0203] V. Image Super-Resolution Reconstruction

[0204] The low-resolution image to be reconstructed (denoted as...) Input the trained network model and perform one forward propagation. The forward propagation process is exactly the same as step three in the training phase: the image sequentially passes through the shallow feature extraction module, the deep feature extraction module, and so on. The process involves sequentially processing each block and a reconstruction module, ultimately outputting a high-resolution image (denoted as ). During inference, no loss needs to be calculated, nor is backpropagation required; all network parameters remain constant.

[0205] Example 2

[0206] The difference between this embodiment and Embodiment 1 lies in the upsampling factor. The values ​​can be different.

[0207] when When, execute Super-resolution task. During the training phase, the cropping size of the high-resolution images remains [missing information]. pixels, corresponding to a low-resolution image cropping size of . Pixels. The number of output channels of the convolutional layer in the reconstruction module is... Subpixel convolutional layers amplify spatial resolution. The remaining network structure and parameters are the same as in Example 1.

[0208] when When, execute Super-resolution task. The corresponding low-resolution image cropping size is... Pixels. The number of output channels of the convolutional layer in the reconstruction module is... Subpixel convolutional layers amplify spatial resolution. The remaining network structure and parameters are the same as in Example 1.

[0209] The network structure of this invention inherently supports different The value can be adjusted by modifying the downsampling rate of the training data, the number of output channels of the convolutional layers in the reconstruction module, and the magnification of the sub-pixel convolutional layers accordingly. The training process under this value is the same as in Example 1, only requiring adjustment based on the target value. Adjust the corresponding parameters.

[0210] Example 3

[0211] The difference between this embodiment and Embodiment 1 lies in the setting of the loss function.

[0212] In addition to L1 loss, perceptual loss is added. Perceptual loss improves the subjective visual quality of the reconstructed image by measuring the difference between the reconstructed image and the high-resolution label in the high-level semantic feature space, making the texture more natural and the details richer.

[0213] The perceptual loss is calculated as follows: the reconstructed image and the high-resolution labeled image are respectively input into a VGG-16 network pre-trained on ImageNet, and the perceptual loss is extracted from the network. The feature maps output by the ReLU activation layer before each pooling layer are then used to calculate the mean squared error between the corresponding feature maps of the two images.

[0214]

[0215] in The feature mapping function output by the VGG-16 network layer mentioned above. , , These represent the number of channels, height, and width of the feature map, respectively. This is the batch size. The VGG-16 network is pre-trained on ImageNet, and its parameters remain fixed during the perceptual loss calculation and are not involved in training updates.

[0216] The total loss function is a weighted sum of L1 loss and perceptual loss:

[0217]

[0218] In this embodiment, the weighting coefficient The value is set to 0.01. This value is determined based on experience and can moderately improve visual quality while maintaining pixel-level accuracy.

[0219] Example 4

[0220] The difference between this embodiment and Embodiment 1 lies in the local window size. The value of .

[0221] In Example 1 In one variant, That is, each local window contains Each token. At this point, the window division is more refined, the adaptive granularity is finer, and it can more accurately distinguish texture differences between different regions. However, the number of windows increases to the original number. The overhead of splicing and padding between windows increases slightly. Correspondingly, The range of values ​​becomes .

[0222] In another variant, That is, each local window contains Each token. At this point, the number of windows is reduced to the original number. While overall computational efficiency has improved, the adaptive granularity has become coarser, potentially making it difficult to accurately distinguish texture changes within a window. Accordingly, The range of values ​​becomes .

[0223] Preferably, A good balance can be achieved between adaptive finesse and computational efficiency, a conclusion confirmed by performing a grid search on the validation set.

[0224] Example 5

[0225] The difference between this embodiment and Embodiment 1 lies in the method of similarity measurement.

[0226] In Example 1, Euclidean distance is used to measure the similarity between keys; the smaller the distance, the more similar they are. In another variant, cosine similarity is used. Cosine similarity calculates the cosine of the angle between two vectors:

[0227]

[0228] Cosine similarity focuses on the direction of vectors rather than their magnitude, and may have better robustness in certain scenarios (such as when the magnitudes of feature vectors differ significantly). When using cosine similarity, the cosine similarity between the key of each token and the reference key is calculated, and the token with the highest cosine similarity is retained. The key and value of each token.

[0229] Preferably, Euclidean distance is used as the similarity metric, which is simple to calculate and has stable results. This is also the choice in Embodiment 1 of the present invention.

[0230] Example 6

[0231] The difference between this embodiment and Embodiment 1 lies in the mapping method of the sparsity rate.

[0232] In Example 1, the Sigmoid function is used to map the normalized local variance to the target sparsity, i.e. The Sigmoid mapping has the characteristics of smoothness and continuity. Its S-shaped curve makes the mapping changes in the middle region more gradual, which is beneficial to the stability of training.

[0233] In another variant, a piecewise linear mapping is used:

[0234]

[0235]

[0236]

[0237] Piecewise linear mapping is simpler to compute, requiring no exponential operations, and has a slight advantage in inference speed. However, the smoothness of the Sigmoid mapping makes the change in sparsity more continuous.

[0238] Preferably, a Sigmoid mapping is used.

[0239] Example 7

[0240] The difference between this embodiment and Embodiment 1 lies in the number of feature channels. The value of .

[0241] In Example 1, the number of feature channels In another variant, At this point, the model has fewer parameters and faster inference speed, but its feature representation capability is somewhat reduced. Correspondingly, the number of heads in multi-head attention... Can be adjusted to Dimensions of each head .

[0242] In yet another variant, At this point, the model has a stronger representational capability, but the number of parameters and computational cost increase. Correspondingly, Can be adjusted to Dimensions of each head .

[0243] Preferably, A good balance can be achieved between model performance and computational efficiency. The value is based on the classic configuration of existing methods such as SwinIR, and its effectiveness has been verified by experiments within the framework of this invention.

[0244] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A lightweight image super-resolution reconstruction method based on an attention mechanism, characterized in that, Includes the following steps: Step S1: Obtain a high-resolution image, downsample the high-resolution image to obtain a corresponding low-resolution image, and combine the high-resolution image and the low-resolution image to form a training sample pair; Step S2: Construct a lightweight image super-resolution network model, which includes a shallow feature extraction module, a deep feature extraction module, and a reconstruction module; The deep feature extraction module consists of multiple cascaded prior sparse self-attention blocks, each of which includes a zero-overhead texture complexity measurement module, a prior sparse self-attention module, and a minimal information compensation module. The zero-overhead texture complexity measurement module is used to: divide the input feature map into several local windows in the spatial dimension, calculate the texture complexity index of each local window, and determine the target sparsity of the window based on the texture complexity index. The texture complexity index is directly calculated based on the statistics of the input feature map. The prior sparse self-attention module is used to: within each local window, based on the similarity between the key of each token and the reference key, retain the keys and values ​​of the tokens with the highest similarity to participate in the self-attention calculation. The number of tokens retained is determined by the target sparsity rate, and the keys and values ​​of the remaining tokens do not participate in the self-attention calculation. The reference key is the key of the token corresponding to the preset reference position within the local window, and each token is a feature vector corresponding to a single spatial position on the feature map. The simplified information compensation module is used to: generate compensation information based on the statistical characteristics of the keys of the discarded tokens, and superimpose the compensation information onto the output of the self-attention calculation; Step S3: Using the training sample pairs constructed in step S1, and with the high-resolution image as the supervision signal, train the network model constructed in step S2 to obtain the trained network model. Step S4: Input the low-resolution image to be reconstructed into the trained network model. The shallow feature extraction module performs preliminary feature extraction on the low-resolution image to be reconstructed and inputs the extracted shallow feature map into the deep feature extraction module. The multiple prior sparse self-attention blocks of the deep feature extraction module sequentially perform internal processing on the input feature map and output deep features. The reconstruction module upsamples and reconstructs the deep features to output a high-resolution image.

2. The method according to claim 1, characterized in that, In the zero-overhead texture complexity measurement module, the texture complexity index is local variance. The specific method for calculating the texture complexity index for each local window is as follows: calculate the variance of all feature values ​​within each local window, and use the calculated variance as the texture complexity index for that window; the variance... Calculate using the following formula: in, For the first Local variance of a local window This represents the total number of feature values ​​within the window. For the first in this window 1 eigenvalue, This is the mean of all feature values ​​within the window.

3. The method according to claim 1, characterized in that, In the zero-overhead texture complexity measurement module, the specific method for determining the target sparsity of the window based on the texture complexity index is as follows: The local variances of all local windows are normalized, and the normalized local variances are mapped to a preset interval. The mapping results are used as the target sparsity of each local window. ; The number of tokens to be retained The total number of tokens within this local window. With the target sparsity Round up the product: .

4. The method according to claim 3, characterized in that, The normalization process is Z-score normalization, the mapping is through the Sigmoid function, and the preset interval is... ,in , ; The Z-score normalization is performed according to the following formula: The mapping via the Sigmoid function is performed according to the following formula: in, For the first Local variance after local window normalization This is the mean of the local variances of all local windows. Let the standard deviation of the local variance of all local windows be denoted as . To prevent division by zero constant, This is the Sigmoid function.

5. The method according to claim 1, characterized in that, In the prior sparse self-attention module, the similarity between the key and the reference key based on each token specifically refers to: the similarity between the key and the reference key based on each token. With the reference key Euclidean distance between The similarity is measured, and the similarity is related to the Euclidean distance. Negative correlation, the Euclidean distance Calculate using the following formula: in, For each dimension of attention head, Indicator key The One portion, Indicates reference key The Each component.

6. The method according to claim 1, characterized in that, In the prior sparse self-attention module, the specific method for participating in self-attention calculation is as follows: the queries of all tokens within each local window are stacked into a query matrix. The remaining keys are stacked into a key matrix. The retained values ​​are stacked into a value matrix. Then calculate the self-attention output according to the following formula: in, For the first The query stacking matrix of all tokens within a local window. The remaining key stacking matrix, Stack matrices to preserve the values. Dimensions for each attention head.

7. The method according to claim 1, characterized in that, In the simplified information compensation module, the specific method for generating compensation information based on the statistical features of the keys of the discarded tokens is as follows: extract the mean of the keys of each discarded token along the channel dimension as the compensation vector. Then, the compensation vector is broadcast and superimposed onto each spatial location of the self-attention output; The compensation vector Extract using the following formula: The broadcast overlay is performed according to the following formula: in, For the set of indices of the discarded keys, The number of keys that were discarded. For the first A discarded key, For the first Sparse self-attention output of a local window The broadcast operation is used to copy the compensation vector to all spatial locations of the sparse self-attention output.

8. The method according to claim 1, characterized in that, The shallow feature extraction module consists of a convolutional kernel with a size of [missing information]. The system consists of convolutional layers with the following input channels: The number of output channels is 180; the reconstruction module consists of a convolutional layer and a sub-pixel convolutional layer connected in sequence. The number of input channels of the convolutional layer is 180, and the number of output channels is... ,in As an upsampling factor, the subpixel convolutional layer is used to amplify the spatial resolution. times; The prior sparse self-attention block employs a multi-head attention mechanism, with the number of attention heads being [number missing]. Each attention head has a dimension of 30; The size of the local window is Pixel; The number of prior sparse self-attention blocks in the deep feature extraction module is: .

9. The method according to claim 1, characterized in that, Each of the prior sparse self-attention blocks further includes a residual connection, which is used to add the input feature map of the prior sparse self-attention block to the compensated feature map output by the minimal information compensation module element by element, and the sum is used as the output feature map of the prior sparse self-attention block: in, The input feature map is the prior sparse self-attention block. This is the compensated feature map output by the simplified information compensation module. The output feature map of the prior sparse self-attention block.

10. A lightweight image super-resolution reconstruction system based on an attention mechanism, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1 to 9.

Images