An image super-resolution method based on multi-scale similarity back-projection

Abstract

This invention provides an image super-resolution method based on multi-scale similarity backprojection, belonging to the field of digital image processing technology. The method includes: constructing a multi-scale cross-attention backprojection network based on the multi-scale similarity of the image; inputting the image to be improved into a trained image super-resolution reconstruction model for inference to obtain an enhanced resolution image; the multi-scale cross-attention backprojection network includes a cascaded shallow feature extraction module, a nonlinear backbone network, and a reconstruction module, wherein the nonlinear backbone network contains multiple residual backprojection groups. The multi-scale cross-attention backprojection block includes a local feature aggregation module, a multi-scale cross-attention backprojection unit, and a convolutional gated feedforward network. This invention models the multi-scale similarity of images through a cross-scale cross-attention mechanism, achieving high-quality super-resolution reconstruction while maintaining lightweight design.

Description

Technical Field

[0001] This invention belongs to the field of digital image processing and computer vision technology, and specifically relates to an image super-resolution method based on multi-scale similarity back projection. Background Technology

[0002] Image super-resolution aims to reconstruct a corresponding high-resolution image from a degraded, downsampled low-resolution image. With the development of deep learning technology, existing methods mainly learn the complex mapping relationship from low-resolution to high-resolution images by constructing deep neural networks and training them on paired datasets containing low-resolution images and their corresponding high-resolution images.

[0003] Existing deep learning-based image super-resolution methods mainly include convolutional neural network (CNN)-based methods, Transformer-based methods, and linear attention-based methods. CNN-based methods extract image features using convolutional layers, learn the dependencies between features through a nonlinear mapping module, and finally complete super-resolution reconstruction using a reconstruction module. However, CNNs are limited by a fixed and finite receptive field and input-independent kernel parameters, making it difficult to fully exploit global contextual information in low-resolution images. While increasing network depth, designing complex residual connections, and introducing attention mechanisms can improve reconstruction quality to some extent, these improvements significantly increase the computational cost and number of parameters, limiting their practical application in resource-constrained scenarios and scenarios with high real-time requirements.

[0004] Transformer-based methods utilize self-attention mechanisms to model nonlocal dependencies, achieving superior performance compared to convolutional neural networks in detail texture recovery. However, the computational complexity of self-attention mechanisms increases quadratically with the input image resolution, severely limiting their application in real-world scenarios. Linear attention-based methods reduce computational overhead by introducing window self-attention, transpose self-attention, or sparse self-attention mechanisms, but these simplifications compromise the self-attention mechanism's ability to model nonlocal dependencies and easily introduce blocky artifacts into the reconstructed image, affecting visual quality.

[0005] In summary, existing image super-resolution methods primarily focus on the model's ability to model nonlocal dependencies. Research on how to leverage inherent multi-scale similarity priors in images to enhance high-frequency information recovery is insufficient. Furthermore, existing methods generally suffer from high computational complexity and a large number of parameters, making it difficult to achieve lightweight design while maintaining high reconstruction quality. Therefore, designing a computationally efficient, lightweight image super-resolution network that effectively utilizes multi-scale similarity priors is an urgent technical problem to be solved in this field. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, in a first aspect, this application proposes an image super-resolution method based on multi-scale similarity back projection, the method comprising: A multi-scale cross-attention backprojection network is constructed based on the multi-scale similarity of images; The image whose resolution needs to be improved is input into the trained image super-resolution reconstruction model for inference, and the enhanced resolution image is obtained. The image super-resolution reconstruction model is trained on the multi-scale cross-attention backprojection network in a supervised manner using a paired image dataset with degraded downsampling as the training dataset.

[0007] Preferably, the multi-scale cross-attention back projection network includes a shallow feature extraction module, a nonlinear backbone network, and a reconstruction module cascaded in sequence. The shallow feature extraction module uses a blueprint-separable convolutional layer consisting of pointwise convolutional layers and depthwise convolutional layers to perform convolution operations on the input image to extract shallow features of the image. The nonlinear backbone network is used to refine and correct the shallow features of the image and restore the detailed features, outputting the depth features; The reconstruction module is used to fuse the shallow features and deep features of the image to obtain super-resolution reconstructed image features. Then, the super-resolution reconstructed image features are mapped back to image pixel information. Finally, the super-resolution image information is periodically permuted from the channel dimension to the spatial dimension to increase the image resolution and output the super-resolution reconstructed image. The shallow feature extraction process of the image described herein satisfies the following formula:

[0008] In the above formula, For shallow image features, The image is used as input to the shallow feature extraction module. This indicates a pointwise convolutional layer. This indicates a depthwise convolutional layer.

[0009] Preferably, the nonlinear backbone network includes multiple residual backprojection groups and a blueprint separable convolutional layer. Each residual backprojection group combines multiple multi-scale cross-attention backprojection blocks and a blueprint separable convolutional layer through residual connections. The multi-scale cross-attention backprojection block includes a local feature aggregation module, a multi-scale cross-attention backprojection unit, and a convolutional gated feedforward network cascaded in sequence. The process of using the nonlinear backbone network to refine and correct the shallow features of the image and restore detailed features, outputting depth features, includes: The local feature aggregation module is used to extract local features from the shallow features of the image, which are then used as input features for the multi-scale cross-attention back projection unit. The multi-scale cross-attention back projection unit is used to process the local features of the image, extract the image region-level features and the image global features, model the non-local dependencies within the scale and the multi-scale similarity, and fuse the multi-scale similarity features to recover high-frequency information and output intermediate features. The convolutional gated feedforward network is used to model the channel information interaction of the intermediate features, aggregate the channel information to refine the features, and output the deep features.

[0010] Preferably, the local feature aggregation module consists of pointwise convolutional layers, nonlinear activation layers, depthwise convolutional layers, and depthwise dilated convolutions; The step of extracting local features from the shallow features of the image using the local feature aggregation module includes: The pointwise convolutional layer is used to perform pointwise convolution operations on the shallow features of the image to aggregate channel information; The nonlinear activation layer is used to perform nonlinear activation processing on the output features of the pointwise convolutional layer, thereby enhancing the nonlinear mapping capability and the sparsity of the features; By employing the combination of depthwise convolution and depthwise dilation convolution, a receptive field larger than the maximum pooling window of the multi-scale cross-attention back projection unit is obtained to extract local spatial features. The spatial local features are scaled using a learnable residual scaling factor, and then fused with the output features of the nonlinear activation layer through residual connections to obtain the local image features. The process of extracting local features from the image satisfies the following formula:

[0011] in, Represents shallow features of the image. Representing local features of an image, Represents a non-linear activation function. This represents the learnable residual scaling factor. This indicates a pointwise convolutional layer. This indicates a depthwise convolutional layer. This indicates depthwise dilation of the convolution.

[0012] Preferably, the multi-scale cross-attention back-projection unit includes a max-pooling downsampling layer, a spatially gated attention module, a multi-scale cross-attention fusion network, a channel attention module, and a nearest neighbor upsampling layer; The multi-scale cross-attention backprojection unit is used to process the local features of the image, extracting image region-level features and global features, modeling intra-scale non-local dependencies and multi-scale similarity, and fusing multi-scale similarity features to recover high-frequency information, outputting intermediate features, including: The max pooling downsampling layer is used to perform max pooling downsampling on the local features of the image to extract region-level features. The spatial gated attention module is used to model nonlocal dependencies of the image region-level features, aggregate global context information, and output global image features. The spatial gated attention module includes three pointwise convolutional layers, a nonlinear activation layer, a depthwise convolutional layer, and a depthwise dilated convolution consisting of two orthogonal strip-shaped depthwise dilated convolutional layers. The cascaded combination of the depthwise convolutional layer and the depthwise dilated convolutional layer aggregates global context information for the output features of the first pointwise convolution, and modulates the output features of the second pointwise convolutional layer with the captured global context information through a gating mechanism. The multi-scale cross-attention fusion network is used to model the multi-scale similarity of the global features of the image, and the multi-scale similar features are fused with the global features of the image to refine and enhance high-frequency information and output fused features. The channel attention module is used to model the inter-channel dependencies of the fused features to obtain channel attention scores to modulate the local features of the image; The nearest neighbor upsampling layer is used to upsample the downsampling residual learned by the multi-scale cross-attention fusion network to the size of the local features of the image, thereby recovering high-frequency information and outputting high-frequency features. The downsampling residual is obtained by subtracting the fused features from the image region-level features. The image region-level feature extraction process satisfies the following formula:

[0013] In the above formula, Represents image region-level features. This indicates a max-pooling downsampling layer. Represents local features of an image; The extraction process of the global features of the image satisfies the following formula:

[0014] In the above formula, Represents global features of the image. The symbol ⊙ represents image region-level features, and ⊙ represents element-wise multiplication. This represents the GeLU nonlinear activation layer. This indicates a pointwise convolutional layer. This indicates a depthwise convolutional layer. This indicates a convolutional layer that expands at depth.

[0015] Preferably, the multi-scale cross-attention fusion network includes three depthwise separable convolutional layers, multi-scale cross-attention units connected to each depthwise separable convolutional layer, and a fusion module; The process involves using the multi-scale cross-attention fusion network to model the multi-scale similarity of the global features of the image, fusing the multi-scale similarity features with the global features of the image, refining and enhancing high-frequency information, and outputting fused features, including: Three stacked depthwise separable convolutional layers are used to extract three progressively larger scale features from the global features of the image; The multi-scale cross-attention unit is based on cross-scale cross-attention. It uses the global features of the image as the query matrix, constructs a key matrix and a value matrix on the scale features extracted by the depthwise separable convolutional layer, calculates the cross-scale cross-attention score matrix with the global features of the image, and aggregates the value matrix of the scale according to the cross-scale cross-attention score matrix to obtain the scale features similar to the global features of the image at the scale. The calculation process of the cross-scale attention score matrix satisfies the following formula:

[0016] In the above formula, The global features of the image are used as the query matrix. This represents the transpose of a key matrix constructed from features at different scales. It is the dimension of the query and key matrix. This represents the normalized exponential function; The fusion module adds the similar features at all different scales to the global image features and outputs the fused features. The fusion feature satisfies the following formula:

[0017] In the above formula, Indicates fusion features, Indicates the first Output features of cross-scale cross-attention =1, 2, 3, Represents a non-linear activation function. This indicates a pointwise convolutional layer. It represents the global features of the image.

[0018] Preferably, the intermediate feature satisfies the following formula:

[0019] in, Indicates intermediate features, This represents the shallow features of the input image after layer normalization. Indicates fusion features, Indicates the nearest neighbor upsampling layer. This indicates the channel attention module.

[0020] Preferably, the convolutional gated feedforward network includes a first pointwise convolutional layer, a nonlinear activation layer, a depthwise convolutional layer, and a second pointwise convolutional layer; The process of using the convolutional gated feedforward network to model the channel information interaction of the intermediate features, aggregating channel information to refine features, and outputting deep features includes: The first pointwise convolutional layer aggregates channel features, maps the output feature channel dimension to twice that of the input, and evenly divides the output feature into a first part and a second part in the channel dimension; The second part of the features is nonlinearly activated using the aforementioned nonlinear activation layer; The depthwise convolutional layer is used to aggregate spatial local information of the second part of the features after nonlinear activation, and a gating mechanism is used to modulate the first part of the features using this spatial context information to enhance the ability of local structure recovery and nonlinear mapping. The second pointwise convolutional layer is used to perform pointwise convolution operations on the gated output results to complete the channel information interaction and output depth features.

[0021] Preferably, the processing of the convolutional gated feedforward network satisfies the following formula:

[0022]

[0023] In the above formula, where, Representing depth features, As an intermediate feature, This indicates that the hidden layer features of the feedforward network are evenly distributed along the channel dimension. This indicates a uniform segmentation operation along the channel dimension. This indicates a pointwise convolutional layer. Represents a non-linear activation function. ⊙ represents a depthwise convolutional layer, and ⊙ represents an element-wise multiplication operation.

[0024] Compared with the closest prior art, the present invention application has the following beneficial effects: First, high-frequency information recovery based on multi-scale similarity. This invention designs a cross-scale cross-attention mechanism based on the prior knowledge of multi-scale image similarity, and develops a multi-scale cross-attention fusion network on this basis. This network can effectively model the multi-scale similarity of images and recover high-frequency information by fusing multi-scale similarity features. Compared with existing Transformer methods that only focus on non-local dependencies, this invention can more accurately recover the correct texture patterns and detailed structures, achieving a superior structural similarity index on complex texture test sets such as Urban100.

[0025] Second, efficient modeling of intra-scale nonlocal dependencies. Based on the large kernel attention design concept, this invention develops a spatially gated attention module with strip-shaped large kernel depthwise convolution and depthwise dilated convolution structures. This module can efficiently model intra-scale nonlocal dependencies, aggregate global directional context information, and significantly reduce the number of parameters and computational overhead by decomposing the two-dimensional depthwise dilated convolution into two orthogonal one-dimensional strip-shaped convolutions.

[0026] Third, the simplified backprojection unit and reduced optimization difficulty. This invention utilizes multi-scale similarity prior learning of images to recover downsampling residuals, thereby simplifying the structure of the backprojection unit and focusing the network optimization objective on recovering more realistic high-frequency details. This design significantly reduces network parameters and computational overhead, lowers training difficulty, and simultaneously improves the quality of super-resolution reconstruction.

[0027] Based on the aforementioned core functional modules and structural design, this invention constructs a lightweight image super-resolution network—Multi-Scale Cross-Attention Back Projection Network (MSCABPN), which achieves excellent super-resolution reconstruction performance on multiple benchmark test sets and surpasses Transformer-based models such as SwinIR-light in structural recovery, while significantly reducing the number of parameters and computational cost. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the overall structure of the MSCABPN network provided in this invention application; Figure 2 A schematic diagram of the structure of the multi-scale cross-attention back projection block MSCABPB provided in this invention application; Figure 3 This is a schematic diagram of the Local Feature Aggregation Module (LFA) provided in this invention application; Figure 4 A schematic diagram of the structure of the multi-scale cross-attention back projection unit MSCABPU provided in this invention application; Figure 5 This is a schematic diagram of the structure of the spatial gating attention SGAttn provided in this invention application; Figure 6 This is a schematic diagram of the structure of the multi-scale cross-attention fusion network MSCAFN provided in this invention application; Figure 7 This is a schematic diagram of the structure of the Convolutional Gated Feedforward Network (CGFN) provided in this invention application; Figure 8 A comparison of the actual reconstruction results of the MSCABPN method provided in this application and the classical method on the Urban100 test set for image 092. Detailed Implementation

[0029] The specific embodiments of this invention will be further described in detail below with reference to the accompanying drawings.

[0030] It should be noted that the core of the image super-resolution method proposed in this invention lies in constructing and training a multi-scale cross-attention backprojection network (MSCABPN) to obtain an image super-resolution reconstruction model. Utilizing the inherent multi-scale similarity prior of the image, the model efficiently recovers the detailed texture of high-resolution images while controlling the number of model parameters, thereby improving the quality of super-resolution reconstruction. The specific embodiments of this invention are described in detail below with reference to the accompanying drawings.

[0031] This invention application proposes an image super-resolution reconstruction method based on a multi-scale similarity back projection network, the method comprising the following steps: Step 1: Construct the network. Construct a multi-scale cross-attention backprojection network based on the multi-scale similarity of images.

[0032] Step 2: Train the model. Use the paired image dataset with degradation downsampling as the training dataset to train the multi-scale cross-attention backprojection network in a supervised manner to obtain the trained image super-resolution reconstruction model.

[0033] Step 3: Inference Reconstruction. Input the image whose resolution needs to be improved into the trained image super-resolution reconstruction model and perform inference to obtain the enhanced resolution image.

[0034] Furthermore, the multi-scale cross-attention back projection network includes a shallow feature extraction module, a nonlinear backbone network, and a reconstruction module cascaded in sequence. The construction of a multi-scale cross-attention backprojection network based on multi-scale image similarity includes the following steps: Step 2.1: Using the shallow feature extraction module, a blueprint separable convolutional layer consisting of a pointwise convolutional layer and a depthwise convolutional layer is used to perform convolution operations on the input image to extract shallow features of the image; Step 2.2: Use the nonlinear backbone network to refine and correct the shallow features of the image and restore the detailed features, outputting the depth features; Step 2.3: The reconstruction module is used to fuse the shallow features and deep features of the image to obtain super-resolution reconstructed image features. The super-resolution reconstructed image features are then mapped back to image pixel information. Finally, the super-resolution image information is periodically permuted from the channel dimension to the spatial dimension to increase the image resolution, and the super-resolution reconstructed image is output. The shallow feature extraction process of the image described herein satisfies the following formula:

[0035] In the above formula, For shallow image features, The image is used as input to the shallow feature extraction module. This indicates a pointwise convolutional layer. This indicates a depthwise convolutional layer.

[0036] like Figure 1 As shown, after the shallow features of the image are extracted, the nonlinear backbone network further refines and corrects the shallow features of the image, recovers the high-frequency features, and outputs the image depth features.

[0037] Furthermore, the nonlinear backbone network includes multiple residual backprojection groups (six in an example) and a blueprint separable convolutional layer. Each residual backprojection group combines multiple multi-scale cross-attention backprojection blocks (two in an example) and a blueprint separable convolutional layer through residual connections. The multi-scale cross-attention backprojection block includes a local feature aggregation module, a multi-scale cross-attention backprojection unit, and a convolutional gated feedforward network cascaded in sequence. Step 2.2, which involves using the nonlinear backbone network to refine and correct the shallow features of the image and restore detailed features, outputting depth features, includes: Step 2.2.1: Use the local feature aggregation module to extract local features from the shallow features of the image, and use them as input features for the multi-scale cross-attention back projection unit; Step 2.2.2: The multi-scale cross-attention back projection unit is used to process the local features of the image, extract the image region-level features and the image global features, model the non-local dependencies within the scale and the multi-scale similarity, and fuse the multi-scale similarity features to recover high-frequency information and output the intermediate features; Step 2.2.3: Use the convolutional gated feedforward network to perform channel information interaction modeling on the intermediate features, aggregate channel information to refine features, and output deep features.

[0038] In the above steps, the local feature aggregation module is one of the key components of this invention. For example... Figure 3 As shown, this module replaces traditional large-kernel convolutional layers with a cascaded combination of depthwise convolutional layers and depthwise dilated convolutional layers, achieving a larger equivalent receptive field with fewer parameters. Simultaneously, this module introduces learnable residual scaling factors, which can enhance feature representation capabilities while maintaining training stability.

[0039] like Figure 2 As shown, after the local image features are extracted, the multi-scale cross-attention backprojection unit continues to process the local image features. This backprojection unit is the core module of this invention for realizing multi-scale similarity modeling and high-frequency information recovery. Figure 4 As shown, this unit consists of an 8×8 window max pooling downsampling layer, a spatial gated attention module (SGAttn), a multi-scale cross-attention fusion network (MSCAFN), a channel attention module (CAttn), and a nearest neighbor upsampling layer.

[0040] Furthermore, the local feature aggregation module consists of pointwise convolutional layers, nonlinear activation layers, depthwise convolutional layers, and depthwise dilated convolutions; Step 2.2.1, which involves using the local feature aggregation module to extract local features from the shallow features of the image, includes: Step a: Perform point-by-point convolution operation on the shallow features of the image using the point-by-point convolutional layer to aggregate channel information; Step b: The nonlinear activation layer is used to perform nonlinear activation processing on the output features of the pointwise convolutional layer to enhance the nonlinear mapping capability and the sparsity of the features; Step c: Using the combination of depthwise convolution and depthwise dilation convolution, a receptive field larger than the maximum pooling window of the multi-scale cross-attention back projection unit is obtained to extract local spatial features; Step d: The spatial local features are scaled using learnable residual scaling factors, and then fused with the output features of the nonlinear activation layer through residual connections to obtain the local image features; The process of extracting local features from the image satisfies the following formula:

[0041] in, Represents shallow features of the image. Representing local features of an image, Represents a non-linear activation function. This represents the learnable residual scaling factor. This indicates a pointwise convolutional layer. This indicates a depthwise convolutional layer. This indicates a convolutional layer that expands at depth.

[0042] Furthermore, the multi-scale cross-attention back projection unit includes a max pooling downsampling layer, a spatially gated attention module, a multi-scale cross-attention fusion network, a channel attention module, and a nearest neighbor upsampling layer; Step 2.2.2 describes using the multi-scale cross-attention backprojection unit to process the local features of the image, extracting image region-level features and global image features, modeling intra-scale non-local dependencies and multi-scale similarity, and fusing multi-scale similarity features to recover high-frequency information, outputting intermediate features, including: Step B1: Max pooling downsampling is applied to the local features of the image to extract region-level features; Step B2: The spatial gated attention module is used to model nonlocal dependencies of the image region-level features, aggregate global context information, and output global image features. The spatial gated attention module includes three pointwise convolutional layers, a nonlinear activation layer, a depthwise convolutional layer, and a depthwise dilated convolution consisting of two orthogonal strip-shaped depthwise dilated convolutional layers. The cascaded combination of the depthwise convolutional layer and the depthwise dilated convolutional layer aggregates global context information for the output features of the first pointwise convolution, and modulates the output features of the second pointwise convolutional layer with the captured global context information through a gating mechanism. Step B3: The multi-scale cross-attention fusion network is used to model the multi-scale similarity of the global features of the image, and the multi-scale similarity features are fused with the global features of the image to refine and enhance high-frequency information and output fused features; Step B4: Model the inter-channel dependencies of the fused features using the channel attention module to obtain channel attention scores to modulate the local features of the image; Step B5: The nearest neighbor upsampling layer is used to upsample the downsampling residual learned by the multi-scale cross-attention fusion network to the size of the local features of the image, recover high-frequency information, and output high-frequency features. The downsampling residual is obtained by subtracting the fused features from the image region-level features. In step B2, the spatial gating attention module adaptively aggregates global context information through a gating mechanism, such as... Figure 5 As shown, this module decomposes the standard depthwise dilation convolution into two orthogonal one-dimensional strip-shaped depthwise dilation convolutions to reduce computation. By combining depthwise convolution and strip-shaped depthwise dilation convolution, a larger rectangular receptive field is obtained, nonlocal dependencies are modeled, global directional features are extracted, and the number of parameters and computation of the module are reduced.

[0043] The image region-level feature extraction process satisfies the following formula:

[0044] In the above formula, Represents image region-level features. This indicates a max-pooling downsampling layer. Represents local features of an image; The extraction process of the global features of the image satisfies the following formula:

[0045] In the above formula, Represents global features of the image. The symbol ⊙ represents image region-level features, and ⊙ represents element-wise multiplication. This represents the GeLU nonlinear activation layer. This indicates a pointwise convolutional layer. This indicates a depthwise convolutional layer. This indicates a convolutional layer that expands at depth.

[0046] After adaptively aggregating global contextual information through spatial gating attention and outputting global image features, this invention further utilizes multi-scale similarity priors of the image to enhance high-frequency information. To this end, this invention designs a multi-scale cross-attention fusion network. For example... Figure 6 As shown, the fusion network includes three depthwise separable convolutional layers, cross-scale attention units connected to each depthwise separable convolutional layer, and a fusion module. For example, the three depthwise separable convolutional layers can use 3×3, 5×5, and 5×5 convolutional kernels respectively, and are stacked to extract features at progressively larger scales.

[0047] Furthermore, the multi-scale cross-attention fusion network includes three depthwise separable convolutional layers, multi-scale cross-attention units connected to each depthwise separable convolutional layer, and a fusion module; Step B3, which involves using the multi-scale cross-attention fusion network to model the multi-scale similarity of the global features of the image, fusing the multi-scale similarity features with the global features of the image, refining and enhancing high-frequency information, and outputting fused features, includes: Step B3.1: Three stacked depthwise separable convolutional layers are used to extract three progressively larger scale features from the global features of the image; Step B3.2: The multi-scale cross-attention unit is based on cross-scale cross-attention. It uses the global features of the image as the query matrix, constructs a key matrix and a value matrix on the scale features extracted by the depthwise separable convolutional layer, calculates the cross-scale cross-attention score matrix with the global features of the image, and aggregates the value matrix of the scale according to the cross-scale cross-attention score matrix to obtain the scale features similar to the global features of the image at the scale. The calculation process of the cross-scale attention score matrix satisfies the following formula:

[0048] In the above formula, The global features of the image are used as the query matrix. This represents the transpose of a key matrix constructed from features at different scales. It is the dimension of the query and key matrix. This represents the normalized exponential function; Step B3.3: The fusion module is used to add all the similar features at different scales to the global features of the image, and output the fused features. The fusion feature satisfies the following formula:

[0049] In the above formula, Indicates fusion features, Indicates the first Output features of cross-scale cross-attention =1, 2, 3, Represents a non-linear activation function. This indicates a pointwise convolutional layer. It represents the global features of the image.

[0050] like Figure 6 As shown, cross-scale cross-attention extends the self-attention mechanism from a single scale to cross-scale scenarios. Specifically, for the input global features, a key matrix and a value matrix are constructed on the scale features extracted by depthwise separable convolution, using these as the query matrix. Then, the attention score matrix is ​​obtained by matrix multiplication of the query matrix and the key matrix. Finally, the value matrix is ​​aggregated using the attention scores through matrix multiplication to obtain features that are similar to the input global features across scales. The cross-scale cross-attention score matrix is ​​calculated using scaled dot product attention, effectively modeling the nonlocal similarity of cross-scale features.

[0051] The multi-scale cross-attention fusion network first performs cross-scale cross-attention calculations at three scales sequentially, obtaining similar features at three different scales. These three cross-scale similar features are then simply added to the global image features and aggregated before being fed into the fusion module. The fusion module consists of a pointwise convolutional layer and a non-linear activation layer, completing the channel information interaction of the aggregated features. After multi-scale similarity modeling and multi-scale similar feature fusion, the fused features not only contain global image context information but also aggregate similar features from multiple scales, thus effectively enhancing the recovery of high-frequency detail information.

[0052] After obtaining the fusion features, such as Figure 4As shown, the multi-scale cross-attention backprojection unit, on the one hand, upsamples the downsampled residual between the fused features and the image region-level features to the size of the local image features through the nearest neighbor upsampling layer, thus recovering high-frequency features; on the other hand, it inputs the fused features into the channel attention module, aggregating channel information to obtain channel attention scores, thereby modulating the local image features. Finally, the upsampled high-frequency features, the channel attention-modulated local image features, and the input local image features are aggregated through residual connections to output intermediate features enhanced with high-frequency information. These intermediate features satisfy the following formula:

[0053] in, Indicates intermediate features, This represents the shallow features of the input image after layer normalization. Indicates fusion features, Indicates the nearest neighbor upsampling layer. This indicates the channel attention module.

[0054] The output features of the aforementioned multi-scale cross-attention backprojection units are then fed into a convolutionally gated feedforward network. For example... Figure 7 As shown, this network introduces depthwise convolution and gating mechanisms on the basis of traditional feedforward networks to enhance the ability to model local spatial structures.

[0055] Furthermore, the convolutional gated feedforward network includes a first pointwise convolutional layer, a nonlinear activation layer, a depthwise convolutional layer, and a second pointwise convolutional layer; Step 2.2.2.5, which involves using the convolutional gated feedforward network to model the channel information interaction of the intermediate features, aggregating channel information to refine the features, and outputting deep features, includes: Step f1: Aggregate channel features using the first pointwise convolutional layer, map the output feature channel dimension to twice that of the input, and evenly divide the output feature into a first part and a second part in the channel dimension; Step f2: The depthwise convolutional layer is used to aggregate spatial local information of the second part of the features after nonlinear activation, and a gating mechanism is used to modulate the first part of the features using the spatial context information to enhance the ability of local structure recovery and nonlinear mapping. Step f3: The second pointwise convolutional layer is used to perform pointwise convolution on the gated output to complete the channel information interaction and output deep features. The above convolutional gated feedforward network processes the input features as follows: First, after the input features pass through the first pointwise convolutional layer, they are uniformly divided into two parts in the channel dimension; then, the second part passes through nonlinear activation and depthwise convolutional layers in sequence to encode spatial local structure information, and is then multiplied element-wise with the first part; finally, the channel information is fused through the second pointwise convolutional layer to output the final deep features.

[0056] The processing procedure of the convolutional gated feedforward network described in step 2.2.2.5 satisfies the following formula:

[0057]

[0058] In the above formula, where, Indicates deep features, These are intermediate features of the input. This indicates that the hidden layer features of the feedforward network are evenly distributed along the channel dimension. This indicates a uniform segmentation operation along the channel dimension. This indicates a pointwise convolutional layer. Represents a non-linear activation function. ⊙ represents a depthwise convolutional layer, and ⊙ represents an element-wise multiplication operation.

[0059] Through the processing of the multi-scale cross-attention backprojection blocks described above, the nonlinear backbone network achieves refinement correction of shallow features and restoration of high-frequency details. For example... Figure 1 As shown, the nonlinear backbone network adds shallow and deep features through long residual connections to obtain high-resolution image estimation features, which are then input into the reconstruction module. The reconstruction module consists of a blueprint-separable convolutional layer and a sub-pixel convolutional layer (PixelShuffle). It first aggregates the high-resolution image estimation features to obtain estimated high-resolution image information, and then periodically permutes the image information from the channel dimension to the spatial dimension to increase the image resolution, finally outputting a super-resolution reconstructed image.

[0060] Experimental test data: To verify the effectiveness of the proposed Multi-Scale Cross-Attention Backprojection Network (MSCABPN), this invention underwent thorough experimental verification. The following is a summary... Figure 8 The experimental results are explained in detail in Tables 1 to 4.

[0061] This invention uses a publicly available paired image dataset with degraded downsampling as the training set, where the low-resolution images are obtained by bicubic downsampling degradation. For training data processing, this invention randomly crops 96×96 sub-images from the low-resolution images as input for training, sets the training batch size to 32, and performs random horizontal and vertical flipping and rotation on the training images for data augmentation. This invention uses the PyTorch framework to build the network and completes training on a graphics processing unit.

[0062] To qualitatively and quantitatively verify the effectiveness of the proposed Multi-Scale Cross-Attention Backprojection Network (MSCABPN), MSCABPN is compared with several typical image super-resolution methods, including lightweight methods based on convolutional neural networks and Transformers. The super-resolution reconstruction performance of the model is evaluated on multiple standard benchmark sets. For quantitative evaluation, this invention calculates the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) of the super-reconstructed image in the Y channel (i.e., the luminance channel) of the YCbCr color space, using these as quantitative evaluation indicators of the model's super-resolution reconstruction quality.

[0063] The super-resolution algorithms compared include: Dense Depth Back Projection Network (D-DBPN), ShuffleMixer image super-resolution convolutional network based on channel segmentation and mixing, a lightweight model of image restoration network based on Swin Transformer (SwinIR-light), and Efficient Super-Resolution Transformer Network (ESRT).

[0064] Quantitative comparison results: Tables 1, 2, and 3 show the objective evaluation results of the super-resolution reconstructed image quality of each method under different upsampling factors (×2, ×3, ×4).

[0065] Table 1. Comparison of ×2 super-resolution reconstruction quality with other methods on five benchmark sets.

[0066] Table 2. Comparison of ×3 super-resolution reconstruction quality with other methods on five benchmark sets.

[0067] Table 3. Comparison of ×4 super-resolution reconstruction quality with other methods on five benchmark sets.

[0068] To fairly compare the performance of super-resolution reconstruction, this invention also compares the number of parameters and floating-point computations used by each method to reconstruct an image ×4 to 1280×720 resolution. Table 4 lists the number of parameters and floating-point computations (FLOPs) of the ×4 upsampling model for each method.

[0069] Table 4 Comparison of parameter count and computational cost of ×4 super-resolution reconstruction models with other methods

[0070] The experimental results in Tables 1 to 3 show that the proposed multi-scale cross-attention back projection network MSCABPN achieves the best results in structural similarity index (SSIM) on multiple test sets, and its peak signal-to-noise ratio (PSNR) is comparable to or even better than that of complex models such as SwinIR-light. Table 4 shows a comparison of the number of parameters and computational cost of each method in the ×4 super-resolution task.

[0071] As can be seen from Table 4, the MSCABPN proposed in this invention achieves excellent super-resolution reconstruction performance with only 659.2K parameters and 16.9 GFLOPs of computation, which is significantly better than the comparison methods.

[0072] The test results show that: 1. Super-resolution reconstruction quality: MSCABPN achieved state-of-the-art results in Structure Integral Simulation (SSIM), demonstrating the effectiveness of MSCABPN in recovering high-frequency detail textures using multi-scale similarity. It also achieved reconstruction quality comparable to or even better than classic Transformer-based methods such as SwinIR-light in terms of PSNR. For example, on the Manga109 dataset, its ×3 super-resolution reconstruction results outperformed SwinIR-light by 0.21 dB, while reducing parameter count and computational cost by 29.9% and 74.7%, respectively.

[0073] 2. Lightweight Model: The proposed MSCABPN has only 659.2K parameters and 16.9 GFLOPs computation in the ×4 super-resolution task, which is significantly lower than the comparison methods, achieving a good balance between model complexity and super-resolution reconstruction quality.

[0074] 3. Visual effects: such as Figure 8As shown, in the comparison of actual super-resolution reconstruction results for complex textured images on the Urban100 test set, the adjacent structural patterns in the red-boxed area are interwoven horizontally and vertically, and the overall scene is dark. Only MSCABPN recovered the correct and clear texture, verifying that the cross-scale attention of this invention can effectively model non-local multi-scale similarity. By fusing multi-scale similarity features, it corrects and recovers the correct and clear texture patterns. While SwinIR-light, even with a more complex self-attention module modeling the non-local dependencies of the image, failed to effectively utilize the multi-scale similarity of the image, the super-resolution reconstructed textures were mostly distorted, overlapping, and blurred. This proves that the prior knowledge of multi-scale similarity of the image can effectively guide the network to recover the correct and clear texture patterns and detailed structures.

[0075] In summary, the multi-scale cross-attention backprojection network based on the multi-scale similarity prior design of images can effectively recover correct and clear high-frequency information, improve the quality of image super-resolution, and significantly reduce the number of parameters and computational overhead. It solves the problem that existing methods cannot balance super-resolution reconstruction quality and computational efficiency, and is applicable to fields such as medical imaging, security monitoring, aerospace remote sensing, and ultra-high-definition video transmission and display.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit its protection scope. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading the present invention, they can still make various changes, modifications or equivalent substitutions to the specific implementation of the application, but these changes, modifications or equivalent substitutions are all within the protection scope of the claims pending approval.

Claims

1. An image super-resolution method based on multi-scale similarity back projection, characterized in that, The method includes: A multi-scale cross-attention backprojection network is constructed based on the multi-scale similarity of images; The image whose resolution needs to be improved is input into the trained image super-resolution reconstruction model for inference, and the enhanced resolution image is obtained. The image super-resolution reconstruction model is trained on the multi-scale cross-attention backprojection network in a supervised manner using a paired image dataset with degraded downsampling as the training dataset.

2. The method according to claim 1, characterized in that, The multi-scale cross-attention back projection network includes a shallow feature extraction module, a nonlinear backbone network, and a reconstruction module cascaded in sequence. The multi-scale cross-attention backprojection network constructed based on image multi-scale similarity includes: The shallow feature extraction module uses a blueprint-separable convolutional layer consisting of pointwise convolutional layers and depthwise convolutional layers to perform convolution operations on the input image to extract shallow features of the image. The nonlinear backbone network is used to refine and correct the shallow features of the image and restore the detailed features, outputting the depth features; The reconstruction module is used to fuse the shallow features and deep features of the image to obtain super-resolution reconstructed image features. Then, the super-resolution reconstructed image features are mapped back to image pixel information. Finally, the super-resolution image information is periodically permuted from the channel dimension to the spatial dimension to increase the image resolution and output the super-resolution reconstructed image. The shallow feature extraction process of the image described herein satisfies the following formula: In the above formula, For shallow image features, The image is used as input to the shallow feature extraction module. This indicates a pointwise convolutional layer. This indicates a depthwise convolutional layer.

3. The method according to claim 2, characterized in that, The nonlinear backbone network includes multiple residual backprojection groups and a blueprint separable convolutional layer. Each residual backprojection group combines multiple multi-scale cross-attention backprojection blocks and a blueprint separable convolutional layer through residual connections. The multi-scale cross-attention backprojection block includes a local feature aggregation module, a multi-scale cross-attention backprojection unit, and a convolutional gated feedforward network cascaded in sequence. The process of using the nonlinear backbone network to refine and correct the shallow features of the image and restore detailed features, outputting depth features, includes: The local feature aggregation module is used to extract local features from the shallow features of the image, which are then used as input features for the multi-scale cross-attention back projection unit. The multi-scale cross-attention back projection unit is used to process the local features of the image, extract the image region-level features and the image global features, model the non-local dependencies within the scale and the multi-scale similarity, and fuse the multi-scale similarity features to recover high-frequency information and output intermediate features. The convolutional gated feedforward network is used to model the channel information interaction of the intermediate features, aggregate the channel information to refine the features, and output the deep features.

4. The method according to claim 3, characterized in that, The local feature aggregation module consists of pointwise convolutional layers, nonlinear activation layers, depthwise convolutional layers, and depthwise dilated convolutions. The step of extracting local features from the shallow features of the image using the local feature aggregation module includes: The pointwise convolutional layer is used to perform pointwise convolution operations on the shallow features of the image to aggregate channel information; The nonlinear activation layer is used to perform nonlinear activation processing on the output features of the pointwise convolutional layer, thereby enhancing the nonlinear mapping capability and the sparsity of the features; By employing the combination of depthwise convolution and depthwise dilation convolution, a receptive field larger than the maximum pooling window of the multi-scale cross-attention back projection unit is obtained to extract local spatial features. The spatial local features are scaled using a learnable residual scaling factor, and then fused with the output features of the nonlinear activation layer through residual connections to obtain the local image features. The process of extracting local features from the image satisfies the following formula: in, Represents shallow features of the image. Representing local features of an image, Represents a non-linear activation function. This represents the learnable residual scaling factor. This indicates a pointwise convolutional layer. This indicates a depthwise convolutional layer. This indicates depthwise dilation of the convolution.

5. The method according to claim 3, characterized in that, The multi-scale cross-attention back projection unit includes a max pooling downsampling layer, a spatial gated attention module, a multi-scale cross-attention fusion network, a channel attention module, and a nearest neighbor upsampling layer; The multi-scale cross-attention backprojection unit is used to process the local features of the image, extracting image region-level features and global features, modeling intra-scale non-local dependencies and multi-scale similarity, and fusing multi-scale similarity features to recover high-frequency information, outputting intermediate features, including: The max pooling downsampling layer is used to perform max pooling downsampling on the local features of the image to extract region-level features. The spatial gated attention module is used to model nonlocal dependencies of the image region-level features, aggregate global context information, and output global image features. The spatial gated attention module includes three pointwise convolutional layers, a nonlinear activation layer, a depthwise convolutional layer, and a depthwise dilated convolution consisting of two orthogonal strip-shaped depthwise dilated convolutional layers. The cascaded combination of the depthwise convolutional layer and the depthwise dilated convolutional layer aggregates global context information for the output features of the first pointwise convolution, and modulates the output features of the second pointwise convolutional layer with the captured global context information through a gating mechanism. The multi-scale cross-attention fusion network is used to model the multi-scale similarity of the global features of the image, and the multi-scale similar features are fused with the global features of the image to refine and enhance high-frequency information and output fused features. The channel attention module is used to model the inter-channel dependencies of the fused features to obtain channel attention scores to modulate the local features of the image; The nearest neighbor upsampling layer is used to upsample the downsampling residual learned by the multi-scale cross-attention fusion network to the size of the local features of the image, thereby recovering high-frequency information and outputting high-frequency features. The downsampling residual is obtained by subtracting the fused features from the image region-level features. The image region-level feature extraction process satisfies the following formula: In the above formula, Represents image region-level features. This indicates a max-pooling downsampling layer. Represents local features of an image; The extraction process of the global features of the image satisfies the following formula: In the above formula, Represents global features of the image. The symbol ⊙ represents image region-level features, and ⊙ represents element-wise multiplication. This represents the GeLU nonlinear activation layer. This indicates a pointwise convolutional layer. This indicates a depthwise convolutional layer. This indicates a convolutional layer that expands at depth.

6. The method according to claim 5, characterized in that, The multi-scale cross-attention fusion network includes three depthwise separable convolutional layers, multi-scale cross-attention units connected to each depthwise separable convolutional layer, and a fusion module. The process involves using the multi-scale cross-attention fusion network to model the multi-scale similarity of the global features of the image, fusing the multi-scale similarity features with the global features of the image, refining and enhancing high-frequency information, and outputting fused features, including: Three stacked depthwise separable convolutional layers are used to extract three progressively larger scale features from the global features of the image; The multi-scale cross-attention unit is based on cross-scale cross-attention. It uses the global features of the image as the query matrix, constructs a key matrix and a value matrix on the scale features extracted by the depthwise separable convolutional layer, calculates the cross-scale cross-attention score matrix with the global features of the image, and aggregates the value matrix of the scale according to the cross-scale cross-attention score matrix to obtain the scale features similar to the global features of the image at the scale. The calculation process of the cross-scale attention score matrix satisfies the following formula: In the above formula, The global features of the image are used as the query matrix. This represents the transpose of a key matrix constructed from features at different scales. It is the dimension of the query and key matrix. This represents the normalized exponential function; The fusion module adds the similar features at all different scales to the global image features and outputs the fused features. The fusion feature satisfies the following formula: In the above formula, Indicates fusion features, Indicates the first Output features of cross-scale cross-attention =1, 2, 3, Represents a non-linear activation function. This indicates a pointwise convolutional layer. It represents the global features of the image.

7. The method according to claim 6, characterized in that, The intermediate feature satisfies the following formula: in, Indicates intermediate features, This represents the shallow features of the input image after layer normalization. Indicates fusion features, Indicates the nearest neighbor upsampling layer. This indicates the channel attention module.

8. The method according to claim 4, characterized in that, The convolutional gated feedforward network includes a first pointwise convolutional layer, a nonlinear activation layer, a depthwise convolutional layer, and a second pointwise convolutional layer. The process of using the convolutional gated feedforward network to model the channel information interaction of the intermediate features, aggregating channel information to refine features, and outputting deep features includes: The first pointwise convolutional layer aggregates channel features, maps the output feature channel dimension to twice that of the input, and evenly divides the output feature into a first part and a second part in the channel dimension; The second part of the features is nonlinearly activated using the aforementioned nonlinear activation layer; The depthwise convolutional layer is used to aggregate spatial local information of the second part of the features after nonlinear activation, and a gating mechanism is used to modulate the first part of the features using this spatial context information to enhance the ability of local structure recovery and nonlinear mapping. The second pointwise convolutional layer is used to perform pointwise convolution operations on the gated output results to complete the channel information interaction and output depth features.

9. The method according to claim 8, characterized in that, The processing of the convolutional gated feedforward network satisfies the following formula: In the above formula, where, Representing depth features, As an intermediate feature, This indicates that the hidden layer features of the feedforward network are evenly distributed along the channel dimension. This indicates a uniform segmentation operation along the channel dimension. This indicates a pointwise convolutional layer. Represents a non-linear activation function. ⊙ represents a depthwise convolutional layer, and ⊙ represents an element-wise multiplication operation.

Images