Wavelet-space collaborative based progressive super-resolution reconstruction method and device

Abstract

The application discloses a wavelet-space collaborative gradual super-resolution reconstruction method and device, relates to the field of image super-resolution reconstruction, and comprises the following steps: S1, a wavelet-space collaborative enhancement gradual reconstruction super-resolution network comprising a first convolutional layer, a plurality of wavelet-space collaborative enhancement blocks and a second convolutional layer is constructed; after low-resolution images extract shallow features through the first convolutional layer, deep feature extraction is performed on the low-resolution images by using the wavelet-space collaborative enhancement blocks connected in sequence, and then the low-resolution images are reconstructed through the second convolutional layer to generate a reconstructed image; S2, the network is trained by using the low-resolution images; and S3, a super-resolution reconstruction network is obtained by using the trained network to perform super-resolution reconstruction to obtain a reconstructed image. The wavelet-space collaborative down-sampling module combining channel down-sampling and spatial down-sampling is introduced into the double-layer routing Transformer architecture and Mamba, so that the calculation cost is reduced, and the detail definition and information integrity of the super-resolution image are improved.

Description

Technical Field

[0001] This invention relates to the field of image super-resolution reconstruction technology, and in particular to a progressive super-resolution reconstruction method and apparatus based on wavelet-space synergy. Background Technology

[0002] Image super-resolution (ISR) is an important task in computer vision, aiming to restore high-resolution images from degraded low-resolution images. Because high-resolution images contain rich detail, they are widely used in scenarios where tasks rely on capturing information from images to achieve interactive functionality, such as remote sensing, medical imaging, face recognition, and video surveillance in computer vision.

[0003] With the evolution and development of deep learning network backbones such as Transformer and Mamba, their powerful performance in text tasks has led to their widespread application in image processing tasks. In deep learning, the Transformer architecture, with its unique self-attention mechanism, enables the model to effectively capture contextual information. Related research confirms that this modeling of global dependencies demonstrates good performance in image SR tasks. However, as the input size increases, the computational overhead and complexity of Transformer also increase. This necessitates that designers optimize computational efficiency while ensuring the model captures complex dependencies. In recent years, Mamba has emerged as another rising network backbone after Transformer. Compared to Transformer, Mamba simplifies computational complexity while still capturing global dependencies.

[0004] Subsequent Transformer variants have all been designed to simplify computation and reduce computational complexity. Examples include the windowing mechanism of the Vision Transformer, the shifted window mechanism of the Swim Transformer, and the two-layer routing self-attention mechanism of the BiFormer. Using more informative pixels for feature aggregation can bring considerable performance gains to image SR; however, the dense and repetitive inefficient self-attention computation hinders the application of Transformers in image SR. Similarly, the traditional single-scale Mamba also affects its performance in image SR. Summary of the Invention

[0005] To address the above problems, this invention proposes a progressive super-resolution reconstruction method and apparatus based on wavelet-spatial collaboration. In the wavelet-assisted two-layer routing Transformer module that implements attention enhancement, a wavelet-spatial downsampling module is introduced for downsampling. This effectively avoids intensive self-attention computation without causing information loss, enabling the network to achieve better performance with less computational overhead. Furthermore, a wavelet-assisted omnidirectional state space scanning module is introduced to compensate for the lack of global contextual information modeling in the wavelet-assisted two-layer routing Transformer module. The introduction of the wavelet-spatial downsampling module in the wavelet-assisted omnidirectional state space scanning module enables multi-scale feature extraction while reducing computational complexity.

[0006] On the one hand, the gradual super-resolution reconstruction method based on wavelet-space coordination has the following specific steps:

[0007] S1, construct a progressive reconstruction super-resolution network including a first convolutional layer, several wavelet-spatial co-enhancement blocks, and a second convolutional layer; after the low-resolution image is processed by the first convolutional layer to extract shallow features, the wavelet-spatial co-enhancement blocks are connected in sequence to extract deep features, and then the image is reconstructed by the second convolutional layer to generate a reconstructed image.

[0008] The wavelet-spatial co-enhancement block includes MB convolution, a wavelet-assisted two-layer routing Transformer module, a first gated convolutional feedforward network, a wavelet-assisted omnidirectional state space scanning module, and a second gated convolutional feedforward network. The input features of the wavelet-spatial co-enhancement block are processed by MB convolution for local feature extraction, by wavelet-assisted two-layer routing Transformer module for attention enhancement, by first gated convolutional feedforward network for nonlinear optimization, by wavelet-assisted omnidirectional state space scanning module for global feature enhancement, and then reconstructed by second gated convolutional feedforward network before being output.

[0009] The wavelet-assisted dual-layer routing Transformer module uses a dual-layer routing Transformer architecture for region selection and performs attention calculation based on the selected region to achieve attention enhancement. The features involved in region selection are downsampled using a wavelet-spatial collaborative downsampling module to simplify dimensions. The wavelet-assisted omnidirectional state space scanning module, after its input is downsampled by the wavelet-spatial collaborative downsampling module to simplify the input dimension and retain multi-scale information, uses a state space model for global feature enhancement. The wavelet-spatial collaborative downsampling module fuses spatial downsampling and wavelet transform-based downsampling.

[0010] S2, use low-resolution images to train the wavelet-space co-enhanced progressive reconstruction super-resolution network to obtain the trained wavelet-space co-enhanced progressive reconstruction super-resolution network;

[0011] S3 uses a trained wavelet-spatial co-enhanced progressive reconstruction super-resolution network to perform super-resolution reconstruction on the low-resolution image to be reconstructed, and obtains the reconstructed image.

[0012] Preferably, the wavelet-assisted two-layer routing Transformer module is implemented as follows:

[0013] The input feature map is pre-partitioned into a window and converted into an input tensor. Where C represents the number of channels in the feature map; x and y represent the length and width of the window after partitioning, respectively; w1 and w2 represent the length and width of the pixels within the partitioned window, respectively.

[0014] The input tensor is rearranged in dimension to obtain... and Using wavelet-spatial co-sampling module to perform Perform downsampling and projection operations to obtain Then The input is fed into a linear convolutional layer with a spread factor of 2, and... The channel was expanded by 2 times. ;

[0015] right Perform window partitioning operations, Divide into N non-overlapping windows of size S ,in, , express Length; express The width; The same partitioning operation is used to obtain ,in, Indicates to The number of non-overlapping windows after window division. Then, using linear layers Will Convert to query vector ;

[0016] The key vector is obtained by splitting the key-value pair. Average pooling is used to transform the query vector Q and key vector K into an average query vector representing each partitioned small window. with average bond vector ;

[0017] Perform similarity calculation, generate a correlation map between windows, and select the top k windows with high correlation values ​​from the correlation map;

[0018] The top k windows with high correlation values ​​are selected from the Map and concatenated to obtain the result. Then Split to obtain the key vector AND value vector Perform fine-grained self-attention computation to obtain enhanced features. ;

[0019] The shape of the enhanced feature Y is changed from N×S 2 ×C is converted back to The output of the wavelet-assisted two-layer routing Transformer module.

[0020] Preferably, the wavelet-assisted omnidirectional state space scanning module is implemented as follows:

[0021] By applying a linear projection layer to the input feature X, the channel dimension is expanded to obtain... ,in, Indicates the length of the feature map. Indicates the width of the feature map;

[0022] Will Perform dimensional splitting operation to obtain and ;

[0023] right Perform depthwise separable convolution and padding operations, via Function activation results ;

[0024] Using a wavelet-spatial co-sampling module to sample the input Perform downsampling operation to obtain ;

[0025] Through the multi-directional scanning module Unfold the 2D image into a 1D sequence along four directions to obtain , , and ;

[0026] in, and These represent expansion along the positive horizontal direction and the positive vertical direction, respectively. and These represent expansion along the opposite horizontal direction and the opposite vertical direction, respectively.

[0027] Using linear layers Dimension mapping is performed on the four 1D sequences respectively, projecting the channel dimension C onto... We obtain four joint tensors. ;in, Indicates the dimension of time step correction. Indicates the dimension of the state parameter;

[0028] Along the channel dimension The parameters required for the Mamba model are broken down, and the time steps are calculated simultaneously.

[0029] The obtained parameters are used for selective state-space computation of the Mamba model to obtain enhanced features;

[0030] Enhance features Reconstructing the 1D sequence back into a 2D structure by reversing the original scanning order, thus restoring its spatial relationships. ;

[0031] Will With size Upsampling is performed on the Kronecker product of the all-1 matrix. The reconstructed and upsampled 2D images from four directions are fused to obtain a fused enhanced image. The fused and enhanced images are then subjected to a gating operation to obtain the final enhanced feature map output. .

[0032] Preferably, the wavelet-spatial collaborative downsampling module integrates spatial downsampling and wavelet transform-based downsampling, specifically as follows: the wavelet-spatial collaborative downsampling module includes a spatial downsampling branch and a channel downsampling branch. The spatial downsampling branch performs spatial downsampling on its input features and outputs them; the channel downsampling branch performs channel downsampling on its input features, performs Haar wavelet transform, and then concatenates the results before outputting them; the channel attention weights of the spatially downsampled features are calculated, and the outputs of the two branches are weighted and fused using these weights.

[0033] Preferably, the wavelet-spatial collaborative downsampling module is implemented as follows:

[0034] The input features are fed into a channel downsampling convolutional layer with a scaling factor of 4 to obtain an intermediate feature map. ;

[0035] The input features are fed into a spatial downsampling convolutional layer with a stride of 2 to obtain intermediate feature maps. ;

[0036] Using Haar wavelet transform to extract intermediate feature maps The image is decomposed into four sub-bands; specifically, the intermediate feature map is processed using one low-frequency filter and three high-frequency filters. Performing a convolution operation with a stride of 2 yields four sub-band images; the filter is represented as:

[0037] ;

[0038] in, Indicates a low-frequency filter; This refers to the high-frequency sub-band filters corresponding to the horizontal low-pass and vertical high-pass filters. This refers to the high-frequency sub-band filters corresponding to the horizontal high-pass and vertical low-pass filters. This represents the high-frequency subband filters corresponding to the horizontal and vertical high-pass filters;

[0039] By stitching together the four sub-band images along the channel dimension, a feature map downsampled by 2 times is obtained. ;

[0040] intermediate feature map The input is fed into the channel attention module to complete the channel attention calculation and obtain the channel attention weight vector. ;

[0041] The channel attention weights calculated by the channel attention mechanism are then subjected to channel-by-channel scaling and feature fusion operations on the outputs of the two branches to obtain... Then utilize Convolutional layers reorganize features to obtain feature maps that are downsampled by a factor of 2. And output it.

[0042] Preferably, the first gated convolutional feedforward network and the second gated convolutional feedforward network have the same construction, specifically implemented as follows:

[0043] use The convolution kernel will input features Perform a projection expansion operation along the channel to obtain ;

[0044] Will After performing padding, use The depth of the convolution kernel can be separable for feature extraction operations, performing convolution independently on each channel to obtain the desired feature. ;

[0045] Will Input dimension segmentation unit, and perform average segmentation along the channel dimension to obtain and ,right Nonlinear transformations are achieved using the GELU function activation. As a gate control unit and the result of nonlinear transformation Multiplication is performed to enhance the representational power of the features, resulting in... ;

[0046] pass Convolution kernel will Perform a projection reduction operation along the channel to obtain And output it.

[0047] On the other hand, a progressive super-resolution reconstruction device based on wavelet-space coordination includes the following:

[0048] The network construction module is used to construct a progressive reconstruction super-resolution network that includes a first convolutional layer, several wavelet-spatial co-enhancement blocks, and a second convolutional layer. After the low-resolution image is processed by the first convolutional layer to extract shallow features, the wavelet-spatial co-enhancement blocks are connected in sequence to extract deep features, and then the image is reconstructed by the second convolutional layer to generate a reconstructed image.

[0049] The wavelet-spatial co-enhancement block includes MB convolution, a wavelet-assisted two-layer routing Transformer module, a first gated convolutional feedforward network, a wavelet-assisted omnidirectional state space scanning module, and a second gated convolutional feedforward network. The input features of the wavelet-spatial co-enhancement block are processed by MB convolution for local feature extraction, by wavelet-assisted two-layer routing Transformer module for attention enhancement, by first gated convolutional feedforward network for nonlinear optimization, by wavelet-assisted omnidirectional state space scanning module for global feature enhancement, and then reconstructed by second gated convolutional feedforward network before being output.

[0050] The wavelet-assisted dual-layer routing Transformer module uses a dual-layer routing Transformer architecture for region selection and performs attention calculation based on the selected region to achieve attention enhancement. The features involved in region selection are downsampled using a wavelet-spatial collaborative downsampling module to simplify dimensions. The wavelet-assisted omnidirectional state space scanning module, after its input is downsampled by the wavelet-spatial collaborative downsampling module to simplify the input dimension and retain multi-scale information, uses a state space model for global feature enhancement. The wavelet-spatial collaborative downsampling module fuses spatial downsampling and wavelet transform-based downsampling.

[0051] The network training module is used to train the wavelet-space co-enhanced progressive reconstruction super-resolution network using low-resolution images, so as to obtain the trained wavelet-space co-enhanced progressive reconstruction super-resolution network.

[0052] The image reconstruction module is used to perform super-resolution reconstruction of the low-resolution image to be reconstructed using a trained wavelet-space co-enhanced progressive reconstruction super-resolution network, thereby obtaining the reconstructed image.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] (1) This invention utilizes a wavelet-assisted two-layer routing Transformer. Based on the two-layer routing self-attention mechanism, it uses the key and value vectors in the attention-guided wavelet-space downsampling modular simplified self-attention calculation. Without causing information loss, it effectively avoids dense self-attention calculation, enabling the model to achieve better performance with less computational overhead.

[0055] (2) This invention introduces wavelet-assisted Mamba to make up for the lack of global context information modeling in the two-layer routing Transformer, and achieves multi-scale feature extraction in Mamba by introducing a wavelet-spatial downsampling module while reducing computational complexity.

[0056] (3) In this invention, gated convolutional feedforward networks are introduced after wavelet-assisted double-layer routing Transformer and wavelet-assisted Mamba respectively to ensure the quality of the image after wavelet-assisted double-layer routing Transformer and wavelet-assisted Mamba during reconstruction, thereby realizing the feature representation capability of SR model. Attached Figure Description

[0057] The present invention will now be described in further detail with reference to the accompanying drawings;

[0058] Figure 1 This is a flowchart of a progressive super-resolution reconstruction method based on wavelet-space collaboration according to an embodiment of the present invention;

[0059] Figure 2 This is a schematic diagram of an efficient progressive super-resolution reconstruction network enhanced by wavelet-space collaboration, which is an embodiment of the progressive super-resolution reconstruction method based on wavelet-space collaboration in this invention.

[0060] Figure 3 This is a schematic diagram of the wavelet-space collaborative enhancement block for a progressive super-resolution reconstruction method based on wavelet-space collaboration according to an embodiment of the present invention.

[0061] Figure 4 This is a schematic diagram of the wavelet-assisted two-layer routing Transformer module of the progressive super-resolution reconstruction method based on wavelet-space collaboration in an embodiment of the present invention.

[0062] Figure 5 This is a schematic diagram of the wavelet-space collaborative downsampling module of the progressive super-resolution reconstruction method based on wavelet-space collaboration in an embodiment of the present invention;

[0063] Figure 6This is a schematic diagram of a gated convolutional feedforward network for a progressive super-resolution reconstruction method based on wavelet-space collaboration, according to an embodiment of the present invention.

[0064] Figure 7 This is a schematic diagram of the wavelet-assisted omnidirectional state space scanning module of the progressive super-resolution reconstruction method based on wavelet-space collaboration in an embodiment of the present invention.

[0065] Figure 8 This is a structural block diagram of a wavelet-space collaborative progressive super-resolution reconstruction device according to an embodiment of the present invention. Detailed Implementation

[0066] The present invention will be further described below through specific embodiments.

[0067] like Figure 1 As shown, the gradual super-resolution reconstruction method based on wavelet-space synergy has the following specific steps:

[0068] The purpose of this invention is to overcome the high computational complexity of existing two-layer routing Transformer methods, while also addressing the issue of local context modeling in two-layer routing Transformers. It provides an efficient progressive reconstruction super-resolution network with wavelet-space co-enhancement. This method utilizes a wavelet-assisted two-layer routing Transformer, building upon the two-layer routing self-attention mechanism. Through attention-guided wavelet-space downsampling, modularized simplified self-attention computation of key and value vectors, it effectively avoids dense self-attention computation without information loss, enabling the model to achieve better performance with less computational overhead. Furthermore, since the two-layer routing Transformer relies more on local context information for feature enhancement in simplifying self-attention computation, wavelet-assisted Mamba is introduced to compensate for the lack of global context modeling. Traditional Mamba is based on single-scale feature extraction; by introducing a wavelet-space downsampling module, multi-scale feature extraction is achieved while reducing computational complexity. To ensure the quality of the image after wavelet-assisted dual-layer routing Transformer and wavelet-assisted Mamba during reconstruction, gated convolutional feedforward networks are introduced after wavelet-assisted dual-layer routing Transformer and wavelet-assisted Mamba respectively to improve the reconstruction quality and thus realize the feature representation capability of the SR model.

[0069] S1, construct a progressive reconstruction super-resolution network with wavelet-space co-enhancement.

[0070] Figure 2 This diagram illustrates an efficient progressive reconstruction super-resolution network for wavelet-space co-enhancement. The computational process is as follows:

[0071] Given a low-resolution image First, shallow features are extracted using a convolutional layer with a 1×1 kernel. Its expression is as follows:

[0072] ;

[0073] Secondly, deep features are extracted using N-layer wavelet-spatial co-enhancement blocks. Its expression is as follows:

[0074] ;

[0075] ;

[0076] in, The function representing the i-th wavelet-space co-enhancement block.

[0077] Finally, a convolutional layer with a 1×1 kernel is used to... Reconstruction is performed to generate a high-resolution image. .

[0078] The wavelet-spatial collaborative enhancement block includes a wavelet-assisted two-layer routing Transformer module, a gated convolutional feedforward network, and a wavelet-assisted omnidirectional state space scanning module. Figure 3 This is a schematic diagram of the wavelet-spatial co-enhancement block. The wavelet-spatial co-enhancement block consists of five parts: MB convolution, a wavelet-assisted two-layer routing Transformer module, a gated convolutional feedforward network, a wavelet-assisted omnidirectional state space scanning module, and a gated convolutional feedforward network. Specifically, given an input feature map... The expression for the wavelet-space co-enhancement block is as follows:

[0079] ;

[0080] in, This represents a gated convolutional feedforward network. This represents a wavelet-assisted two-layer routing Transformer module. Represented as MB convolution, This represents a wavelet-assisted omnidirectional state space scanning module. This represents the intermediate feature map output by the wavelet-assisted two-layer routing Transformer module. The output represents the wavelet-space co-enhancement block.

[0081] The wavelet-assisted two-layer routing Transformer module is constructed based on the attention-guided wavelet-spatial co-sampling module; Figure 4This is a schematic diagram of a wavelet-assisted two-layer routing Transformer module. Its calculation process is as follows:

[0082] A wavelet-assisted two-layer routing Transformer module is constructed based on an attention-guided wavelet-spatial co-sampling module. The two-layer routing self-attention module is divided into two parts: one route is used to select relevant regions to complete the routing decision operation, and the other route is used to perform self-attention calculation on the selected regions to achieve feature enhancement operation.

[0083] Given an input feature map Where C, H, and W represent the number of channels, length, and width of the feature map, respectively. The feature map is pre-partitioned into a window and converted into an input tensor. Where C, x and y, w1 and w2 represent the number of channels in the feature map, the length and width of the window after partitioning (coarse-grained window), and the length and width of the pixels within the partitioned window (fine-grained window), respectively. The tensor... As input to the wavelet-assisted two-layer routing Transformer module.

[0084] First, perform a dimension rearrangement operation on the input tensor to obtain... and Using wavelet-spatial co-sampling module to perform Perform downsampling and projection operations to obtain Then The input is fed into a linear convolutional layer with a spread factor of 2, and... The channel was expanded by 2 times. Its expression is as follows:

[0085] ;

[0086] in, This indicates that wavelet-space co-sampling operation is used. This represents a channel upsampling convolution with a spread factor of 2, which will be discussed later. Let H1 represent it. Let W1 represent it.

[0087] Then, for Perform window partitioning operations, Divide into N non-overlapping windows of size S ,in ; The same partitioning operation is used to obtain ,in Then, using a linear layer... Will Convert to query vector Its expression is as follows:

[0088] ;

[0089] Next, the key-value pairs are split to obtain the key vectors. Average pooling is used to transform the query vector Q and key vector K into an average query vector representing each partitioned small window. with average bond vector Its expression is as follows:

[0090] ;

[0091] Then, similarity calculation is performed to generate a correlation map between windows. The top k windows with high correlation values ​​are selected from the correlation map for subsequent feature enhancement, thus implementing the routing decision operation. The expression is as follows:

[0092] ;

[0093] in, This indicates the transpose operation. This indicates the operation of retrieving the first N windows.

[0094] Next, the top k windows with high relevance values ​​selected after the routing decision are chosen from the Map graph and concatenated to obtain the desired result. Then Split to obtain the key vector AND value vector Then, fine-grained self-attention computation is performed to obtain enhanced features. The expression is as follows:

[0095] ;

[0096] in, This indicates that the splicing operation has been selected. The function representing Softmax normalization.

[0097] Finally, the shape of Y is N×S 2 ×C was reconverted to And it serves as the output of the wavelet-assisted two-layer routing Transformer module.

[0098] Figure 5 This is a schematic diagram of the attention-guided wavelet-space co-sampling module. Its calculation process is as follows:

[0099] Given an input feature map Where C, H, and W represent the number of channels, length, and width of the feature map, respectively, the overall calculation process of the attention-guided wavelet-spatial co-sampling module is as follows:

[0100] First, the input X is subjected to parallel dual-path convolution operations. Path one: X is input into a channel downsampling convolutional layer with a scaling factor of 4, thereby reducing the number of channels of X by a factor of 4 to obtain an intermediate feature map. Branch 2: Input X into a spatial downsampling convolutional layer with a stride of 2, thereby reducing the spatial size of the input X by a factor of 4 and obtaining an intermediate feature map. Its expression is as follows:

[0101] ;

[0102] in, This represents a channel downsampling convolution with a scaling factor of 4, and its kernel size is... ; This represents a spatial downsampling convolution with a scaling factor of 4, and its kernel size is... .

[0103] Next, the Haar wavelet transform is used to... Decomposed into 4 sub-band images , , and .in, , , and Using low-frequency filters and three high-frequency filters , , In These four filters are obtained by performing a convolution operation with a stride of 2, and are defined as follows:

[0104] ;

[0105] The calculation process of the Haar wavelet transform can be expressed as follows:

[0106] ;

[0107] in, This represents the convolution operation. This indicates spatial downsampling with a scaling factor of 2.

[0108] Next, the four sub-band images are processed along the channel dimension. By concatenating the data, the dimensionality is reshaped after the Haar wavelet transform, resulting in a feature map that is downsampled by a factor of 2. Its expression is as follows:

[0109] ;

[0110] in, This indicates a splicing operation along the channel dimension.

[0111] Then The input is fed into the channel attention module to complete the channel attention calculation and obtain the channel attention weight vector. Its expression is as follows:

[0112] ;

[0113] in, This indicates a global average pooling operation. This represents the global max pooling operation. and The representatives consist of two Bottleneck structure built from convolution , Indicates use Function activation operation, Indicates use Function activation operation.

[0114] Finally, the channel attention weights calculated by the channel attention mechanism are applied to the outputs of the two branches through channel-by-channel scaling and feature fusion operations. Then utilize Convolutional layers reorganize features to obtain feature maps that are downsampled by a factor of 2. Its expression is as follows:

[0115] ;

[0116] in, This represents element-wise multiplication. express Convolutional layer.

[0117] Figure 6 This is a schematic diagram of a gated convolutional feedforward network. Its calculation process is as follows:

[0118] Given an input feature map Where C, H, and W represent the number of channels, length, and width of the feature map, respectively.

[0119] First, using Convolution kernel will Perform a projection expansion operation along the channel to obtain Its expression is:

[0120] ;

[0121] in, This indicates that the input is projected and extended along the channel.

[0122] Then, After performing padding, use The depth of the convolution kernel can be separable for feature extraction operations, performing convolution independently on each channel to obtain the desired feature. Its expression is:

[0123] ;

[0124] in, This indicates that a depthwise separable convolution operation is performed on the input.

[0125] Immediately afterwards, In the input dimension segmentation unit, average segmentation is performed along the channel dimension to obtain... and ,right Nonlinear transformations are achieved using the GELU function activation. As a gate control unit and Multiplication is performed to enhance the representational power of the features, resulting in... Its expression is:

[0126] ;

[0127] in, Indicates use Function activation operation.

[0128] Finally, through Convolution kernel will Perform a projection reduction operation along the channel to obtain Its expression is:

[0129] ;

[0130] The omnidirectional state space scanning module is constructed based on the wavelet-space co-sampling module. Figure 7 This is a schematic diagram of a wavelet-assisted omnidirectional state space scanning module. Its calculation process is as follows:

[0131] Given an input feature map Where C, H, and W represent the number of channels, length, and width of the feature map, respectively.

[0132] First, the input X is subjected to a linear projection layer to expand the channel dimension. Its expression is:

[0133] ;

[0134] in, This indicates that a linear layer projection operation is performed on the input.

[0135] Then, the obtained Perform dimensional splitting operation to obtain and Its expression is:

[0136] ;

[0137] in, This indicates that a dimension splitting operation is performed on the input.

[0138] Again Perform depthwise separable convolution and padding operations to ensure that the output after convolution maintains the same dimensions as the input. Finally, pass through... Function activation results Its expression is as follows:

[0139] ;

[0140] in, express Function activation operation.

[0141] Next, the wavelet-spatial co-sampling module is used to sample the input. Perform downsampling operation to obtain Its expression is as follows:

[0142] ;

[0143] Through the multi-directional scanning module Unfold the 2D image into a 1D sequence along four directions to obtain , , and ,in, and It unfolds along the positive horizontal direction and the positive vertical direction. and It unfolds in the opposite direction of the horizontal and the opposite direction of the vertical.

[0144] Reuse linear layers Dimension mapping is performed on the four 1D sequences respectively, projecting the channel dimension C onto... (in Correcting the dimension for the time step. (For the state parameter dimension), we obtain four joint tensors. Its expression is:

[0145] ;

[0146] in, This indicates that a dimension mapping operation is performed on a 1D sequence.

[0147] Then, along the channel dimension Parameters required to split into Mamba B, C, and simultaneously calculate the time step. ,in , , (Use L to represent the length of the 1D sequence) ).

[0148] Next, the obtained parameters are used for selective state-space computation using Mamba to obtain enhanced features. Its expression is as follows:

[0149] ;

[0150] in, This represents the hidden state, an intermediate quantity, in the selective state-space computation process. This represents the hidden state of the previous time step. This represents the input at the current moment.

[0151] Finally, the enhanced features will be... Reconstructing the 1D sequence back into a 2D structure by reversing the original scanning order, thus restoring its spatial relationships. .Will With size Upsampling is performed on the Kronecker product of the all-1 matrix. The reconstructed and upsampled 2D images from four directions are fused to obtain a fused enhanced image. The fused and enhanced images are then subjected to a gating operation to obtain the final enhanced feature map output. Its expression is as follows:

[0152] ;

[0153] Where K represents the size A matrix of all 1s This indicates a fusion operation.

[0154] S2, use low-resolution images to train the wavelet-space co-enhanced progressive reconstruction super-resolution network to obtain the trained wavelet-space co-enhanced progressive reconstruction super-resolution network;

[0155] S3 uses a trained wavelet-spatial co-enhanced progressive reconstruction super-resolution network to perform super-resolution reconstruction of the low-resolution image, resulting in a reconstructed image.

[0156] like Figure 8 As shown, the present invention also discloses a progressive super-resolution reconstruction device based on wavelet-space coordination, comprising:

[0157] The network construction module 801 is used to construct a progressive reconstruction super-resolution network including a first convolutional layer, several wavelet-spatial co-enhancement blocks, and a second convolutional layer. After the low-resolution image is processed by the first convolutional layer to extract shallow features, the wavelet-spatial co-enhancement blocks are connected in sequence to extract deep features, and then the image is reconstructed by the second convolutional layer to generate a reconstructed image.

[0158] The wavelet-spatial co-enhancement block includes MB convolution, a wavelet-assisted two-layer routing Transformer module, a first gated convolutional feedforward network, a wavelet-assisted omnidirectional state space scanning module, and a second gated convolutional feedforward network. The input features of the wavelet-spatial co-enhancement block are processed by MB convolution for local feature extraction, by wavelet-assisted two-layer routing Transformer module for attention enhancement, by first gated convolutional feedforward network for nonlinear optimization, by wavelet-assisted omnidirectional state space scanning module for global feature enhancement, and then reconstructed by second gated convolutional feedforward network before being output.

[0159] The wavelet-assisted dual-layer routing Transformer module uses a dual-layer routing Transformer architecture for region selection and performs attention calculation based on the selected region to achieve attention enhancement. The features involved in region selection are downsampled using a wavelet-spatial collaborative downsampling module to simplify dimensions. The wavelet-assisted omnidirectional state space scanning module, after its input is downsampled by the wavelet-spatial collaborative downsampling module to simplify the input dimension and retain multi-scale information, uses a state space model for global feature enhancement. The wavelet-spatial collaborative downsampling module fuses spatial downsampling and wavelet transform-based downsampling.

[0160] The network training module 802 is used to train the wavelet-space co-enhanced progressive reconstruction super-resolution network using low-resolution images to obtain the trained wavelet-space co-enhanced progressive reconstruction super-resolution network.

[0161] Image reconstruction module 803 is used to perform super-resolution reconstruction of the low-resolution image to be reconstructed using a trained wavelet-spatial co-enhanced progressive reconstruction super-resolution network, thereby obtaining the reconstructed image.

[0162] The specific implementation of the wavelet-spatial collaborative progressive super-resolution reconstruction device is the same as that of the wavelet-spatial collaborative progressive super-resolution reconstruction method, and will not be described again in this embodiment.

[0163] The above are merely specific embodiments of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial modifications made to the present invention using this concept shall be considered as infringing upon the protection scope of the present invention.

Claims

1. A progressive super-resolution reconstruction method based on wavelet-space coordination, characterized in that, Includes the following steps: S1, construct a progressive reconstruction super-resolution network including a first convolutional layer, several wavelet-spatial co-enhancement blocks, and a second convolutional layer; after the low-resolution image is processed by the first convolutional layer to extract shallow features, the wavelet-spatial co-enhancement blocks are connected in sequence to extract deep features, and then the image is reconstructed by the second convolutional layer to generate a reconstructed image. The wavelet-spatial co-enhancement block includes MB convolution, a wavelet-assisted two-layer routing Transformer module, a first gated convolutional feedforward network, a wavelet-assisted omnidirectional state space scanning module, and a second gated convolutional feedforward network. The input features of the wavelet-spatial co-enhancement block are processed by MB convolution for local feature extraction, by wavelet-assisted two-layer routing Transformer module for attention enhancement, by first gated convolutional feedforward network for nonlinear optimization, by wavelet-assisted omnidirectional state space scanning module for global feature enhancement, and then reconstructed by second gated convolutional feedforward network before being output. The wavelet-assisted dual-layer routing Transformer module uses a dual-layer routing Transformer architecture for region selection and performs attention calculation based on the selected region to achieve attention enhancement. The features involved in region selection are downsampled using a wavelet-spatial collaborative downsampling module to simplify dimensions. The wavelet-assisted omnidirectional state space scanning module, after its input is downsampled by the wavelet-spatial collaborative downsampling module to simplify the input dimension and retain multi-scale information, uses a state space model for global feature enhancement. The wavelet-spatial collaborative downsampling module fuses spatial downsampling and wavelet transform-based downsampling. S2, use low-resolution images to train the wavelet-space co-enhanced progressive reconstruction super-resolution network to obtain the trained wavelet-space co-enhanced progressive reconstruction super-resolution network; S3 uses a trained wavelet-spatial co-enhanced progressive reconstruction super-resolution network to perform super-resolution reconstruction on the low-resolution image to be reconstructed, and obtains the reconstructed image.

2. The progressive super-resolution reconstruction method based on wavelet-space coordination according to claim 1, characterized in that, The wavelet-assisted two-layer routing Transformer module is implemented as follows: The input feature map is pre-partitioned into a window and converted into an input tensor. Where C represents the number of channels in the feature map; x and y represent the length and width of the window after partitioning, respectively; w1 and w2 represent the length and width of the pixels within the partitioned window, respectively. The input tensor is rearranged in dimension to obtain... and Using wavelet-spatial co-sampling module to perform Perform downsampling and projection operations to obtain Then The input is fed into a linear convolutional layer with a spread factor of 2, and... The channel was expanded by 2 times. ; right Perform window partitioning operations, Divide into N non-overlapping windows of size S ,in, , express Length; express The width; The same partitioning operation is used to obtain ,in, Indicates to The number of non-overlapping windows after window division. Then, using linear layers Will Convert to query vector ; The key vector is obtained by splitting the key-value pair. Average pooling is used to transform the query vector Q and key vector K into an average query vector representing each partitioned small window. with average bond vector ; Perform similarity calculation, generate a correlation map between windows, and select the top k windows with high correlation values ​​from the correlation map; The top k windows with high correlation values ​​are selected from the Map and concatenated to obtain the result. Then Split to obtain the key vector AND value vector Perform fine-grained self-attention computation to obtain enhanced features. ; The shape of the enhanced feature Y is changed from N×S 2 ×C is converted back to The output of the wavelet-assisted two-layer routing Transformer module.

3. The progressive super-resolution reconstruction method based on wavelet-space coordination according to claim 1, characterized in that, The wavelet-assisted omnidirectional state space scanning module is implemented as follows: By applying a linear projection layer to the input feature X, the channel dimension is expanded to obtain... ,in, Indicates the length of the feature map. Indicates the width of the feature map; Will Perform dimensional splitting operation to obtain and ; right Perform depthwise separable convolution and padding operations, via Function activation results ; Using a wavelet-spatial co-sampling module to sample the input Perform downsampling operation to obtain ; Through the multi-directional scanning module Unfold the 2D image into a 1D sequence along four directions to obtain , , and ; in, and These represent expansion along the positive horizontal direction and the positive vertical direction, respectively. and These represent expansion along the opposite horizontal direction and the opposite vertical direction, respectively. Using linear layers Dimension mapping is performed on the four 1D sequences respectively, projecting the channel dimension C onto... We obtain four joint tensors. ;in, Indicates the dimension of time step correction. Indicates the dimension of the state parameter; Along the channel dimension The parameters required for the Mamba model are broken down, and the time steps are calculated simultaneously. The obtained parameters are used for selective state-space computation of the Mamba model to obtain enhanced features; Enhance features Reconstructing the 1D sequence back into a 2D structure by reversing the original scanning order, thus restoring its spatial relationships. ; Will With size Upsampling is performed on the Kronecker product of the all-1 matrix. The reconstructed and upsampled 2D images from four directions are fused to obtain a fused enhanced image. The fused and enhanced images are then subjected to a gating operation to obtain the final enhanced feature map output. .

4. The progressive super-resolution reconstruction method based on wavelet-space coordination according to claim 1, characterized in that, The wavelet-spatial collaborative downsampling module integrates spatial downsampling and wavelet transform-based downsampling, specifically as follows: The wavelet-spatial collaborative downsampling module includes a spatial downsampling branch and a channel downsampling branch. The spatial downsampling branch performs spatial downsampling on its input features and outputs the results. The channel downsampling branch performs channel downsampling on its input features, performs Haar wavelet transform, and then concatenates the results before outputting the results. The channel attention weights of the spatially downsampled features are calculated, and the outputs of the two branches are weighted and fused using these weights.

5. The progressive super-resolution reconstruction method based on wavelet-space coordination according to claim 4, characterized in that, The wavelet-space coordinated downsampling module is implemented as follows: The input features are fed into a channel downsampling convolutional layer with a scaling factor of 4 to obtain an intermediate feature map. ; The input features are fed into a spatial downsampling convolutional layer with a stride of 2 to obtain intermediate feature maps. ; Using Haar wavelet transform to extract intermediate feature maps The image is decomposed into four sub-bands; specifically, the intermediate feature map is processed using one low-frequency filter and three high-frequency filters. Performing a convolution operation with a stride of 2 yields four sub-band images; the filter is represented as: ； in, Indicates a low-frequency filter; This refers to the high-frequency sub-band filters corresponding to the horizontal low-pass and vertical high-pass filters. This refers to the high-frequency sub-band filters corresponding to the horizontal high-pass and vertical low-pass filters. This represents the high-frequency subband filters corresponding to the horizontal and vertical high-pass filters; By stitching together the four sub-band images along the channel dimension, a feature map downsampled by 2 times is obtained. ; intermediate feature map The input is fed into the channel attention module to complete the channel attention calculation and obtain the channel attention weight vector. ; The channel attention weights calculated by the channel attention mechanism are then subjected to channel-by-channel scaling and feature fusion operations on the outputs of the two branches to obtain... Then utilize Convolutional layers reorganize features to obtain feature maps that are downsampled by a factor of 2. And output it.

6. The progressive super-resolution reconstruction method based on wavelet-space coordination according to claim 1, characterized in that, The first gated convolutional feedforward network and the second gated convolutional feedforward network have the same construction, and are implemented as follows: use The convolution kernel will input features Perform a projection expansion operation along the channel to obtain ; Will After performing padding, use The depth of the convolution kernel can be separable for feature extraction operations, performing convolution independently on each channel to obtain the desired feature. ; Will Input dimension segmentation unit, and perform average segmentation along the channel dimension to obtain and ,right Nonlinear transformations are achieved using the GELU function activation. As a gate control unit and the result of nonlinear transformation Multiplication is performed to enhance the representational power of the features, resulting in... ; pass Convolution kernel will Perform a projection reduction operation along the channel to obtain And output it.

7. A progressive super-resolution reconstruction device based on wavelet-space coordination, characterized in that, Including the following: The network construction module is used to construct a progressive reconstruction super-resolution network that includes a first convolutional layer, several wavelet-spatial co-enhancement blocks, and a second convolutional layer. After the low-resolution image is processed by the first convolutional layer to extract shallow features, the wavelet-spatial co-enhancement blocks are connected in sequence to extract deep features, and then the image is reconstructed by the second convolutional layer to generate a reconstructed image. The wavelet-spatial co-enhancement block includes MB convolution, a wavelet-assisted two-layer routing Transformer module, a first gated convolutional feedforward network, a wavelet-assisted omnidirectional state space scanning module, and a second gated convolutional feedforward network. The input features of the wavelet-spatial co-enhancement block are processed by MB convolution for local feature extraction, by wavelet-assisted two-layer routing Transformer module for attention enhancement, by first gated convolutional feedforward network for nonlinear optimization, by wavelet-assisted omnidirectional state space scanning module for global feature enhancement, and then reconstructed by second gated convolutional feedforward network before being output. The wavelet-assisted dual-layer routing Transformer module uses a dual-layer routing Transformer architecture for region selection and performs attention calculation based on the selected region to achieve attention enhancement. The features involved in region selection are downsampled using a wavelet-spatial collaborative downsampling module to simplify dimensions. The wavelet-assisted omnidirectional state space scanning module, after its input is downsampled by the wavelet-spatial collaborative downsampling module to simplify the input dimension and retain multi-scale information, uses a state space model for global feature enhancement. The wavelet-spatial collaborative downsampling module fuses spatial downsampling and wavelet transform-based downsampling. The network training module is used to train the wavelet-space co-enhanced progressive reconstruction super-resolution network using low-resolution images, so as to obtain the trained wavelet-space co-enhanced progressive reconstruction super-resolution network. The image reconstruction module is used to perform super-resolution reconstruction of the low-resolution image to be reconstructed using a trained wavelet-space co-enhanced progressive reconstruction super-resolution network, thereby obtaining the reconstructed image.

Images