Lightweight image super-resolution reconstruction method based on frequency domain-spatial domain assisted Mama

Abstract

The invention relates to the technical field of image processing, and discloses a lightweight image super-resolution reconstruction method based on frequency domain-spatial domain assisted Mama, which comprises the following steps: acquiring an original image data set and carrying out data preprocessing on the original image data set to acquire a training data set and a test data set; introducing a frequency domain-spatial domain Mama layer and an interaction layer based on transpose and self-attention features, and constructing a lightweight image super-resolution reconstruction model based on frequency domain-spatial domain assisted Mama; inputting the training data set into a lightweight image super-resolution reconstruction model based on frequency domain-spatial domain assisted Mama for training to obtain a trained lightweight image super-resolution reconstruction model with weight based on frequency domain-spatial domain assisted Mama for reconstructing a low-resolution image into a high-resolution image; inputting the test data set into a trained lightweight image super-resolution reconstruction model with weight and based on frequency domain-space domain assisted Mama to obtain a reconstructed high-resolution visual image; according to the method, lightweight image super-resolution is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a lightweight image super-resolution reconstruction method based on frequency domain-spatial domain assisted Mamba. Background Art

[0002] In the context of the digital age, image quality has become one of the key indicators for measuring technological progress and user experience. With the continuous iteration of technology and the increasing needs of users, high-resolution images have become the standard configuration and general expectation in many fields. With its excellent resolution, rich details and better visual experience, it brings users a more vivid and realistic visual enjoyment. However, there are still a large number of low-resolution images in real life, which makes image super-resolution technology of vital importance for improving image resolution.

[0003] Traditionally, methods to improve image super-resolution have focused on physical improvements, such as improving the quality and pixel density of camera sensors. Although this approach can improve image resolution and quality, it also comes with significant disadvantages and limitations. Specifically, physical improvements often require expensive hardware upgrades and are not applicable to all scenarios. Therefore, due to technical and cost factors, it may not be practical to fully upgrade the physical level of the device.

[0004] In recent years, with the rapid development of deep learning technology, a series of image super-resolution methods based on deep learning have emerged, including methods based on convolutional neural networks (CNNs) and methods based on Transformers. However, these methods are faced with challenges of different problems. First, the computational complexity of existing models based on convolutional neural networks increases significantly with the increasing depth of the model. At the same time, the fixed size of the convolution kernel limits the receptive field of the model, making it difficult to efficiently capture global context information. Secondly, compared with the convolutional neural network model, the traditional Transformer-based method can better capture context information, but its computational complexity is proportional to the square of the input token length, resulting in higher computational overhead and inference time. Recently, although the Mamba-based image super-resolution model has achieved linear computational complexity, it has insufficient consideration in capturing local features. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a lightweight image super-resolution reconstruction method based on frequency domain-spatial domain assisted Mamba, which is used to solve the technical problems of large number of parameters and high computational complexity of the current image super-resolution model.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0007] The lightweight image super-resolution reconstruction method based on frequency domain-spatial domain assisted Mamba includes the following steps:

[0008] S1. Obtain the original image data set and perform data preprocessing on it to obtain the training data set and the test data set;

[0009] S2, introduce the frequency-spatial domain Mamba layer and the transposition and self-attention feature interaction layer to build a lightweight image super-resolution reconstruction model based on the frequency-spatial domain assisted Mamba;

[0010] S3, inputting the training data set into the lightweight image super-resolution reconstruction model based on the frequency domain-spatial domain assisted Mamba for training, and obtaining a trained lightweight image super-resolution reconstruction model based on the frequency domain-spatial domain assisted Mamba with weights, which is used to reconstruct the low-resolution image into a high-resolution image;

[0011] S4. Input the test data set into the trained lightweight image super-resolution reconstruction model based on frequency domain-spatial domain assisted Mamba with weights to obtain a reconstructed high-resolution visualization image.

[0012] Furthermore, step S1 specifically includes:

[0013] S11, obtaining a public DIV2K dataset, using the DIV2K dataset as a HR image dataset, and downsampling each image in the HR image dataset by 2, 3, and 4 times using a bicubic interpolation method to obtain images with multiple downsampling multiples;

[0014] S12, performing random flipping and rotation operations on the images with multiple downsampling times to obtain enhanced low-resolution LR images;

[0015] S13, using the HR image data set as the first label data, using the enhanced low-resolution LR image as the first original data, and using the first label data and the first original data as the training data set;

[0016] S14, obtain the public Set5 dataset, Set14 dataset, B100 dataset, Urban100 dataset and Manga109 dataset and use them as HR images of the test set, and use the bicubic interpolation method to downsample each dataset by 2, 3 and 4 times respectively to obtain the LR images of the test set;

[0017] S15. Use the HR image of the test set as the second label data, use the LR image of the test set as the second original data, and use the second label data and the second original data as the test data set.

[0018] Furthermore, the lightweight image super-resolution reconstruction model based on frequency-space domain assisted Mamba in step S2 includes a shallow feature extraction module, a deep feature extraction module, and a high-resolution image reconstruction module;

[0019] The deep feature extraction module includes M feature mixing group modules and the second 3×3 convolutional layer;

[0020] The feature mixing group module includes N multi-level feature interaction layers and the first 3×3 convolutional layer;

[0021] The multi-level feature interaction layer includes the first LayerNorm layer, the frequency-spatial domain Mamba layer, the second LayerNorm layer, and the transposition and self-attention based feature interaction layer;

[0022] The frequency-spatial domain Mamba layer includes a visual Mamba layer, a frequency-spatial domain enhancement layer, and the first 1×1 convolutional layer;

[0023] The visual Mamba layer includes the upper branch module, the lower branch module, and the third Linear layer;

[0024] The upper branch module includes the first Linear layer, the first 3×3 depth convolution layer, the first SiLU layer, the 2D selective scanning module, and the third LayerNorm layer;

[0025] The lower branch module includes the second Linear layer and the second SiLU layer;

[0026] The frequency domain-spatial domain enhancement layer includes a frequency domain enhancement module and a spatial domain enhancement module;

[0027] The frequency domain enhancement module includes a second 3×3 depth convolution layer, a Fourier transform layer, a second 1×1 convolution layer, a first ReLU layer, and an inverse Fourier transform layer;

[0028] The spatial domain enhancement module includes a third 3×3 depth convolution layer and a third 1×1 convolution layer;

[0029] The feature interaction layer based on transposition and self-attention includes the third 3×3 convolution layer, spatial channel permutation layer, spatial self-attention layer, channel self-attention layer, and the fifth 1×1 convolution layer;

[0030] The spatial channel permutation layer includes a first permutation layer, a fifth 3×3 depth convolution layer, a second permutation layer, and a sixth 3×3 depth convolution layer;

[0031] The channel self-attention layer includes an adaptive average pooling layer, a fourth 3×3 depth convolution layer, and a second sigmoid layer;

[0032] The spatial self-attention layer includes the second ReLU layer, the fourth 1×1 convolutional layer, and the first sigmoid layer.

[0033] Furthermore, step S3 specifically includes:

[0034] S31, input the training data set into the shallow feature extraction module of the lightweight image super-resolution reconstruction model based on the frequency domain-spatial domain assisted Mamba, and perform convolution processing using a 3×3 convolution layer to obtain the shallow feature F SH ;

[0035] S32, the shallow feature F SH Input the deep feature extraction module to extract mixed features and obtain deep features

[0036] S33, deep features With shallow features F SH After adding, the residual connection is realized, and then input into the high-resolution image reconstruction module to realize image reconstruction by using convolution layer and PixelShuffle upsampling to obtain the reconstructed feature I SR , and the L1 loss function is used to optimize the lightweight image super-resolution reconstruction model based on frequency domain-spatial domain assisted Mamba in training, and a trained lightweight image super-resolution reconstruction model based on frequency domain-spatial domain assisted Mamba with weights is obtained.

[0037] Further, step S32 specifically includes:

[0038] S321, the shallow feature F SH Input the first LayerNorm layer for normalization operation to obtain the first normalized feature X, and evenly divide the first normalized feature X from the channel dimension to obtain feature X1 and feature X2, that is:

[0039] X1,X2=Split(X,dim=C)

[0040] Among them, Split represents the feature segmentation function, dim represents the segmentation dimension, and C represents the number of channels;

[0041] S322, feature X1 and feature X2 are respectively input into the visual Mamba layer of the frequency domain-spatial domain Mamba layer and the frequency domain-spatial domain enhancement layer for visual Mamba and frequency domain-spatial domain enhancement processing, and then spliced ​​in the channel dimension, and input into the first 1×1 convolutional layer for preliminary feature interaction to obtain preliminary global-local feature F g_l , including:

[0042] S3221, input feature X1 into the visual Mamba layer for visual Mamba processing to capture context information and obtain the global feature F m ;

[0043] S3222, input feature X2 into the frequency domain-spatial domain enhancement layer for frequency domain-spatial domain enhancement processing to enhance context information and local information, and obtain frequency domain-spatial domain enhanced feature F f_s ;

[0044] S3223, the global feature F m and frequency-spatial domain enhancement feature F f_s The concatenation is performed in the channel dimension and input into the first 1×1 convolutional layer for preliminary feature interaction to obtain the preliminary global-local feature F g_l ;

[0045] S323, the preliminary global-local feature F g_l With shallow features F SH Add together to realize local residual connection and obtain the final global-local feature F G_L ;

[0046] S324, the final global-local feature F G_L Input the second LayerNorm layer for normalization operation to obtain the second normalized feature

[0047] S325, the second normalized feature After the input is subjected to feature interaction fusion based on the transposition and self-attention feature interaction layers, it is combined with the final global-local feature F G_L Add, realize local residual connection, and obtain interactive features

[0048] S326, interactive features After repeating steps S321-S325 N times, the first 3×3 convolutional layer is input to obtain the mixed features.

[0049] S327, Mix the features After repeating step S326 M times, the second 3×3 convolutional layer is input to obtain the deep features.

[0050] Furthermore, step S3221 specifically includes:

[0051] S32211. Input feature X1 into the first Linear layer for linear transformation operation to obtain the upper branch transformation feature F u_l , and input into the first 3×3 depth convolutional layer for encoding operation to obtain the upper branch encoding feature F u_c , and input into the first SiLU layer for activation operation to obtain the upper branch activation feature F u_s, and input into the 2D selective scanning module for global feature extraction, and input into the third LayerNorm layer for normalization operation to obtain the first feature F u ;

[0052] S32212, input feature X1 into the second Linear layer for linear transformation operation to obtain the lower branch transformation feature F d_l , and input into the second SiLU layer for activation operation to obtain the second feature F d ;

[0053] S32213, the first feature F u With the second feature F d Multiply and input into the third Linear layer for linear activation to obtain the global feature F m .

[0054] Further, step S3222 specifically includes:

[0055] S32221. Input feature X2 into the second 3×3 deep convolution layer for deep convolution operation and extract local features to obtain the first local feature of feature X2, and input it into the Fourier transform layer for Fourier transform to convert the first local feature of feature X2 from the spatial domain to the frequency domain, and input it into the second 1×1 convolution layer for convolution operation and then into the first ReLU layer for activation operation to obtain activated frequency domain features, and input it into the inverse Fourier transform layer for inverse transformation to convert the activated frequency domain features from the frequency domain to the spatial domain to obtain the frequency domain enhanced feature F. f ;

[0056] S32222, input feature X2 into the third 3×3 deep convolution layer for deep convolution operation to extract local features, obtain the second local feature of feature X2, and input it into the third 1×1 convolution layer for convolution operation to obtain spatial domain enhanced feature F s ;

[0057] S32223, the frequency domain enhancement feature F f And the spatial domain enhanced feature F s Add together to get the frequency domain-spatial domain enhanced feature F f_s .

[0058] Further, step S325 specifically includes:

[0059] S3251, the second normalized feature Input the third 3×3 convolutional layer for preliminary feature fusion and channel compression, compress it to C / 4, and obtain the initialization feature F i ;

[0060] S3252, initialize feature F iThe first permutation layer of the input spatial channel permutation layer performs a permutation operation by initializing the feature F i The spatial features of are replaced by channel dimension features to obtain the first replacement feature F i_p1 , and input the fifth 3×3 deep convolutional layer to extract the fineness feature, and input the second permutation layer to perform a permutation operation, converting the fineness feature from the channel dimension to a spatial feature to obtain the second permutation feature F i_p2 , and input into the sixth 3×3 deep convolutional layer for spatial feature encoding to obtain the permutation fusion feature F i_p ;

[0061] S3253, initialize feature F i The input is fed into the second ReLU layer of the spatial self-attention layer for nonlinear activation, and then fed into the fourth 1×1 convolutional layer to reduce the channel dimension to 1, and then fed into the first sigmoid layer for activation to obtain the first activation feature F i_s1 , the first activated feature F i_s1 With the initialization feature F i Multiply them together to get the spatial self-attention fusion feature F i_s ;

[0062] S3254, initialize feature F i The adaptive average pooling layer of the input channel self-attention layer performs a pooling operation to obtain an initialization feature with a spatial size of 1×1, and enters the fourth 3×3 deep convolutional layer for encoding, and enters the second sigmoid layer for nonlinear activation operation to obtain the second activation feature F i_s2 , the second activation feature F i_s2 With the initialization feature F i Multiply them together to get the channel self-attention fusion feature F i_c ;

[0063] S3255, replace the fusion feature F i_p , spatial self-attention fusion feature F i_s And the channel self-attention fusion feature F i_c Add, and input the fifth 1×1 convolutional layer to expand the number of channels to C, and then add it to the final global-local feature F G_L Add together to get the interaction feature

[0064] Furthermore, in step S33, the calculation formula for optimizing the lightweight image super-resolution reconstruction model based on frequency domain-spatial domain assisted Mamba in training using the L1 loss function is:

[0065] L1=‖I HR -I SR ‖

[0066] Among them, IHR Represents the high-resolution image features corresponding to the HR image dataset.

[0067] Furthermore, step S4 specifically includes:

[0068] According to the trained lightweight image super-resolution reconstruction model based on frequency domain-spatial domain assisted Mamba with weights, the test data set is input into the trained lightweight image super-resolution reconstruction model based on frequency domain-spatial domain assisted Mamba with ×2, ×3 and ×4 weights for testing, and the quantitative results and reconstructed high-resolution visual images are obtained;

[0069] Among them, quantitative results include peak signal-to-noise ratio and structural similarity;

[0070] The peak signal-to-noise ratio is calculated as:

[0071]

[0072] Where MSE represents the mean square error between image Y and image Z in the test data set, a represents the row index of the image position, b represents the column index of the image position, H represents the height of the image, W represents the width of the image, PSNR represents the peak signal-to-noise ratio, n represents the number of bits per pixel in the image, log 10 represents the logarithmic function;

[0073] The calculation formula of structural similarity is:

[0074]

[0075] Among them, SSIM(Y,Z) represents the structural similarity value between image Y and image Z, μ Y represents the average value of image Y, μ Z represents the mean value of image Z, σ YZ represents the covariance between image Y and image Z, σ Y represents the variance of image Y, σ Z represents the variance of the image Z, c1 and c2 represent the first constant and the second constant used to prevent the denominator from being zero, k1 and k2 represent constants, and L represents the dynamic range of the pixel values ​​in the image.

[0076] The present invention has the following beneficial effects:

[0077] The lightweight image super-resolution reconstruction method based on frequency domain-spatial domain assisted Mamba proposed in the present invention introduces frequency domain and spatial domain enhancements to supplement local information and enhance contextual information on the basis of Mamba. At the same time, in order to fully interact the above three different levels of information, a feature interaction layer based on transposition and self-attention is proposed to realize the interactive fusion of features. The relief of feature redundancy promotes the efficiency of image reconstruction, accelerates model reasoning and reduces computational complexity, and finally realizes lightweight image super-resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 A schematic flow chart of a lightweight image super-resolution reconstruction method based on frequency domain-spatial domain assisted Mamba proposed in the present invention;

[0079] Figure 2 It is a schematic diagram of the structure of a lightweight image super-resolution reconstruction model based on frequency-spatial domain assisted Mamba;

[0080] Figure 3 This is a schematic diagram of the structure of the visual Mamba layer;

[0081] Figure 4 Schematic diagram of the structure of the frequency domain-spatial domain enhancement layer;

[0082] Figure 5 Schematic diagram of the structure based on transposition and self-attention feature interaction layer. DETAILED DESCRIPTION

[0083] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0084] like Figure 1 As shown, the lightweight image super-resolution reconstruction method based on frequency domain-spatial domain assisted Mamba includes the following steps S1-S4:

[0085] S1. Obtain the original image dataset and perform data preprocessing on it to obtain the training dataset and the test dataset.

[0086] In this embodiment, the purpose of data preprocessing of the original image data set is to obtain high-resolution images and low-resolution images, and use the high-resolution images as labels of the low-resolution images so as to be used for training the lightweight image super-resolution reconstruction model based on the frequency domain-spatial domain assisted Mamba in the subsequent steps.

[0087] Specifically, step S1 includes S11-S15:

[0088] S11. Obtain the public DIV2K dataset, use the DIV2K dataset as the HR image dataset, and use the bicubic interpolation method to downsample each image in the HR image dataset by 2, 3, and 4 times to obtain images with multiple downsampling multiples.

[0089] S12, randomly flipping and rotating the images with multiple downsampling times to obtain enhanced low-resolution LR images.

[0090] S13: Use the HR image data set as first label data, use the enhanced low-resolution LR image as first original data, and use the first label data and the first original data as training data sets.

[0091] S14. Obtain the publicly available Set5, Set14, B100, Urban100 and Manga109 datasets as the HR images of the test set, and use the bicubic interpolation method to downsample each dataset by 2, 3 and 4 times respectively to obtain the LR images of the test set.

[0092] S15. Use the HR image of the test set as the second label data, use the LR image of the test set as the second original data, and use the second label data and the second original data as the test data set.

[0093] S2. Introduce the frequency-spatial domain Mamba layer and the transposition and self-attention feature interaction layer to build a lightweight image super-resolution reconstruction model based on the frequency-spatial domain assisted Mamba.

[0094] In this embodiment, the frequency-spatial domain Mamba layer and the transposition and self-attention feature interaction layer are introduced to construct a lightweight image super-resolution reconstruction model based on the frequency-spatial domain assisted Mamba. The connection relationship and structure are as follows: Figure 2 As shown, the lightweight image super-resolution reconstruction model based on frequency domain-spatial domain assisted Mamba includes a shallow feature extraction module, a deep feature extraction module, and a high-resolution image reconstruction module. The shallow feature extraction module is used to extract shallow features, the deep feature extraction module is used to extract deep features, and the high-resolution image reconstruction module is used for image reconstruction.

[0095] Specifically, the deep feature extraction module includes M feature mixing group modules and a second 3×3 convolutional layer; the feature mixing group module includes N multi-level feature interaction layers and a first 3×3 convolutional layer; the multi-level feature interaction layer includes a first LayerNorm layer, a frequency-spatial domain Mamba layer, a second LayerNorm layer, and a transposition- and self-attention-based feature interaction layer.

[0096] Among them, the frequency-spatial domain Mamba layer includes a visual Mamba layer, a frequency-spatial domain enhancement layer, and a first 1×1 convolutional layer.

[0097] Among them, the structure and connection relationship of the visual Mamba layer are as follows Figure 3 As shown, it includes an upper branch module, a lower branch module, and a third Linear layer; the upper branch module includes a first Linear layer, a first 3×3 depth convolution layer, a first SiLU layer, a 2D selective scanning module, and a third LayerNorm layer; the lower branch module includes a second Linear layer and a second SiLU layer.

[0098] Among them, the structure and connection relationship of the frequency domain-spatial domain enhancement layer are as follows: Figure 4 As shown, it includes a frequency domain enhancement module and a spatial domain enhancement module; the frequency domain enhancement module includes a second 3×3 deep convolution layer, a Fourier transform layer, a second 1×1 convolution layer, a first ReLU layer, and an inverse Fourier transform layer; the spatial domain enhancement module includes a third 3×3 deep convolution layer and a third 1×1 convolution layer.

[0099] Among them, the structure and connection relationship based on the transposition and self-attention feature interaction layer are as follows Figure 5 As shown, it includes a third 3×3 convolutional layer, a spatial channel permutation layer, a spatial self-attention layer, a channel self-attention layer, and a fifth 1×1 convolutional layer; the spatial channel permutation layer includes a first permutation layer, a fifth 3×3 deep convolutional layer, a second permutation layer, and a sixth 3×3 deep convolutional layer; the channel self-attention layer includes an adaptive average pooling layer, a fourth 3×3 deep convolutional layer, and a second sigmoid layer; the spatial self-attention layer includes a second ReLU layer, a fourth 1×1 convolutional layer, and a first sigmoid layer.

[0100] In addition, from Figure 2 It can also be seen that there is a residual connection structure in the first summation symbol ⊕ after the shallow feature extraction module and the deep feature extraction module, and there is also a residual connection structure between the first summation symbol ⊕ of the deep feature extraction module and the second summation symbol ⊕ of the deep feature extraction module, and there is a residual structure connected between the third summation symbol ⊕ after the deep feature extraction module and the shallow feature extraction module to keep the training stable.

[0101] S3. Input the training data set into the lightweight image super-resolution reconstruction model based on the frequency domain and spatial domain assisted Mamba for training, and obtain a trained lightweight image super-resolution reconstruction model based on the frequency domain and spatial domain assisted Mamba with weights, which is used to reconstruct the low-resolution image into a high-resolution image.

[0102] In this implementation, when training the lightweight image super-resolution reconstruction model based on frequency domain-spatial domain assisted Mamba, the feature mixing group is set to 4, the multi-level feature interaction layer is set to [4, 6, 4, 4], the channel is set to 72, and the L1 loss function optimization model is used to train the lightweight image super-resolution reconstruction model, so as to obtain a trained lightweight image super-resolution reconstruction model based on frequency domain-spatial domain assisted Mamba with weights, which is used to reconstruct low-resolution images into high-resolution images. In addition, the number of iterations during the training process is 500K, and the initial learning rate is set to 2e -4 , and when the iteration numbers are 250K, 350K, 400K, and 450K, the learning rate is continuously halved to keep the training stable.

[0103] Specifically, step S3 specifically includes S31-S33:

[0104] S31, input the training data set into the shallow feature extraction module of the lightweight image super-resolution reconstruction model based on the frequency domain-spatial domain assisted Mamba, and perform convolution processing using a 3×3 convolution layer to obtain the shallow feature F SH .

[0105] In this embodiment, the training data set is used as the input feature I LR ∈R H×W×3 Input the shallow feature extraction module of the lightweight image super-resolution reconstruction model based on frequency domain-spatial domain assisted Mamba to perform shallow feature extraction, where F SH ∈R H ×W×C , R represents the matrix, H represents the height, W represents the width, and C represents the number of channels.

[0106] S32, the shallow feature F SH Input the deep feature extraction module to extract mixed features and obtain deep features

[0107] Specifically, step S32 specifically includes S321-S327:

[0108] S321, the shallow feature F SH Input the first LayerNorm layer for normalization operation to obtain the first normalized feature X, and evenly divide the first normalized feature X from the channel dimension to obtain feature X1 and feature X2, that is:

[0109] X1,X2=Split(X,dim=C)

[0110] Among them, Split represents the feature segmentation function, dim represents the segmentation dimension, and C represents the number of channels.

[0111] In this embodiment, X∈R H×W×C , and the first normalized feature X is evenly divided from the channel dimension to obtain feature X1 and feature X2, X1,X2∈R H×W×C′ , That is, C′ is half of the number of channels C.

[0112] S322, feature X1 and feature X2 are respectively input into the visual Mamba layer of the frequency domain-spatial domain Mamba layer and the frequency domain-spatial domain enhancement layer for visual Mamba and frequency domain-spatial domain enhancement processing, and then spliced ​​in the channel dimension, and input into the first 1×1 convolutional layer for preliminary feature interaction to obtain preliminary global-local feature F g_l , including:

[0113] S3221, input feature X1 into the visual Mamba layer for visual Mamba processing to capture context information and obtain the global feature F m .

[0114] In this embodiment, F m ∈R H×W×C′ .

[0115] Specifically, step S3221 specifically includes S32211-S32213:

[0116] S32211. Input feature X1 into the first Linear layer for linear transformation operation to obtain the upper branch transformation feature F u_l , and input into the first 3×3 depth convolutional layer for encoding operation to obtain the upper branch encoding feature F u_c , and input into the first SiLU layer for activation operation to obtain the upper branch activation feature F u_s , and input into the 2D selective scanning module for global feature extraction, and input into the third LayerNorm layer for normalization operation to obtain the first feature F u .

[0117] In this embodiment, feature X1 is input into the first Linear layer for linear transformation operation. At this time, the number of channels is halved, so

[0118] S32212, input feature X1 into the second Linear layer for linear transformation operation to obtain the lower branch transformation feature F d_l , and input into the second SiLU layer for activation operation to obtain the second feature F d .

[0119] In this embodiment, feature X1 is input into the second Linear layer for linear transformation operation. At this time, the number of channels is halved, so

[0120] S32213, the first feature F u With the second feature F d Multiply and input into the third Linear layer for linear activation to obtain the global feature F m .

[0121] In this embodiment, according to steps S32211-S32213, the global feature F m The expression is:

[0122]

[0123] S3222, input feature X2 into the frequency domain-spatial domain enhancement layer for frequency domain-spatial domain enhancement processing to enhance context information and local information, and obtain frequency domain-spatial domain enhanced feature F f_s .

[0124] In this embodiment, F f_s ∈R H×W×C′ .

[0125] Specifically, step S3222 specifically includes S32221-S32223:

[0126] S32221. Input feature X2 into the second 3×3 deep convolution layer for deep convolution operation and extract local features to obtain the first local feature of feature X2, and input it into the Fourier transform layer for Fourier transform to convert the first local feature of feature X2 from the spatial domain to the frequency domain, and input it into the second 1×1 convolution layer for convolution operation and then into the first ReLU layer for activation operation to obtain activated frequency domain features, and input it into the inverse Fourier transform layer for inverse transformation to convert the activated frequency domain features from the frequency domain to the spatial domain to obtain the frequency domain enhanced feature F. f .

[0127] In this embodiment, F f ∈R H×W×C′ , and the frequency domain enhancement feature F f The expression is:

[0128] F f =RFFT(ReLU1(Conv2 1×1 (FFT(DConv2 3×3 (X2)))))

[0129] Among them, FFT stands for Fourier transform, and RFFT stands for inverse Fourier transform.

[0130] S32222, input feature X2 into the third 3×3 deep convolution layer for deep convolution operation to extract local features, obtain the second local feature of feature X2, and input it into the third 1×1 convolution layer for convolution operation to obtain spatial domain enhanced feature F s .

[0131] In this embodiment, F s ∈R H×W×C′ , and the spatial domain enhancement feature F s The expression is:

[0132] F s =Conv3 1×1 (DConv3 3×3 (X2)).

[0133] S32223, the frequency domain enhancement feature F f And the spatial domain enhanced feature F s Add together to get the frequency domain-spatial domain enhanced feature F f_s .

[0134] In this embodiment, F f_s ∈R H×W×C′ .

[0135] S3223, the global feature F m and frequency-spatial domain enhancement feature F f_s The concatenation is performed in the channel dimension and input into the first 1×1 convolutional layer for preliminary feature interaction to obtain the preliminary global-local feature F g_l .

[0136] In this embodiment, F g_l ∈R H×W×C , and the preliminary global-local features F g_l The expression is:

[0137] F g_l =Conv1 1×1 (Cat((F m ,F f_s ),dim=C))

[0138] Among them, Cat represents the splicing operation, and dim=C represents the splicing dimension.

[0139] S323, the preliminary global-local feature F g_l With shallow features F SH Add together to realize local residual connection and obtain the final global-local feature F G_L .

[0140] In this embodiment, F G_L ∈R H×W×C .

[0141] S324, the final global-local feature F G_L Input the second LayerNorm layer for normalization operation to obtain the second normalized feature

[0142] In this embodiment,

[0143] S325, the second normalized feature After the input is subjected to feature interaction fusion based on the transposition and self-attention feature interaction layers, it is combined with the final global-local feature F G_L Add, realize local residual connection, and obtain interactive features

[0144] In this embodiment,

[0145] Specifically, step S325 specifically includes S3251-S3255:

[0146] S3251, the second normalized feature Input the third 3×3 convolutional layer for preliminary feature fusion and channel compression, compress it to C / 4, and obtain the initialization feature F i .

[0147] In this embodiment, And initialize the feature F i The expression is:

[0148]

[0149] S3252, initialize feature F i The first permutation layer of the input spatial channel permutation layer performs a permutation operation by initializing the feature F i The spatial features of are replaced by channel dimension features to obtain the first replacement feature F i_p1 , and input the fifth 3×3 deep convolutional layer to extract the fineness feature, and input the second permutation layer to perform a permutation operation, converting the fineness feature from the channel dimension to a spatial feature to obtain the second permutation feature F i_p2 , and input into the sixth 3×3 deep convolutional layer for spatial feature encoding to obtain the permutation fusion feature F i_p .

[0150] In this embodiment, the space channel replacement layer performs cross replacement through space and channel; wherein, And replace the fusion feature F i_p The expression is:

[0151] F i_p =DConv63×3 (δ2(DConv5 3×3 (δ1(F i ))))

[0152] Here, δ1 represents the first substitution layer, and δ2 represents the second substitution layer.

[0153] S3253, initialize feature F The input is fed into the second ReLU layer of the spatial self-attention layer for nonlinear activation, and then fed into the fourth 1×1 convolutional layer to reduce the channel dimension to 1, and then fed into the first sigmoid layer for activation to obtain the first activation feature F i_s1 , the first activated feature F i_s1 With the initialization feature F i Multiply them together to get the spatial self-attention fusion feature F i_s .

[0154] In this embodiment, the spatial self-attention layer fuses information from different levels into a single channel; And the spatial self-attention fusion feature F i_s The expression is: F i_s =Sigmoid1(Conv4 1×1 (ReLU2(F i )))*F i .

[0155] S3254, initialize feature F i The adaptive average pooling layer of the input channel self-attention layer performs a pooling operation to obtain an initialization feature with a spatial size of 1×1, and enters the fourth 3×3 deep convolutional layer for encoding, and enters the second sigmoid layer for nonlinear activation operation to obtain the second activation feature F i_s2 , the second activation feature F i_s2 With the initialization feature F i Multiply them together to get the channel self-attention fusion feature F i_c .

[0156] In this embodiment, the channel self-attention layer fuses information from different levels into a single space; And the channel self-attention fusion feature F i_c The expression is:

[0157] F i_c =Sigmoid2(DConv4 3×3 (AdaptiveAvgPool2d(F i )))*F i

[0158] Among them, AdaptiveAvgPool2d is adaptive average pooling.

[0159] S3255, replace the fusion feature F i_p , spatial self-attention fusion feature F i_s And the channel self-attention fusion feature F i_c Add, and input the fifth 1×1 convolutional layer to expand the number of channels to C, and then add it to the final global-local feature F G_L Add together to get the interaction feature

[0160] In this embodiment, And interactive features The expression is:

[0161] S326, interactive features After repeating steps S321-S325 N times, the first 3×3 convolutional layer is input to obtain the mixed features.

[0162] In this embodiment,

[0163] S327, Mix the features After repeating step S326 M times, the second 3×3 convolutional layer is input to obtain the deep features.

[0164] In this embodiment,

[0165] S33, deep features With shallow features F SH After adding, the residual connection is realized, and then input into the high-resolution image reconstruction module to realize image reconstruction by using convolution layer and PixelShuffle upsampling to obtain the reconstructed feature I SR , and the L1 loss function is used to optimize the lightweight image super-resolution reconstruction model based on frequency domain-spatial domain assisted Mamba in training, and a trained lightweight image super-resolution reconstruction model based on frequency domain-spatial domain assisted Mamba with weights is obtained.

[0166] In this embodiment, the reconstruction feature I SR The expression is:

[0167]

[0168] Among them, P represents PixelShuffle upsampling, γ represents the upsampling multiple, Conv 3×3 is convolutional processing through a 3×3 convolutional layer, and ISR ∈R H×W×3 .

[0169] Specifically, in step S33, the calculation formula for optimizing the lightweight image super-resolution reconstruction model based on frequency domain-spatial domain assisted Mamba in training using the L1 loss function is:

[0170] L1=‖I HR -I SR ‖

[0171] Among them, I HR Represents the high-resolution image features corresponding to the HR image dataset.

[0172] S4. Input the test data set into the trained lightweight image super-resolution reconstruction model based on frequency domain-spatial domain assisted Mamba with weights to obtain a reconstructed high-resolution visualization image.

[0173] Specifically, step S4 includes:

[0174] According to the trained lightweight image super-resolution reconstruction model based on frequency domain and spatial domain assisted Mamba with weights, the test data sets were input into the trained lightweight image super-resolution reconstruction model based on frequency domain and spatial domain assisted Mamba with ×2, ×3 and ×4 weights for testing, and quantitative results and reconstructed high-resolution visual images were obtained; among them, the quantitative results include peak signal-to-noise ratio and structural similarity.

[0175] The peak signal-to-noise ratio is calculated as:

[0176]

[0177] Where MSE represents the mean square error between image Y and image Z in the test data set, a represents the row index of the image position, b represents the column index of the image position, H represents the height of the image, W represents the width of the image, PSNR represents the peak signal-to-noise ratio, n represents the number of bits per pixel in the image, log 10 Represents a logarithmic function.

[0178] The calculation formula of structural similarity is:

[0179]

[0180] Among them, SSIM(Y,Z) represents the structural similarity value between image Y and image Z, μ Y represents the average value of image Y, μ Z represents the mean value of image Z, σ YZ represents the covariance between image Y and image Z, σ Y represents the variance of image Y, σZ represents the variance of the image Z, c1 and c2 represent the first constant and the second constant used to prevent the denominator from being zero, k1 and k2 represent constants, and L represents the dynamic range of the pixel values ​​in the image.

[0181] In this embodiment, a higher peak signal-to-noise ratio PSNR indicates that the difference between the reconstructed high-resolution visualization image and the HR (high resolution) image is smaller, while a structural similarity value SSIM close to 1 indicates that the structural similarity between the reconstructed high-resolution visualization image and the HR (high resolution) image is higher.

[0182] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0183] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A lightweight image super-resolution reconstruction method based on frequency domain-spatial domain assisted Mamba, characterized in that: The following steps are involved: S1. Obtain the original image data set and perform data preprocessing on it to obtain the training data set and the test data set; S2, introduce the frequency-spatial domain Mamba layer and the transposition and self-attention feature interaction layer to build a lightweight image super-resolution reconstruction model based on the frequency-spatial domain assisted Mamba; S3, inputting the training data set into the lightweight image super-resolution reconstruction model based on the frequency domain-spatial domain assisted Mamba for training, and obtaining a trained lightweight image super-resolution reconstruction model based on the frequency domain-spatial domain assisted Mamba with weights, which is used to reconstruct the low-resolution image into a high-resolution image; S4. Input the test data set into the trained lightweight image super-resolution reconstruction model based on frequency domain-spatial domain assisted Mamba with weights to obtain a reconstructed high-resolution visualization image.

2. The lightweight image super-resolution reconstruction method based on frequency domain-spatial domain assisted Mamba according to claim 1, characterized in that: Step S1 specifically includes: S11, obtaining a public DIV2K dataset, using the DIV2K dataset as a HR image dataset, and downsampling each image in the HR image dataset by 2, 3, and 4 times using a bicubic interpolation method to obtain images with multiple downsampling multiples; S12, performing random flipping and rotation operations on the images with multiple downsampling times to obtain enhanced low-resolution LR images; S13, using the HR image data set as the first label data, using the enhanced low-resolution LR image as the first original data, and using the first label data and the first original data as the training data set; S14, obtain the public Set5 dataset, Set14 dataset, B100 dataset, Urban100 dataset and Manga109 dataset and use them as HR images of the test set, and use the bicubic interpolation method to downsample each dataset by 2, 3 and 4 times respectively to obtain the LR images of the test set; S15. Use the HR image of the test set as the second label data, use the LR image of the test set as the second original data, and use the second label data and the second original data as the test data set.

3. The lightweight image super-resolution reconstruction method based on frequency domain-spatial domain assisted Mamba according to claim 2 is characterized in that: In step S2, the lightweight image super-resolution reconstruction model based on frequency domain-spatial domain assisted Mamba includes a shallow feature extraction module, a deep feature extraction module, and a high-resolution image reconstruction module; The deep feature extraction module includes M feature mixing group modules and the second 3×3 convolutional layer; The feature mixing group module includes N multi-level feature interaction layers and the first 3×3 convolutional layer; The multi-level feature interaction layer includes the first LayerNorm layer, the frequency-spatial domain Mamba layer, the second LayerNorm layer, and the transposition- and self-attention-based feature interaction layer; The frequency-spatial domain Mamba layer includes a visual Mamba layer, a frequency-spatial domain enhancement layer, and the first 1×1 convolutional layer; The visual Mamba layer includes the upper branch module, the lower branch module, and the third Linear layer; The upper branch module includes the first Linear layer, the first 3×3 depth convolution layer, the first SiLU layer, the 2D selective scanning module, and the third LayerNorm layer; The lower branch module includes the second Linear layer and the second SiLU layer; The frequency domain-spatial domain enhancement layer includes a frequency domain enhancement module and a spatial domain enhancement module; The frequency domain enhancement module includes a second 3×3 depth convolution layer, a Fourier transform layer, a second 1×1 convolution layer, a first ReLU layer, and an inverse Fourier transform layer; The spatial domain enhancement module includes a third 3×3 depth convolution layer and a third 1×1 convolution layer; The feature interaction layer based on transposition and self-attention includes the third 3×3 convolution layer, spatial channel permutation layer, spatial self-attention layer, channel self-attention layer, and the fifth 1×1 convolution layer; The spatial channel permutation layer includes a first permutation layer, a fifth 3×3 depth convolution layer, a second permutation layer, and a sixth 3×3 depth convolution layer; The channel self-attention layer includes an adaptive average pooling layer, a fourth 3×3 depth convolution layer, and a second sigmoid layer; The spatial self-attention layer includes the second ReLU layer, the fourth 1×1 convolutional layer, and the first sigmoid layer.

4. The lightweight image super-resolution reconstruction method based on frequency domain-spatial domain assisted Mamba according to claim 3 is characterized in that: Step S3 specifically includes: S31, input the training data set into the shallow feature extraction module of the lightweight image super-resolution reconstruction model based on the frequency domain-spatial domain assisted Mamba, and perform convolution processing using a 3×3 convolution layer to obtain the shallow feature F SH ; S32, the shallow feature F SH Input the deep feature extraction module to extract mixed features and obtain deep features S33, deep features With shallow features F SH After adding, the residual connection is realized, and then input into the high-resolution image reconstruction module to realize image reconstruction by using convolution layer and PixelShuffle upsampling to obtain the reconstructed feature I SR , and the L1 loss function is used to optimize the lightweight image super-resolution reconstruction model based on frequency domain-spatial domain assisted Mamba in training, and a trained lightweight image super-resolution reconstruction model based on frequency domain-spatial domain assisted Mamba with weights is obtained.

5. The lightweight image super-resolution reconstruction method based on frequency domain-spatial domain assisted Mamba according to claim 4 is characterized in that: Step S32 specifically includes: S321, the shallow feature F SH Input the first LayerNorm layer for normalization operation to obtain the first normalized feature X, and evenly divide the first normalized feature X from the channel dimension to obtain feature X1 and feature X2, that is: X1,X2=Split(X,dim=C) Among them, Split represents the feature segmentation function, dim represents the segmentation dimension, and C represents the number of channels; S322, feature X1 and feature X2 are respectively input into the visual Mamba layer of the frequency domain-spatial domain Mamba layer and the frequency domain-spatial domain enhancement layer for visual Mamba and frequency domain-spatial domain enhancement processing, and then spliced ​​in the channel dimension, and input into the first 1×1 convolutional layer for preliminary feature interaction to obtain preliminary global-local feature F g_l , specifically including: S3221, input feature X1 into the visual Mamba layer for visual Mamba processing to capture context information and obtain the global feature F m ; S3222, input feature X2 into the frequency domain-spatial domain enhancement layer for frequency domain-spatial domain enhancement processing to enhance context information and local information, and obtain frequency domain-spatial domain enhanced feature F f_s ; S3223, the global feature F m and frequency-spatial domain enhancement feature F f_s The concatenation is performed in the channel dimension and input into the first 1×1 convolutional layer for preliminary feature interaction to obtain the preliminary global-local feature F g_l ; S323, the preliminary global-local feature F g_l With shallow features F SH Add together to realize local residual connection and obtain the final global-local feature F G_L ; S324, the final global-local feature F G_L Input the second LayerNorm layer for normalization operation to obtain the second normalized feature S325, the second normalized feature After the input is subjected to feature interaction fusion based on the transposition and self-attention feature interaction layers, it is combined with the final global-local feature F G_L Add, realize local residual connection, and obtain interactive features S326, interactive features After repeating steps S321-S325 N times, the first 3×3 convolutional layer is input to obtain the mixed features. S327, Mix the features After repeating step S326 M times, the second 3×3 convolutional layer is input to obtain the deep features.

6. The lightweight image super-resolution reconstruction method based on frequency domain-spatial domain assisted Mamba according to claim 5, characterized in that: Step S3221 specifically includes: S32211. Input feature X1 into the first Linear layer for linear transformation operation to obtain the upper branch transformation feature F u_l , and input into the first 3×3 depth convolutional layer for encoding operation to obtain the upper branch encoding feature F u_c , and input into the first SiLU layer for activation operation to obtain the upper branch activation feature F u_s , and input into the 2D selective scanning module for global feature extraction, and input into the third LayerNorm layer for normalization operation to obtain the first feature F u ; S32212, input feature X1 into the second Linear layer for linear transformation operation to obtain the lower branch transformation feature F d_l , and input into the second SiLU layer for activation operation to obtain the second feature F d ; S32213, the first feature F u With the second feature F d Multiply and input into the third Linear layer for linear activation to obtain the global feature F m .

7. The lightweight image super-resolution reconstruction method based on frequency domain-spatial domain assisted Mamba according to claim 6, characterized in that: Step S3222 specifically includes: S32221. Input feature X2 into the second 3×3 deep convolution layer for deep convolution operation and extract local features to obtain the first local feature of feature X2, and input it into the Fourier transform layer for Fourier transform to convert the first local feature of feature X2 from the spatial domain to the frequency domain, and input it into the second 1×1 convolution layer for convolution operation and then into the first ReLU layer for activation operation to obtain activated frequency domain features, and input it into the inverse Fourier transform layer for inverse transformation to convert the activated frequency domain features from the frequency domain to the spatial domain to obtain the frequency domain enhanced feature F. f ; S32222, input feature X2 into the third 3×3 deep convolution layer for deep convolution operation to extract local features, obtain the second local feature of feature X2, and input it into the third 1×1 convolution layer for convolution operation to obtain spatial domain enhanced feature F s ; S32223, the frequency domain enhancement feature F f And the spatial domain enhanced feature F s Add together to get the frequency domain-spatial domain enhanced feature F f_s .

8. The lightweight image super-resolution reconstruction method based on frequency domain-spatial domain assisted Mamba according to claim 7, characterized in that: Step S325 specifically includes: S3251, the second normalized feature Input the third 3×3 convolutional layer for preliminary feature fusion and channel compression, compress it to C / 4, and obtain the initialization feature F i ; S3252, initialize feature F i The first permutation layer of the input spatial channel permutation layer performs a permutation operation by initializing the feature F i The spatial features of are replaced by channel dimension features to obtain the first replacement feature F i_p1 , and input the fifth 3×3 deep convolutional layer to extract the fineness feature, and input the second permutation layer to perform a permutation operation, converting the fineness feature from the channel dimension to a spatial feature to obtain the second permutation feature F i_p2 , and input into the sixth 3×3 deep convolutional layer for spatial feature encoding to obtain the permutation fusion feature F i_p ; S3253, initialize feature F i The input is fed into the second ReLU layer of the spatial self-attention layer for nonlinear activation, and then fed into the fourth 1×1 convolutional layer to reduce the channel dimension to 1, and then fed into the first sigmoid layer for activation to obtain the first activation feature F i_s1 , the first activated feature F i_s1 With the initialization feature F i Multiply them together to get the spatial self-attention fusion feature F i_s ; S3254, initialize feature F i The adaptive average pooling layer of the input channel self-attention layer performs a pooling operation to obtain an initialization feature with a spatial size of 1×1, and enters the fourth 3×3 deep convolutional layer for encoding, and enters the second sigmoid layer for nonlinear activation operation to obtain the second activation feature F i_s2 , the second activation feature F i_s2 With the initialization feature F i Multiply them together to get the channel self-attention fusion feature F i_c ; S3255, replace the fusion feature F i_p , spatial self-attention fusion feature F i_s And the channel self-attention fusion feature F i_c Add, and input the fifth 1×1 convolutional layer to expand the number of channels to C, and then add it to the final global-local feature F G_L Add together to get the interaction feature 9. The lightweight image super-resolution reconstruction method based on frequency domain-spatial domain assisted Mamba according to claim 8, characterized in that: In step S33, the calculation formula for optimizing the lightweight image super-resolution reconstruction model based on frequency domain-spatial domain assisted Mamba in training using the L1 loss function is: L1=‖I HR -I SR ‖ Among them, I HR Represents the high-resolution image features corresponding to the HR image dataset.

10. The lightweight image super-resolution reconstruction method based on frequency domain-spatial domain assisted Mamba according to claim 9, characterized in that: Step S4 specifically includes: According to the trained lightweight image super-resolution reconstruction model based on frequency domain-spatial domain assisted Mamba with weights, the test data set is input into the trained lightweight image super-resolution reconstruction model based on frequency domain-spatial domain assisted Mamba with ×2, ×3 and ×4 weights for testing, and the quantitative results and reconstructed high-resolution visual images are obtained; Among them, quantitative results include peak signal-to-noise ratio and structural similarity; The peak signal-to-noise ratio is calculated as: Where MSE represents the mean square error between image Y and image Z in the test data set, a represents the row index of the image position, b represents the column index of the image position, H represents the height of the image, W represents the width of the image, PSNR represents the peak signal-to-noise ratio, n represents the number of bits per pixel in the image, log 10 represents the logarithmic function; The calculation formula of structural similarity is: Among them, SSIM(Y,Z) represents the structural similarity value between image Y and image Z, μ Y represents the average value of image Y, μ Z represents the mean value of image Z, σ YZ represents the covariance between image Y and image Z, σ Y represents the variance of image Y, σ Z represents the variance of the image Z, c1 and c2 represent the first constant and the second constant used to prevent the denominator from being zero, k1 and k2 represent constants, and L represents the dynamic range of the pixel values ​​in the image.

Images