Lightweight image super-resolution reconstruction method fusing spatial domain and frequency domain features

Abstract

The invention relates to a lightweight image super-resolution reconstruction method fusing spatial domain and frequency domain features, and the method comprises the steps: S1, transmitting a low-resolution image to a convolution layer, and extracting initial local structure information, S2, transmitting the initial local structure information to a spatial domain-frequency domain feature fusion module, and carrying out the fusion of the spatial domain and frequency domain features; s3, transmitting the low-resolution image to a deep convolutional neural network to obtain a shallow feature representation with a specific semantic inspiration effect, S4, transmitting the shallow feature representation to a cross-scale feature fusion module to obtain a cross-scale feature fusion result, and S5, carrying out cross-scale feature fusion on the cross-scale feature fusion result to obtain a cross-scale feature fusion result. S5, the spatial features and the deep semantic features are spliced in the channel dimension, feature fusion is achieved through a convolution layer, and then the features obtained after feature fusion are subjected to resolution improvement through an up-sampling module and then a reconstructed image is output; the method has the advantage that high-quality image reconstruction is realized while the lightweight class is ensured.

Description

Technical Field

[0001] This invention belongs to the field of image super-resolution reconstruction technology, specifically relating to a lightweight image super-resolution reconstruction method that integrates spatial and frequency domain features. Background Technology

[0002] Image super-resolution reconstruction technology processes low-resolution images to recover high-resolution images, and has wide applications in security monitoring, remote sensing imaging, and mobile devices. While existing deep learning methods have made significant progress in image reconstruction quality, they generally suffer from large model parameters and high computational costs, making them difficult to deploy on resource-constrained edge devices. To address this, researchers have proposed various lightweight methods to reduce model size; however, in practical applications, these methods still suffer from reduced reconstruction quality, weak shallow feature extraction capabilities, and insufficient global modeling capabilities, especially when dealing with complex textures and edge information, leading to blurring or loss of detail. These problems mainly stem from the compression of network expressive power in lightweight designs; how to improve efficiency while maintaining image quality remains a key technical challenge. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide a lightweight image super-resolution reconstruction method that fuses spatial and frequency domain features to achieve high-quality image reconstruction while maintaining lightweightness. This method involves globally modeling and fusing features in the spatial and frequency domains.

[0004] The technical solution of the present invention is as follows:

[0005] A lightweight image super-resolution reconstruction method that integrates spatial domain and frequency domain features includes the following steps;

[0006] S1: Feed the low-resolution image to the convolutional layer and extract preliminary local structural information;

[0007] S2. The preliminary local structural information is sent to the spatial-frequency domain feature fusion module to mine the spatial context and frequency domain information of the low-resolution image, thereby obtaining spatial features;

[0008] S3: Feed the low-resolution image into a deep convolutional neural network to obtain shallow feature representations that provide specific semantic inspiration;

[0009] S4: The shallow feature representation is fed into the cross-scale feature fusion module, which outputs deep semantic features.

[0010] S5: Spatial features and deep semantic features are concatenated along the channel dimension, and feature fusion is achieved through a convolutional layer. The fused features are then upsampled to improve resolution before outputting the reconstructed image.

[0011] Furthermore, the spatial-frequency feature fusion module includes a spatial-domain global feature extraction module and a frequency-domain global feature extraction module. The spatial-domain global feature extraction module is used to extract spatial-domain global features from the preliminary local structure information, and the frequency-domain global feature extraction module is used to extract frequency-domain global features from the preliminary local structure information. The preliminary local structure information is sent to the spatial-domain global feature extraction module and the frequency-domain global feature extraction module after being processed by layer normalization.

[0012] Furthermore, the feature extraction steps of the spatial domain global feature extraction module include:

[0013] S21. The features after layer normalization. The input spatial domain global feature extraction module reshapes the features after layer normalization into a two-dimensional matrix. ;

[0014] S22: Subsequently, three feature matrices—query Q, key K, and value V—are generated through linear projection:

[0015]

[0016] in, , is the science department parameter matrix;

[0017] S23: In the scaled dot product attention mechanism, the similarity between Q and K is calculated and scaled, and then V is weighted and aggregated after Softmax normalization, thereby effectively capturing global contextual information and extracting spatially aware global feature representations.

[0018]

[0020] Furthermore, the spatial domain global feature extraction module is followed by a contrast-aware channel attention mechanism, which performs channel-dimensional weighting on the spatial domain global features.

[0021] Furthermore, the frequency domain global feature extraction module structure includes the spatial domain global feature extraction module structure and adjusts the similarity between features to the frequency correlation between channels.

[0022] Furthermore, the feature extraction steps of the cross-scale feature fusion module include:

[0023] S41: Perform downsampling operations on the shallow feature representations sequentially to obtain three sets of low-resolution feature maps;

[0024] S42: Each low-resolution feature map first extracts spatial-frequency domain information through the SFFFM module, and then performs channel fusion processing through a 1×1 convolutional layer to obtain downsampled features;

[0025] S43: The downsampled features are restored to their original size through upsampling to obtain feature map A. Feature map A and shallow feature representations are concatenated along the channel dimension and then nonlinearly mapped and compressed through a 1×1 convolutional layer and the GELU activation function to obtain the fused features.

[0026] S44: The fused features are added to the shallow feature representations through skip connections to form the final deep semantic features.

[0027] Furthermore, the upsampling module consists of pixel rearrangement operations and 3×3 convolutional layers.

[0028] Compared with the prior art, the beneficial effects of the present invention are:

[0029] 1. This invention, by introducing a spatial-frequency domain fusion mechanism and a multi-scale semantic information integration strategy, effectively improves the structural restoration capability and texture detail representation of reconstructed images while maintaining a lightweight model, demonstrating superior perceptual quality. Due to its low parameter count and high inference speed, it effectively reduces hardware requirements, making it particularly suitable for real-time or mobile deployments. Thanks to its unique fusion mechanism, this invention exhibits strong generalization capabilities when processing different types of images. Attached Figure Description

[0030] Figure 1 This is a diagram of the SFFF-Net framework of the present invention;

[0031] Figure 2 This is a CSFFM framework diagram of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] like Figures 1 to 2 As shown, a lightweight image super-resolution reconstruction method that integrates spatial and frequency domain features includes the following:

[0034] 1. Spatial Feature Branches

[0035] In this branch, the input low-resolution image is first passed through a 3×3 convolutional layer to extract preliminary local structural information. Subsequently, the extracted preliminary local structural information is fed into the Spatial-Frequency Feature Fusion Module (SFFFM) to further mine the spatial context and frequency domain related information of the low-resolution image, thereby obtaining more discriminative spatial features.

[0036] The structure of the SFFFM module is as follows: Figure 1 As shown, the input features first undergo layer normalization to unify the feature distribution and improve the model's adaptability to different input images. Next, the module utilizes the Spatial-Domain Global Feature Extraction Module (SGFEM) and the Frequency-Domain Global Feature Extraction Module (FGFEM) to process the features, capturing a broader range of image representations.

[0037] In SGFEM, input features Reshaped into a two-dimensional matrix Then, three feature matrices—query Q, key K, and value V—are generated through linear projection:

[0038]

[0039] in, , where is the parameter matrix of the science department.

[0040] In the scaled dot product attention mechanism, the similarity between Q and K is calculated and scaled, then normalized by Softmax and weighted to aggregate V, thereby capturing global contextual information and extracting spatially aware global feature representations.

[0041]

[0042] To further enhance the response to important features, SGFEM is followed by a Contrast-Aware Channel Attention (CCA) mechanism, which weights the global features in the spatial domain by channel dimension to highlight salient information regions.

[0043] The FGFEM module structure is similar to the SGFEM module, the difference being that the FGFEM module focuses on the frequency correlation between channels. Specifically, it reshapes the input features into... Then get the same result Furthermore, by applying the same attention and CCA structure, the interaction between channels is modeled from the frequency dimension, further supplementing the global modeling capability of spatial branches.

[0044] 2. Semantic Feature Branch

[0045] In the semantic feature extraction path, the low-resolution image is first passed through the first layer of a pre-trained VGG19 network to obtain a shallow feature representation with semantic inspiration. This part mainly captures basic semantic information such as edge contours and texture trends in the image.

[0046] To further enhance semantic expressive power, this branch is followed by a Cross-Scale Feature Fusion Module (CSFFM), the structure of which is as follows: Figure 2 As shown, CSFFM aims to improve the model's ability to recover details at different spatial resolutions by fusing contextual semantic features from a multi-scale perspective.

[0047] The CSFFM module first downsamples the shallow feature representation Input sequentially, resulting in three sets of low-resolution feature maps (Input1, Input2, and Input3). Each downsampled low-resolution feature map is first processed by the SFFFM module to extract spatial-frequency information, and then fused through a 1×1 convolutional layer to obtain the downsampled features. Subsequently, these processed downsampled features are upsampled to restore their original size, resulting in feature map A, ensuring spatial size consistency for subsequent fusion. After concatenating the feature maps of all scales (feature map A and the shallow feature representation) along the channel dimension, a 1×1 convolutional layer is applied and non-linearly mapped and compressed using the GELU activation function. Finally, CSFFM adds the fused features to the original input through skip connections, forming the final deep semantic feature output.

[0048] 3. Feature Fusion and Image Reconstruction

[0049] Spatial features extracted by the dual branches are concatenated with deep semantic features along the channel dimension, and feature fusion is achieved through 1×1 convolution. This fusion process enhances the interaction and complementarity between different information sources, providing a unified representation for the final image reconstruction. The fused features are then input into an upsampling module for resolution enhancement.

[0050] The upsampling module consists of pixel shuffle operations and 3×3 convolutional layers, which can effectively transform images from low resolution to high resolution while preserving structural information and detailed textures, ultimately outputting high-quality reconstructed images.

[0051] This invention constructs a neural network structure for image super-resolution reconstruction, called SFFF-Net (Spatial-Frequency Feature Fusion Network). This neural network structure, by introducing a spatial-frequency domain fusion mechanism and a multi-scale semantic information integration strategy, effectively improves the structural restoration capability and texture detail representation of the reconstructed image while maintaining the model's lightweight nature. Figure 1 As shown, SFFF-Net adopts a dual-branch structure, extracting and reconstructing information from the input low-resolution image from two dimensions: spatial features and semantic features.

[0052] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A lightweight image super-resolution reconstruction method that integrates spatial and frequency domain features, characterized in that: Includes the following steps; S1: Feed the low-resolution image to the convolutional layer and extract preliminary local structural information; S2. The preliminary local structural information is sent to the spatial-frequency domain feature fusion module to mine the spatial context and frequency domain information of the low-resolution image, thereby obtaining spatial features; S3: Feed the low-resolution image into a deep convolutional neural network to obtain shallow feature representations that provide specific semantic inspiration; S4: The shallow feature representation is fed into the cross-scale feature fusion module, which outputs deep semantic features. S5: Spatial features and deep semantic features are concatenated along the channel dimension, and feature fusion is achieved through a convolutional layer. The fused features are then upsampled to improve resolution before outputting the reconstructed image.

2. The lightweight image super-resolution reconstruction method fusing spatial and frequency domain features according to claim 1, characterized in that: The spatial-frequency feature fusion module includes a spatial-domain global feature extraction module and a frequency-domain global feature extraction module. The spatial-domain global feature extraction module is used to extract spatial-domain global features from preliminary local structure information, and the frequency-domain global feature extraction module is used to extract frequency-domain global features from preliminary local structure information. The preliminary local structure information is sent to the spatial-domain global feature extraction module and the frequency-domain global feature extraction module after being processed by layer normalization.

3. The lightweight image super-resolution reconstruction method fusing spatial and frequency domain features according to claim 2, characterized in that: The feature extraction steps of the spatial domain global feature extraction module include: S21. The features after layer normalization. The input spatial domain global feature extraction module reshapes the features after layer normalization into a two-dimensional matrix. ; S22: Subsequently, three feature matrices—query Q, key K, and value V—are generated through linear projection: in, , is the science department parameter matrix; S23: In the scaled dot product attention mechanism, the similarity between Q and K is calculated and scaled, and then V is weighted and aggregated after Softmax normalization, thereby capturing global contextual information and extracting spatially aware global feature representations. 。 4. The lightweight image super-resolution reconstruction method fusing spatial and frequency domain features according to claim 3, characterized in that: The spatial domain global feature extraction module is followed by a contrast perception channel attention mechanism, which performs channel dimension weighting on the spatial domain global features.

5. The lightweight image super-resolution reconstruction method fusing spatial and frequency domain features according to claim 4, characterized in that: The structure of the frequency domain global feature extraction module is the same as that of the spatial domain global feature extraction module, and the similarity between features is adjusted to the frequency correlation between channels.

6. The lightweight image super-resolution reconstruction method fusing spatial and frequency domain features according to claim 1, characterized in that: The feature extraction steps of the cross-scale feature fusion module include: S41: Perform downsampling operations on the shallow feature representations sequentially to obtain three sets of low-resolution feature maps; S42: Each low-resolution feature map first extracts spatial-frequency domain information through the SFFFM module, and then performs channel fusion processing through a 1×1 convolutional layer to obtain downsampled features; S43: The downsampled features are restored to their original size through upsampling to obtain feature map A. Feature map A and shallow feature representations are concatenated along the channel dimension and then nonlinearly mapped and compressed through a 1×1 convolutional layer and the GELU activation function to obtain the fused features. S44: The fused features are added to the shallow feature representations through skip connections to form the final deep semantic features.

7. The lightweight image super-resolution reconstruction method fusing spatial and frequency domain features according to claim 1, characterized in that: The upsampling module consists of pixel rearrangement operations and 3×3 convolutional layers.

Images