An image super-resolution reconstruction method based on space-frequency dual-domain representation learning

Abstract

The application discloses an image super-resolution reconstruction method based on space-frequency dual-domain representation learning, and relates to the technical field of image super-resolution reconstruction. The method constructs a multi-scale receptive field through four-way differentiated convolution, synchronously captures coarse-grained structure and fine-grained texture information, and combines a convolution feedforward network to complete feature refinement; the method introduces a window multi-head self-attention mechanism to capture spatial long-range dependencies to optimize low-frequency global representation, and then realizes high-frequency information refinement from global to local through an efficient distillation structure; the method modulates spectral space information adaptively by means of a learnable frequency domain filter, simultaneously generates spatial attention guided by high-frequency prior, and promotes deep interaction of dual-domain features; the method realizes deep feature mapping through six recursively connected feature representation groups, and completes high-resolution image reconstruction in combination with a pixel shuffling layer. While keeping the model lightweight, the application effectively breaks through the inherent limitations of single architecture and single-domain representation, and significantly improves the comprehensive performance of super-resolution reconstruction.

Description

Technical Field

[0001] This application relates to the field of image super-resolution reconstruction technology, and in particular to an image super-resolution reconstruction method based on spatial-frequency dual-domain representation learning. Background Technology

[0002] Image super-resolution reconstruction, a core task in computer vision and image processing, aims to recover key information such as edge contours and texture details from high-resolution (HR) images using algorithms. It has been widely applied in various important fields, including medical imaging, remote sensing monitoring, consumer electronics, security monitoring, smart mines, and cultural heritage protection. For example, in medical imaging, this technology can improve the spatial resolution of CT and MRI images, aiding in the accurate identification of minute lesions; in smart mine scenarios, it can adapt to the limited hardware resources underground, restoring low-quality monitoring images to restore personnel features and equipment details; and in the consumer electronics field, it can meet the demand for high-definition image display on mobile devices while balancing computing power and bandwidth constraints.

[0003] With technological advancements, super-resolution methods have evolved from traditional interpolation and reconstruction to deep learning-driven approaches. However, despite performance breakthroughs achieved by models based on Convolutional Neural Networks (CNNs) and Transformers, CNNs are limited by their local receptive fields, making it difficult to capture long-distance pixel dependencies and easily leading to global structural distortion. While Transformers possess global modeling capabilities, their self-attention mechanism incurs excessive computational overhead, making them unsuitable for resource-constrained scenarios such as terminal devices.

[0004] To address the inherent limitations of CNNs and Transformers, such as local receptive field constraints and excessive computational overhead, researchers have proposed strategies like information distillation, channel compression, and attention mechanism optimization to achieve lightweight model deployment. However, significant limitations remain. Existing lightweight image super-resolution reconstruction methods, with limited parameters and computational overhead, struggle to simultaneously maintain high-frequency detail fidelity and global structural consistency. Specifically, single spatial domain representations cannot fully utilize global information in the frequency domain, dual-domain models lack deep interaction between spatial and frequency features, and the inherent limitations of CNNs and Transformers are not effectively addressed collaboratively, leading to problems such as blurred details and structural breaks in reconstructed images. Summary of the Invention

[0005] Therefore, it is necessary to provide an image super-resolution reconstruction method based on spatial-frequency dual-domain representation learning to address the aforementioned technical problems.

[0006] The following technical solution is adopted in this specification: This specification provides an image super-resolution reconstruction method based on spatial-frequency dual-domain representation learning, including: Acquire a low-resolution natural image to be reconstructed; An image super-resolution reconstruction model is constructed; the image super-resolution reconstruction model includes a shallow image feature extraction module, a deep image feature extraction module, and an image reconstruction module; the deep image feature extraction module includes six cascaded feature representation groups, each of which includes a multi-scale spatial domain feature extraction block, a spatial frequency information distillation block, a spatial frequency intermodulation attention module, and a standard convolution that are connected in sequence; The image shallow feature extraction module extracts shallow features from the low-resolution natural image to be reconstructed, and obtains a shallow feature map. The multi-scale spatial feature extraction block captures the coarse-grained structure and fine-grained texture information of the shallow feature map, refining its features. The spatial frequency information distillation block captures the long-range spatial dependencies of the refined shallow feature map, obtaining high-frequency information refinement features from global to local levels. The spatial frequency intermodulation attention module adaptively modulates the spectral spatial information of the high-frequency information refinement features and guides spatial attention generation with high-frequency priors, resulting in deep interactive dual-domain features. Finally, the standard convolution is applied to the deep interactive dual-domain features to obtain a deep feature map. The image reconstruction module adds the deep feature map to the low-resolution natural image to obtain the reconstructed high-resolution image.

[0007] Furthermore, the process of performing image super-resolution reconstruction using the aforementioned image super-resolution reconstruction model specifically includes: Low-resolution natural image to be reconstructed Through the shallow feature extraction module Convolution is used to extract shallow features, resulting in a shallow feature map. ; The shallow features Features are extracted and mapped using the six feature representation groups of the deep feature mapping module to obtain a deep feature map. ; The deep features By rebuilding the module Standard convolution and low-resolution images After adding the pixels, the pixels are shuffled to obtain the reconstructed super-resolution image. ; The shallow feature extraction module is represented as follows: The deep feature mapping module is represented as follows: The reconstruction module is represented as follows: in, Represented as a low-resolution image; The model uses the extracted shallow image features as input to the deep feature extraction module; express Standard convolution; Indicates the first A feature characterization group; Indicates the first The output characteristics of each FRG; This refers to the number of FRGs used; The deep features are obtained by deep feature mapping; The reconstructed super-resolution image; Shuffle the pixels.

[0008] Furthermore, the multi-scale spatial feature extraction block consists of a multi-scale feature extraction module and a convolutional feedforward network; The multi-scale feature extraction module consists of a small-scale convolutional layer, an equivalent medium-scale convolutional layer, and two orthogonal large-scale strip convolutional layers. The small-scale convolutional layer is used to extract local detail texture features of the shallow feature map. The equivalent medium-scale convolutional layer consists of cascaded depthwise convolution, depth dilation convolution, and pointwise convolution, used to capture local structural information of the shallow feature map. The orthogonal large-scale strip convolutional layer consists of two depthwise separable strip convolutions, used to extract large-scale features of the shallow feature map. The convolutional feedforward network consists of three convolutional layers and two GELU activation functions; it is used to refine the multi-scale features extracted by the multi-scale feature extraction module and to enhance the effective multi-scale feature information through nonlinear activation functions.

[0009] Furthermore, the feature extraction process of the multi-scale spatial feature extraction block specifically includes: Feature map output by the shallow feature extraction module After layer normalization (LN), the feature map is processed along the channel dimension. Divided into four groups of features; The separated features are input into different parallel branches, and feature extraction is performed through convolution operations respectively; The outputs of each branch are concatenated, and then channel attention weighting is applied. Input the weighted features into Smoothing and residual connections are performed in standard convolutions to obtain multi-scale features; The multi-scale features are refined and extracted using a convolutional feedforward network, and the effective multi-scale feature information is enhanced by a nonlinear activation function to obtain multi-scale spatial features.

[0010] Furthermore, the space-frequency information distillation block consists of multiple It consists of a standard convolutional layer, a depthwise separable convolutional layer, a window-based multi-head self-attention mechanism, and three efficient feature refinement blocks; The window-based multi-head self-attention mechanism consists of multiple linear projection transformations, window splitting, window merging operations, and a convolutional feedforward network. The efficient feature refinement block consists of two convolutional layers.

[0011] Furthermore, the feature extraction process of the space frequency information distillation block specifically includes: By dividing the multi-scale feature map extracted by the multi-scale spatial feature extraction block into non-overlapping local windows, the feature vector of each window is transformed into the form of multi-head attention through tensor dimension rearrangement; The feature vector is mapped into query, key, and value vectors through three linear projection transformations. Each attention head maps the input vector to the subspace represented by the attention head and performs self-attention computation; The output vectors of all attention heads are concatenated and aggregated along the channel dimension, and a linear projection mapping is used to obtain an output containing local dependency information, resulting in a local feature map; the size of the local feature map is then converted to the size of the original feature map by window merging. pass The depthwise separable convolution extracts the coarse-grained high-frequency texture of the local feature map, and then... Standard convolution and GELU activation function are used to fuse channel information to obtain a coarse-grained feature map; and through... Standard convolution and sigmoid activation function are used to extract fine-grained information from the local feature map pixel by pixel to obtain a fine-grained feature map; The fine-grained feature map is weighted onto the coarse-grained feature map to obtain high-frequency detail information features fused across multiple scales.

[0012] Furthermore, the spatial frequency intermodulation attention module includes an adaptive frequency domain attention module and an efficient spatial attention module; The adaptive frequency domain attention module consists of three It consists of a standard convolution, two LeakyReLU activation functions, and a learnable frequency domain filter; The high-efficiency spatial attention module is composed of It consists of standard convolution, stride convolution, depthwise separable convolution groups, average pooling layers, deconvolution upsampling, and the GELU activation function.

[0013] Furthermore, the feature extraction process of the adaptive frequency domain attention module specifically includes: Based on the high-frequency detail information features extracted from the space-frequency information distillation block, the spectra of multiple image blocks are obtained through window splitting and fast Fourier transform. The real and imaginary parts of the spectrum are separated and processed by three... The bottleneck network, consisting of standard convolution and two LeakyReLU activation functions, fuses channel information to obtain spectral spatial information. The spectral spatial information is adaptively modulated using a learnable frequency domain filter, and a frequency domain attention map is obtained through frequency domain jump connections, inverse fast Fourier transform, and window merging.

[0014] Furthermore, the feature extraction process of the efficient spatial attention module specifically includes: pass Standard convolution reduces the dimensionality of high-frequency detail features extracted from the spatial frequency information distillation block to one-quarter of the channels; and uses a stride of 2. Depthwise separable convolution completes the first downsampling; based on the downsampled features, lightweight local thinning and average pooling are performed to obtain a low-scale representation of one-quarter of the image size; based on the low-scale representation features, through... A depthwise separable convolutional group is used to perform context aggregation to obtain low-resolution features; and the low-resolution features are mapped back to the original scale through a transposed convolution with a stride of 4. The structural information of high-frequency detail information features extracted by local convolution is obtained by extracting the spatial frequency information distillation block; and based on the explicit computation of high-frequency prior information structure, rapidly changing texture and gradient information are separated from the high-frequency detail information features to obtain high-frequency enhanced features. Based on the aforementioned high-frequency enhancement features and low-resolution features, after fusion in the high-resolution space according to the channel stitching method, through... Standard convolution and sigmoid activation function dynamically generate spatial attention feature maps.

[0015] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects: In the image super-resolution reconstruction method based on spatial-frequency dual-domain representation learning provided in this specification, an image deep feature extraction module consisting of six cascaded feature representation groups is constructed. Each feature representation group consists of a multi-scale spatial domain feature extraction block, a spatial-frequency information distillation block, a spatial frequency intermodulation attention module, and a standard convolution, which are connected in sequence. Based on the image deep feature extraction module combined with the image shallow feature extraction module and the image reconstruction module, an image super-resolution reconstruction model is constructed.

[0016] Existing lightweight image super-resolution reconstruction methods, with limited parameters and computational overhead, struggle to simultaneously maintain high-frequency detail fidelity and global structural consistency. Specifically, single spatial domain representations cannot fully utilize global information in the frequency domain, dual-domain models lack deep interaction between spatial and frequency features, and the inherent defects of CNNs and Transformers have not been effectively addressed in a collaborative manner, leading to problems such as blurred details and structural breaks in reconstructed images.

[0017] Among them, the combination of spatial multi-scale feature extraction and frequency global structure modeling by using scale spatial feature extraction block, spatial frequency information distillation block and spatial frequency intermodulation attention module not only retains the CNN's ability to accurately capture high-frequency details such as edges and textures, but also optimizes the consistency of the overall image structure by leveraging the global receptive field of the frequency domain. This solves the global fragmentation problem caused by the local receptive field of CNN, while avoiding the inflexibility of high-frequency detail extraction in the pure Transformer architecture. It also solves the inherent limitations of single architecture and single-domain representation in existing methods. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This document provides a flowchart illustrating an image super-resolution reconstruction method based on spatial-frequency dual-domain representation learning. Figure 2 A schematic diagram of the overall network structure of an image super-resolution reconstruction method based on spatial-frequency dual-domain representation learning, provided in this specification; Figure 3 This specification provides a schematic diagram of the multi-scale spatial feature extraction block (MSFEB) network structure for an image super-resolution reconstruction method based on spatial-frequency dual-domain representation learning. Figure 4 A schematic diagram of the spatial frequency information distillation block (SFIDB) network structure of an image super-resolution reconstruction method and apparatus based on spatial frequency dual-domain representation learning, as provided in this specification. Figure 5This specification provides a schematic diagram of the spatial frequency intermodulation attention (SFIA) network structure for an image super-resolution reconstruction method based on spatial frequency dual-domain representation learning. Figure 6 A visual comparison diagram of images reconstructed by the algorithm of this invention and other algorithms in Urban100 with a four-fold reconstruction of image img044; Figure 7 A visual comparison diagram of images reconstructed by the algorithm of this invention and other algorithms in Urban100 with a four-fold reconstruction (img056). Figure 8 This is a visual comparison diagram of the image reconstructed by the algorithm of the present invention provided in this specification and other algorithms reconstructed four times in Urban100 (img062). Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this application.

[0021] In existing technologies, traditional interpolation methods (such as bicubic interpolation) are simple to implement, but they are prone to causing image blurring and artifacts; reconstruction methods (such as iterative back projection and maximum a posteriori probability) rely on manually designed priors, have limited generalization ability and low computational efficiency; shallow learning methods (such as sparse representation and neighborhood embedding) have established the core idea of ​​data-driven approach, but their feature representation ability is insufficient and they are difficult to cope with the reconstruction needs of complex images.

[0022] The image super-resolution reconstruction method based on spatial-frequency dual-domain representation learning provided by this invention can solve the following core problems existing in the prior art: The inherent limitations of single architecture and single domain representation: existing lightweight models mostly rely on a single CNN or Transformer architecture, focusing only on learning features in a single dimension of the spatial or frequency domain. CNNs are difficult to model long-distance dependencies, resulting in insufficient global consistency. Transformers lack flexibility in extracting high-frequency details and have high computational costs. Single domain representation cannot fully characterize the high and low frequency information of an image. Inefficient dual-domain feature fusion: Although some dual-domain models attempt to combine spatial frequency information, the fusion method is simple (such as direct splicing or weighted summation), which fails to achieve deep interaction and adaptive modulation of spatial frequency features, resulting in low matching degree between high-frequency details and global structure and insufficient feature utilization. The challenge of balancing lightweightness and performance: Existing lightweight models often simplify feature extraction and fusion modules to control the number of parameters, resulting in weak multi-scale detail capture capabilities. This makes it difficult to simultaneously meet the actual requirements of detail accuracy, global structural consistency, and computational efficiency in reconstructed images, thus limiting their deployment effectiveness in resource-constrained scenarios.

[0023] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0024] Figure 1 This is a schematic diagram of the image super-resolution reconstruction method based on spatial-frequency dual-domain representation learning in this specification, which specifically includes the following steps: S101: Construct an image super-resolution reconstruction model framework based on spatial-frequency dual-domain representation learning and extract shallow features.

[0025] A framework for image super-resolution reconstruction based on spatial-frequency dual-domain representation learning is constructed. This framework consists of three parts: shallow image feature extraction, deep image feature mapping, and image reconstruction; the following formula applies: in, Represented as a low-resolution image, The model uses the extracted shallow image features as input to the deep feature extraction module. express Standard convolution, Indicates the first A set of feature representations, Indicates the first The output features of each FRG For shallow image features, It refers to the number of FRGs used. The deep features are obtained from deep feature mapping. The reconstructed super-resolution image. Shuffle the pixels.

[0026] S102: A multi-scale spatial feature extraction block is proposed. A multi-scale receptive field is constructed by four convolutions of different sizes to capture multi-level spatial information from coarse to fine granularity. The features are then further refined by a convolutional feedforward network.

[0027] A proposed multi-scale spatial feature extraction block (MSFEB) is as follows: Figure 4 As shown, it includes: To address the issues of high memory access costs and slow execution speed in large-kernel deep convolutions, this invention proposes a multi-scale spatial feature extraction block to reduce the number of parameters and improve execution efficiency while maintaining model performance. This block extracts multi-scale features through convolutions of different sizes, providing rich feature inputs for subsequent enhancement and refinement, and utilizes channel segmentation to reduce complexity.

[0028] For input features After performing layer-normalized LN, we divide it into four groups of features along the channel dimension. Then, we input the separated features into different parallel branches, as shown in the following formula: in, Presentation layer normalization operation, This indicates a channel splitting operation. express Depth-separable convolution, express Depth-separable convolution, express Depth-separable convolution, express Depth-separable dilated convolution, express The standard convolution.

[0029] Furthermore, the outputs of each branch are concatenated, then channel attention weighting is applied, and finally input into... Smoothing is performed during standard convolution, and residual connections are added at the end.

[0030] in, This indicates a channel splicing operation. For channel attention operations, express The standard convolution.

[0031] Multi-scale features are refined using a Convolutional Feed-Forward Network (ConvFFN), and effective multi-scale feature information is further enhanced by a non-linear activation function. This process can be represented as follows: in, Presentation layer normalization operation, This represents the GELU activation function. This represents the output features of the multi-scale spatial feature extraction block.

[0032] S103: Construct a Spatial-frequency Information Distillation Block (SFIDB), which introduces multi-head self-attention with a window into the traditional distillation structure to accurately capture long-range dependencies in the spatial domain, and then uses an efficient information distillation structure to achieve refined extraction of high-frequency information from global to local.

[0033] To address the shortcomings of traditional information distillation architectures, which rely solely on local convolutions in CNNs, making it difficult to model long-distance pixel dependencies and neglecting low-frequency information, resulting in insufficient global consistency of image structure, this invention proposes a spatial-frequency information distillation block. This block integrates the complementary advantages of Transformer self-attention and CNN distillation structures, enhancing the network's ability to capture effective information in both spatial and frequency domains. This process can be represented as follows: in, This indicates a window-based multi-head self-attention mechanism. Indicates the first indivual Distillation operation of standard convolution, For the first The saved features. Indicates the first EFRB For the first Features of each EFRB output.

[0034] Window-based Multi-head Self-Attention (W-MSA) divides the feature map into non-overlapping local windows. The feature vector of each window is transformed into a multi-head attention form through tensor dimensionality rearrangement. Three linear projection transformations are used to map the feature vectors into query, key, and value vectors. Each attention head maps the input vector into its own representation subspace and performs corresponding self-attention computation. The output vectors of all attention heads are concatenated and aggregated along the channel dimension, and a linear projection mapping is used to obtain an output containing local dependency information. The local feature maps are then merged into the original feature map size through window merging. For a given input feature... This process can be represented as:

[0035] in, This is the input feature map for this module. This indicates a window splitting operation. This indicates a tensor dimension rearrangement operation. Represents a linear projection transformation. and They represent the first A query, key, and value vector with a self-attention head. For the number of heads that receive self-attention, The dimension of the feature vector. Indicates the first Key vector in a self-attention head transpose, Indicates the first The relative position offset of each self-attention head This indicates a window merging operation.

[0036] The Efficient Feature Refinement Block (EFRB) mainly consists of two branches to extract high-frequency detail features at different granularities. The lower branch reduces complexity by using channel splitting. Depth-separable convolution extracts coarse-grained high-frequency texture, and then through... Standard convolution and GELU activation function are used to fuse channel information. The upper branch passes through... Standard convolution and the sigmoid activation function extract fine-grained information pixel by pixel and simultaneously weight it onto the coarse-grained feature map of the next branch, efficiently fusing multi-scale high-frequency detail information to achieve pixel-level attention weighting. This process can be represented as:

[0037] in, This indicates a channel splitting operation. This represents the GELU activation function. express Depth-separable convolution, This represents the Sigmoid activation function. This represents the pixel-level spatial attention map extracted from the upper branch.

[0038] S104: Design Spatial Frequency Intermodulation Attention (SFIA) as follows Figure 5 As shown, by adaptive modulation of spectral spatial information and spatial attention generation guided by high-frequency information, deep interaction of spatial and frequency dual-domain features is promoted, thereby improving the network's ability to represent high-frequency information.

[0039] It mainly includes the Adaptive Frequency Domain Attention Module (AFDAM) and the Efficient Spatial Attention Module (ESAM).

[0040] The Adaptive Frequency Domain Attention Module (AFDAM) first performs a windowing operation on the input feature map, then performs a Fast Fourier Transform (FFT) to obtain the spectrum of each image block. To ensure that the phase angle is not destroyed, the real and imaginary parts of the spectrum are separated and fed into three separate inputs. A bottleneck network consisting of standard convolutions and two LeakyReLU activation functions is used to fuse channel information. Next, a learnable frequency-domain filter is used to adaptively modulate the spectral information, while frequency-domain skip connections are introduced to preserve the original input information. Finally, an attention map is obtained through inverse Fast Fourier Transform and window merging operations. This process can be represented as:

[0041] in, This is the input feature map for this module. This indicates a window splitting operation. This indicates the Fast Fourier Transform operation. This represents the spectrum obtained by FFT. This represents the LeakyReLU activation function. This represents an adaptive frequency domain filter. This represents the feature map after modulation by the filter. This indicates the inverse fast Fourier transform operation. This indicates a window merging operation. This is the attention map output by the adaptive frequency domain attention module.

[0042] The High-Efficiency Spatial Attention Module (ESAM) firstly... Standard convolution reduces the feature dimension to one-quarter of the channels, making subsequent computations more lightweight. Then, a stride of 2 is used... Depthwise separable convolution performs the first downsampling, and then performs lightweight local thinning on this feature. Average pooling is then used to further obtain a low-scale representation of one-quarter of the image size. This is achieved at a more compact spatial scale through... Depthwise separable convolutional groups are used for context aggregation, enabling the network to obtain a larger effective receptive field and encode structural dependencies across regions. Then, a transposed convolution with a stride of 4 maps the low-resolution features back to the original scale in one pass, aligning them with the high-resolution detail branches. This process can be represented as:

[0043] in, This is the input feature map for this module. This indicates a step size of 2. Depth-separable convolution, This indicates the average pooling operation. This represents a depthwise separable convolutional group. This indicates a transposed convolution upsampling operation with a stride of 4. This indicates the output result of the low-resolution branch.

[0044] To enhance sensitivity to edge and texture regions, this module incorporates a lightweight high-frequency detail enhancement component for the high-resolution branch, in addition to the main branch. This branch extracts structural information through local convolutions and introduces explicitly computed high-frequency prior information structures to separate rapidly changing texture and gradient information from the input features, making the attention map more responsive to detail regions. This process can be represented as: in, To approximate the high-frequency residual features obtained by explicit extraction using the Laplacian operator, This represents the GELU activation function. This is the output feature enhanced by the high-resolution branch.

[0045] Next, the high-frequency enhanced features of the high-resolution branch and the features obtained by upsampling from the low-resolution branch are fused in the high-resolution space according to the channel stitching method, and then... Standard convolutions and sigmoid activation functions dynamically generate spatial attention maps. Finally, this attention mask is applied to the input features, enabling the network to adaptively enhance key detail regions and suppress redundant regions, thus achieving rich global context and accurate local texture localization while maintaining lightweight operation.

[0046] in, This represents the Sigmoid activation function. This represents the attention map generated by the efficient spatial attention module. This represents the output features after spatial attention weighting.

[0047] S105: Construct multiple feature representation groups containing residual connections and connect them recursively to further realize deep feature mapping. Finally, reconstruct a high-resolution image through upsampling.

[0048] The following formula applies to the feature representation set: in, Indicates the input feature map, For multi-scale spatial domain feature extraction blocks, For space frequency information distillation block, For spatial frequency intermodulation attention, express Standard convolution, The feature representation group represents the input feature map. The output of .

[0049] The algorithm framework of this invention is as follows: Figure 2 As shown, the algorithm consists of three modules: shallow image feature extraction, deep image feature extraction, and image reconstruction.

[0050] The image shallow feature extraction module consists of a convolutional kernel with a size of Standard convolutional composition, low-resolution image First pass through one Convolution is used to extract shallow features, resulting in shallow features. ; The deep image feature extraction module consists of six feature representation groups. Each feature representation group comprises a multi-scale spatial domain feature extraction block, a spatial frequency information distillation block, a spatial frequency intermodulation attention block, and a [missing information]. Standard convolutional composition, shallow features Deep features are obtained through multiple feature representation groups. ; The image reconstruction module consists of Composed of standard convolutional and pixel shuffling layers, deep features go through Standard convolution and low-resolution images After adding the pixels, shuffling them yields the reconstructed super-resolution image. .

[0051] While maintaining the principle of lightweight model design, this invention mines more comprehensive image features through spatial-frequency dual-domain collaborative representation learning, and at the same time constructs a multi-module collaborative feature enhancement system, which significantly improves the overall performance of super-resolution reconstruction. First, an image super-resolution reconstruction model framework based on spatial-frequency dual-domain representation learning is built, covering four core links: shallow image feature extraction, deep spatial-frequency dual-domain feature extraction, dual-domain feature fusion, and image reconstruction. In the deep dual-domain feature extraction module, a multi-scale spatial feature extraction block MSFEB is used. Differentiated receptive fields are constructed through four convolutions of different sizes to simultaneously capture coarse-grained structure and fine-grained texture information. Combined with the convolutional feedforward network ConvFFN, feature refinement is completed, which enhances the representation capability of multi-scale details in the spatial domain. Subsequently, a spatial frequency information distillation block (SFIDB) is introduced. Through multi-head self-attention via a window, long-range spatial dependencies are accurately captured to optimize low-frequency global representation. Then, an efficient distillation structure is used to refine high-frequency information from global to local levels, balancing global consistency with local detail accuracy. To achieve deep interaction between dual-domain features, a spatial frequency intermodulation attention mechanism (SFIA) is designed. This mechanism adaptively modulates spectral spatial information using a learnable frequency domain filter, while high-frequency priors guide spatial attention generation, significantly improving the effectiveness of dual-domain feature fusion. In summary, this invention effectively overcomes the inherent limitations of single architecture and single-domain representation, solving the performance degradation problems such as global structural breaks and high-frequency detail blurring that easily occur in lightweight image super-resolution networks when controlling parameter quantity and computational cost.

[0052] The technical effects of this invention are described in detail below based on performance tests and experimental analysis. Increased scale exacerbates the loss of high-frequency features in images; the spatial-frequency dual-domain representation network of this invention can effectively recover high-frequency details. Quantitative analysis shows that, under a scaling factor of 4, the model of this invention is significantly superior to current advanced lightweight image super-resolution reconstruction models, especially in recovering high-frequency detail features. The comparison results are shown in Table 1. Note that "-" indicates that the parameter is default.

[0053] Table 1 Comparison of Indicators under a 4x Scaling Factor Observing Table 1, it can be seen that the model proposed in this invention, under a scaling factor of 4, has significantly higher PSNR and SSIM indices than other models while maintaining a low number of parameters.

[0054] To further demonstrate the superiority of the network model proposed in this invention, image reconstruction experiments were conducted on the Urban100 dataset with complex textures, increasing the number of reconstructions by a factor of four. The visual effects of the images reconstructed using this algorithm were compared with those of other algorithms. The comparison results are as follows: Figure 6 , Figure 7 and Figure 8 As shown in the images, it is clear from these image results that, compared to other models, the algorithm model proposed in this invention can reconstruct images more realistically; Figure 6 As can be seen from the image, the algorithm model proposed in this invention clearly recovers the shape and number of grids, while the remaining images all exhibit obvious blurring and structural inconsistencies; Figure 7 As can be seen, the lines in the original super-resolution image are straight. Images reconstructed by other models suffer from visual problems such as blurred details, distorted lines, and severe jagged edges. However, the image reconstructed by the algorithm proposed in this invention is essentially identical to the original image. Figure 8 As can be seen, the image reconstructed by the algorithm proposed in this invention has a neat and consistent appearance, while other methods fail to accurately recover the surface lines of buildings. This demonstrates that the algorithm model proposed in this invention has significant advantages over other algorithm models in image super-resolution reconstruction.

[0055] To further demonstrate the effectiveness of the proposed image super-resolution reconstruction model based on spatial-frequency dual-domain representation learning, ablation experiments were conducted to further analyze the impact of different modules on network performance. Four algorithm experiments were designed, including single-module and multi-module combinations. Algorithm 0 served as the baseline model, and Algorithm 3 represented the proposed network model using all three modules. After 150K iterations of training, a test with a scaling factor of 4 was performed on the Urban100 dataset. The test results are shown in Table 2, where × indicates that this operation was not used, and √ indicates that this operation was used.

[0056] Table 2 Impact of different modules on network performance As observed in Table 2, compared to Algorithm 0, Algorithm 1 increases PSNR and SSIM by 0.21 dB and 0.0077, respectively; compared to Algorithm 1, Algorithm 2 increases PSNR and SSIM by 0.71 dB and 0.0208, respectively, while the number of parameters and FLOPs only increases by 24.6% and 23.4%, respectively. This indicates that the Spatial Frequency Information Distillation Block (SFIDB) proposed in this invention has a significant performance improvement effect; compared to Algorithm 2, Algorithm 3 increases PSNR and SSIM by 0.05 dB and 0.0016, respectively, while the number of parameters and FLOPs only increases by 13.4% and 12.6%, respectively. This indicates that the Spatial Frequency Intermodulation Attention (SFIA) proposed in this invention can effectively fuse effective information from both the spatial and frequency domains and achieve better reconstruction results.

[0057] This invention also provides an image super-resolution reconstruction device based on space-frequency dual-domain representation learning, comprising: Acquire a low-resolution natural image to be reconstructed; A framework for image super-resolution reconstruction model based on spatial-frequency dual-domain representation learning is constructed, which includes a shallow feature extraction module, a deep feature mapping module, and a reconstruction module. The model framework utilizes a depthwise separable convolution, a depthwise separable dilated convolution, and a... Standard convolutions are used to construct equivalent mid-scale convolutional layers; two depthwise separable strip convolutions are used to construct orthogonal large-scale strip convolutional layers; a multi-scale feature extraction module is constructed using a small-scale convolutional layer, one equivalent mid-scale convolutional layer, and two orthogonal large-scale strip convolutional layers. A convolutional feedforward network is constructed using three convolutional layers and two GELU activation functions. The multi-scale feature extraction module, the convolutional feedforward network, and layer-normalized LN are integrated to form a multi-scale spatial feature extraction block (MSFEB) containing different branches.

[0058] A window-based multi-head self-attention mechanism, W-MSA, is constructed using multiple linear projection transformations, window splitting, window merging operations, and a convolutional feedforward network. Two convolutional layers are used to extract feature information at different granularities to construct an efficient feature refinement block (EFRB). Multiple... Standard convolutional layers, a window-based multi-head self-attention mechanism, and three efficient feature refinement blocks are integrated to form the space-frequency information distillation block SFIDB, which contains multiple distillation branches;

[0059] Use three An adaptive frequency domain attention module (AFDAM) is constructed using standard convolution, two Leaky ReLU activation functions, and a learnable frequency domain filter; [The last part, "utilizing," appears to be incomplete and lacks context. It's unclear what "AFDAM" refers to.] A high-efficiency spatial attention module (ESAM) is constructed using standard convolutions, strided convolutions, depthwise separable convolutional groups, average pooling layers, deconvolution upsampling, and the GELU activation function. The adaptive frequency domain attention module and the high-efficiency spatial attention module are then integrated to form a spatial-frequency intermodulation attention module (SFIA).

[0060] The feature representation group (FRG) is formed by integrating a multi-scale spatial feature extraction block, a spatial frequency information distillation block, a spatial frequency intermodulation attention layer, and a convolutional layer through residual connections.

[0061] Six recursively connected feature representation groups are fused into a deep feature mapping module within the reconstruction model framework. This module is further combined with a shallow feature extraction module and an image reconstruction module to construct a spatial-frequency dual-domain representation learning image super-resolution reconstruction network model. The DIV2K dataset is used to train the parameter sets of the shallow feature extraction module (containing convolutional layers), the deep feature mapping module (containing multiple feature representation groups), and the reconstruction module (containing convolutional layers and pixel shuffling layers) within the spatial-frequency dual-domain representation learning image super-resolution reconstruction network model, respectively, to obtain the trained spatial-frequency dual-domain representation learning image super-resolution reconstruction network model. The low-resolution image to be reconstructed is input into the trained image super-resolution reconstruction model based on spatial-frequency dual-domain representation learning. Features are extracted from the image through convolutional layers in the shallow feature extraction module to obtain a shallow feature map. Features are then extracted from the shallow feature map through multiple recursively connected feature representation groups in the deep feature mapping module to obtain a deep feature map. Features in the deep feature map are extracted through convolutional layers and pixel shuffling layers in the reconstruction module. The feature elements in the deep feature map and the shallow feature map are summed, and pixel reconstruction is performed to obtain the reconstructed high-resolution image.

[0062] The image super-resolution reconstruction method and apparatus based on spatial-frequency dual-domain representation learning provided in this invention have the following advantages compared with the prior art: This invention addresses the inherent limitations of single architecture (CNN / Transformer) and single-domain representation in existing SISR methods by constructing a spatial-frequency dual-domain collaborative learning framework. This framework effectively compensates for the representational shortcomings of traditional spatial methods. On the one hand, by combining multi-scale feature extraction in the spatial domain with global structure modeling in the frequency domain, it retains the CNN's ability to accurately capture high-frequency details such as edges and textures. On the other hand, it optimizes the consistency of the overall image structure by leveraging the global receptive field in the frequency domain, solving the global fragmentation problem caused by the local receptive field of CNN. At the same time, it avoids the inflexibility of high-frequency detail extraction in the pure Transformer architecture.

[0063] To address the need for multi-scale representation of spatial details, this invention designs a multi-scale spatial feature extraction block (MSFEB). By constructing differentiated receptive fields through four convolutional networks of different sizes, it can simultaneously capture coarse-grained structural and fine-grained texture information. Combined with the feature refinement capabilities of the convolutional feedforward network, it significantly enhances the network's accuracy in depicting details at different scales. Compared with traditional single-scale spatial modules, the reconstruction effect of complex texture images is more in line with real-world scenarios.

[0064] In balancing efficient modeling and lightweight design of spatial-frequency features, the SFIDB (Spatial-Frequency Information Distillation Block) of this invention accurately captures long-range spatial dependencies through multi-head self-attention via a window to optimize low-frequency global representation. At the same time, it uses an efficient distillation structure to refine high-frequency information from global to local. This avoids the high computational overhead of full-scale self-attention and solves the problem that traditional frequency domain methods have difficulty associating high-frequency spatial locations. While ensuring feature representation capabilities, it significantly reduces the number of model parameters and computational costs, making it more suitable for resource-constrained scenarios such as terminals.

[0065] To address the inefficient fusion of dual-domain features, this invention proposes Spatial Frequency Intermodulation Attention (SFIA). SFIA achieves adaptive modulation of spectral spatial information through a learnable frequency domain filter, while simultaneously guiding the generation of spatial attention with high-frequency priors. This enables deep interaction between spatial and frequency dual-domain features, rather than simple splicing or weighted fusion, effectively improving the utilization rate of dual-domain features. As a result, the reconstructed image possesses both clear high-frequency texture and maintains the coherence of the global structure.

[0066] In summary, while achieving a lightweight architecture design, this invention balances the detail accuracy, global consistency, and computational efficiency of image super-resolution reconstruction through collaborative representation and efficient fusion of spatial and frequency domains. Compared with existing technologies, it significantly improves the overall quality of reconstructed images while reducing computational costs, making it more suitable for super-resolution requirements in real-world scenarios.

[0067] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. An image super-resolution reconstruction method based on spatial-frequency dual-domain representation learning, characterized in that, Includes the following steps: Acquire a low-resolution natural image to be reconstructed; An image super-resolution reconstruction model is constructed; the image super-resolution reconstruction model includes a shallow image feature extraction module, a deep image feature extraction module, and an image reconstruction module; the deep image feature extraction module includes six cascaded feature representation groups, each of which includes a multi-scale spatial domain feature extraction block, a spatial frequency information distillation block, a spatial frequency intermodulation attention module, and a standard convolution that are connected in sequence; The image shallow feature extraction module extracts shallow features from the low-resolution natural image to be reconstructed, and obtains a shallow feature map. The multi-scale spatial feature extraction block captures the coarse-grained structure and fine-grained texture information of the shallow feature map to refine its features. The spatial long-range spatial dependence of the refined shallow feature map is captured by the spatial frequency information distillation block to obtain high-frequency information refinement features from global to local; the spatial frequency intermodulation attention module adaptively modulates the spectral spatial information of the high-frequency information refinement features and high-frequency priors to guide spatial attention generation to obtain deep interactive dual-domain features. The deep feature map is obtained by convolving the dual-domain features of the deep interaction using the standard convolution. The image reconstruction module adds the deep feature map to the low-resolution natural image to obtain the reconstructed high-resolution image.

2. The image super-resolution reconstruction method based on spatial-frequency dual-domain representation learning as described in claim 1, characterized in that, The process of image super-resolution reconstruction using the aforementioned image super-resolution reconstruction model specifically includes: Low-resolution natural image to be reconstructed Through the shallow feature extraction module Convolution is used to extract shallow features, resulting in a shallow feature map. ; The shallow features Features are extracted and mapped using the six feature representation groups of the deep feature mapping module to obtain a deep feature map. ; The deep features By rebuilding the module Standard convolution and low-resolution images After adding the pixels, the pixels are shuffled to obtain the reconstructed super-resolution image. ; The shallow feature extraction module is represented as follows: The deep feature mapping module is represented as follows: The reconstruction module is represented as follows: in, Represented as a low-resolution image; The model uses the extracted shallow image features as input to the deep feature extraction module; express Standard convolution; Indicates the first A feature characterization group; Indicates the first The output characteristics of each FRG; This refers to the number of FRGs used; The deep features are obtained by deep feature mapping; The reconstructed super-resolution image; Shuffle the pixels.

3. The image super-resolution reconstruction method based on spatial-frequency dual-domain representation learning as described in claim 1, characterized in that, The multi-scale spatial feature extraction block consists of a multi-scale feature extraction module and a convolutional feedforward network. The multi-scale feature extraction module consists of a small-scale convolutional layer, an equivalent medium-scale convolutional layer, and two orthogonal large-scale strip convolutional layers; the small-scale convolutional layer is used to extract local detail texture features of the shallow feature map; The equivalent mesoscale convolutional layer consists of cascaded depthwise convolution, depthwise dilated convolution, and pointwise convolution, used to capture local structural information of shallow feature maps; the orthogonal large-scale strip convolutional layer consists of two depthwise separable strip convolutions, used to extract large-scale features of shallow feature maps. The convolutional feedforward network consists of three convolutional layers and two GELU activation functions; it is used to refine the multi-scale features extracted by the multi-scale feature extraction module and to enhance the effective multi-scale feature information through nonlinear activation functions.

4. The image super-resolution reconstruction method based on spatial-frequency dual-domain representation learning as described in claim 3, characterized in that, The feature extraction process of the multi-scale spatial feature extraction block specifically includes: Feature map output by the shallow feature extraction module After layer normalization (LN), the feature map is processed along the channel dimension. Divided into four groups of features; The separated features are input into different parallel branches, and feature extraction is performed through convolution operations respectively; The outputs of each branch are concatenated, and then channel attention weighting is applied. Input the weighted features into Smoothing and residual connections are performed in standard convolutions to obtain multi-scale features; The multi-scale features are refined and extracted using a convolutional feedforward network, and the effective multi-scale feature information is enhanced by a nonlinear activation function to obtain multi-scale spatial features.

5. The image super-resolution reconstruction method based on spatial-frequency dual-domain representation learning as described in claim 1, characterized in that, The space frequency information distillation block consists of multiple It consists of a standard convolutional layer, a depthwise separable convolutional layer, a window-based multi-head self-attention mechanism, and three efficient feature refinement blocks; The window-based multi-head self-attention mechanism consists of multiple linear projection transformations, window splitting, window merging operations, and a convolutional feedforward network. The efficient feature refinement block consists of two convolutional layers.

6. The image super-resolution reconstruction method based on spatial-frequency dual-domain representation learning as described in claim 5, characterized in that, The feature extraction process of the space frequency information distillation block specifically includes: By dividing the multi-scale feature map extracted by the multi-scale spatial feature extraction block into non-overlapping local windows, the feature vector of each window is transformed into the form of multi-head attention through tensor dimension rearrangement; The feature vector is mapped into query, key, and value vectors through three linear projection transformations. Each attention head maps the input vector to the subspace represented by the attention head and performs self-attention computation; The output vectors of all attention heads are concatenated and aggregated along the channel dimension, and a linear projection mapping is used to obtain an output containing local dependency information, resulting in a local feature map; the size of the local feature map is then converted to the size of the original feature map by window merging. pass The depthwise separable convolution extracts the coarse-grained high-frequency texture of the local feature map, and then... Standard convolution and GELU activation function are used to fuse channel information to obtain a coarse-grained feature map; and through... Standard convolution and sigmoid activation function are used to extract fine-grained information from the local feature map pixel by pixel to obtain a fine-grained feature map; The fine-grained feature map is weighted onto the coarse-grained feature map to obtain high-frequency detail information features fused across multiple scales.

7. The image super-resolution reconstruction method based on spatial-frequency dual-domain representation learning as described in claim 1, characterized in that, The spatial frequency intermodulation attention module includes an adaptive frequency domain attention module and an efficient spatial attention module; The adaptive frequency domain attention module consists of three It consists of a standard convolution, two LeakyReLU activation functions, and a learnable frequency domain filter; The high-efficiency spatial attention module is composed of It consists of standard convolution, stride convolution, depthwise separable convolution groups, average pooling layers, deconvolution upsampling, and the GELU activation function.

8. The image super-resolution reconstruction method based on spatial-frequency dual-domain representation learning as described in claim 7, characterized in that, The feature extraction process of the adaptive frequency domain attention module specifically includes: Based on the high-frequency detail information features extracted from the space-frequency information distillation block, the spectra of multiple image blocks are obtained through window splitting and fast Fourier transform. The real and imaginary parts of the spectrum are separated and processed by three... The bottleneck network, consisting of standard convolution and two LeakyReLU activation functions, fuses channel information to obtain spectral spatial information. The spectral spatial information is adaptively modulated using a learnable frequency domain filter, and a frequency domain attention map is obtained through frequency domain jump connections, inverse fast Fourier transform, and window merging.

9. The image super-resolution reconstruction method based on spatial-frequency dual-domain representation learning as described in claim 7, characterized in that, The feature extraction process of the efficient spatial attention module specifically includes: pass Standard convolution reduces the dimensionality of high-frequency detail features extracted from the spatial frequency information distillation block to one-quarter of the channels; and uses a stride of 2. Depthwise separable convolution completes the first downsampling; based on the downsampled features, lightweight local thinning and average pooling are performed to obtain a low-scale representation of one-quarter of the image size; based on the low-scale representation features, through... A depthwise separable convolutional group is used to perform context aggregation to obtain low-resolution features; and the low-resolution features are mapped back to the original scale through a transposed convolution with a stride of 4. The structural information of high-frequency detail information features extracted by local convolution is obtained by extracting the spatial frequency information distillation block; and based on the explicit computation of high-frequency prior information structure, rapidly changing texture and gradient information are separated from the high-frequency detail information features to obtain high-frequency enhanced features. Based on the aforementioned high-frequency enhancement features and low-resolution features, after fusion in the high-resolution space according to the channel stitching method, through... Standard convolution and sigmoid activation function dynamically generate spatial attention feature maps.

Images