Lightweight image super-resolution method, system and device based on frequency domain and medium

Abstract

The invention discloses a lightweight image super-resolution method, a lightweight image super-resolution system, lightweight image super-resolution equipment and a lightweight image super-resolution medium based on a frequency domain, which are corresponding schemes. The feature modeling of the global receptive field is realized under the condition that the calculation complexity is not obviously increased; moreover, through a frequency domain high and low frequency decomposition and high-frequency prior guidance mechanism, the expression ability of the model to high-frequency detail information of the image is enhanced; in addition, a frequency enhancement loss function is also designed, the model is guided from an optimization target level to focus on learning of a high-frequency region, and the reconstruction quality is improved. On the whole, the method provided by the invention can realize effective modeling of global context information and high-frequency detail information while ensuring the calculation efficiency, thereby improving the quality of a reconstructed image.

Description

Technical Field

[0001] This invention relates to the field of image super-resolution technology, and in particular to a lightweight image super-resolution method, system, device and medium based on the frequency domain. Background Technology

[0002] Single Image Super-Resolution (SISR) is a typical low-level vision problem that aims to reconstruct a corresponding high-resolution image from a degraded low-resolution image. This technology has wide applications in practical scenarios such as security surveillance imaging and medical image processing.

[0003] Since recovering high-resolution images from low-resolution images is mathematically an ill-posed inverse problem, existing research typically relies on prior modeling to constrain the solution space. In recent years, deep learning-based methods have been widely applied in this field, including convolutional neural network-based super-resolution methods, visual Transformer-based methods, and related schemes based on state-space models. Among these, the Transformer structure based on self-attention mechanisms has gradually become a key focus of single-image super-resolution research due to its ability to model long-range dependencies.

[0004] Existing research, HAT "Activating more pixels in image super-resolution transformer, CVPR 2023," demonstrates that introducing more global contextual information helps improve the quality of reconstructed images. However, the computational complexity of self-attention mechanisms increases quadratically with input resolution, leading to significant computational and storage overhead in high-resolution image processing scenarios, thus limiting their deployment in practical applications.

[0005] To reduce computational complexity, some methods, such as SwinIR, ELAN's "Swinir: Image restoration using swin transformer, ICCVW2021" and "Efficient long-range attention network for image super-resolution, ECCV2022," employ a self-attention mechanism based on local windows. This mechanism divides the image into multiple local regions to reduce computation. However, such methods disrupt the spatial continuity of the image and require additional operations such as window translation and feature rearrangement, which weakens the model's ability to model global contextual information to some extent.

[0006] On the other hand, existing research, "Feature modulation transformer: Cross-refinement of global representation via high-frequency prior for image super-resolution," ICCV 2023, also found that the self-attention mechanism tends to model low-frequency information during feature aggregation, behaving somewhat like a low-pass filter. This can easily lead to over-smoothing of features, thus weakening the ability to express high-frequency details (such as edges and textures). This characteristic means that most existing Transformer-based super-resolution methods mainly focus on spatial domain feature interactions, failing to fully utilize the potential advantages of frequency domain modeling in expressing high-frequency information, thereby limiting further improvements in reconstruction performance.

[0007] In general, existing methods have some common problems: (1) It is difficult to balance performance and computational efficiency.

[0008] To improve reconstruction accuracy, existing methods typically rely on expanding the receptive field or introducing complex self-attention mechanisms to model global contextual information. However, the computational complexity increases significantly with the input resolution, making it difficult to meet the requirements for computational efficiency and resource consumption in practical applications. Furthermore, local modeling or windowing strategies adopted to reduce computational overhead weaken the ability to model global information to some extent, resulting in limited reconstruction performance.

[0009] (2) Insufficient ability to model high-frequency information.

[0010] Most existing spatial domain-based modeling methods focus more on the expression of low-frequency information during feature aggregation. Their behavior is similar to low-pass filtering to some extent, which can easily lead to over-smoothing of features. This weakens the ability to model high-frequency details such as image edges and textures, and thus affects the performance of super-resolution reconstruction results in terms of detail fidelity.

[0011] In view of this, the present invention is hereby proposed. Summary of the Invention

[0012] The purpose of this invention is to provide a lightweight image super-resolution method, system, device, and medium based on the frequency domain, which can effectively model global context information and high-frequency detail information while ensuring computational efficiency, thereby improving the quality of reconstructed images.

[0013] The objective of this invention is achieved through the following technical solution: A lightweight image super-resolution method based on the frequency domain includes: Collect a training set, each sample consisting of two images with different resolutions, where the image with lower resolution is called the low-resolution image and the other is called the true high-resolution image; A frequency-domain adaptive super-resolution network is constructed and trained using a training set. The frequency-domain adaptive super-resolution network includes a feature extractor and a super-resolution image decoder. A low-resolution image is input, and the feature extractor performs initial feature extraction, attention modeling, and enhancement to obtain enhanced features. The super-resolution image decoder uses the initial and enhanced features to output a decoded image, called the predicted high-resolution image. A high-frequency mask is generated using the gradient maps of the predicted high-resolution image and the real high-resolution image, and a frequency enhancement loss is constructed. The frequency-domain adaptive super-resolution network is trained based on the frequency enhancement loss. The feature extractor includes a frequency-domain channel attention module and a frequency-domain spatial attention module to achieve interaction between the spatial and channel dimensions. Furthermore, the frequency-domain channel attention module and the frequency-domain spatial attention module extract high-frequency priors based on high- and low-frequency decomposition in the frequency domain, respectively, to guide the attention modeling process of the feature extractor. The image to be super-resolution is input into the trained frequency domain adaptive super-resolution network to obtain the predicted high-resolution image.

[0014] A lightweight image super-resolution system based on the frequency domain, for implementing the aforementioned method, includes: The training set collection unit is used to collect the training set. Each sample includes two images with different resolutions, where the image with lower resolution is called the low-resolution image and the other is called the true high-resolution image. A network construction and training unit is used to construct a frequency-domain adaptive super-resolution network and train it using a training set. The frequency-domain adaptive super-resolution network includes a feature extractor and a super-resolution image decoder. A low-resolution image is input, and the feature extractor performs initial feature extraction, attention modeling, and enhancement to obtain enhanced features. The super-resolution image decoder uses the initial and enhanced features to output a decoded image, called the predicted high-resolution image. A high-frequency mask is generated using the gradient maps of the predicted high-resolution image and the real high-resolution image, and a frequency enhancement loss is constructed. The frequency-domain adaptive super-resolution network is trained based on the frequency enhancement loss. The feature extractor includes a frequency-domain channel attention module and a frequency-domain spatial attention module to achieve interaction between the spatial and channel dimensions. The frequency-domain channel attention module and the frequency-domain spatial attention module extract high-frequency priors based on high- and low-frequency decomposition in the frequency domain, respectively, to guide the attention modeling process of the feature extractor. The super-resolution unit is used to input the image to be super-resolution into the trained frequency domain adaptive super-resolution network to obtain a predicted high-resolution image.

[0015] A processing device includes: one or more processors; and a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0016] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0017] As can be seen from the technical solutions provided by the present invention: (1) Efficient super-resolution is achieved based on frequency domain modeling, and feature modeling of the global receptive field is realized without significantly increasing computational complexity; (2) The ability of the model to express high-frequency details of the image is enhanced through frequency domain high-low frequency decomposition and high-frequency prior guidance mechanism; (3) A frequency enhancement loss function is designed to guide the model to focus on learning high-frequency regions from the optimization target level, thereby improving the reconstruction quality. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart of a lightweight image super-resolution method based on the frequency domain is provided for an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of the structure of the frequency domain adaptive super-resolution network provided in an embodiment of the present invention.

[0021] Figure 3 This is a schematic diagram of the frequency domain channel attention module provided in an embodiment of the present invention.

[0022] Figure 4 This is a schematic diagram of the frequency domain spatial attention module provided in an embodiment of the present invention.

[0023] Figure 5 This is a schematic diagram of the high-frequency prior injection mechanism provided in an embodiment of the present invention.

[0024] Figure 6 This is a schematic diagram illustrating the calculation principle of frequency enhancement loss provided in an embodiment of the present invention.

[0025] Figure 7 This is a schematic diagram illustrating the analysis of gradient value statistics of the entire training dataset provided in an embodiment of the present invention.

[0026] Figure 8 This diagram illustrates the effectiveness analysis results of FCA, FSA, and frequency enhancement loss function during the training process, as provided in this embodiment of the invention.

[0027] Figure 9 A schematic diagram illustrating the effective receptive field characteristics analysis of the FANSR proposed in this invention and two existing advanced methods, ELAN and OmniSR, provided for embodiments of this invention.

[0028] Figure 10 This is a schematic diagram illustrating the visual quality comparison results provided in an embodiment of the present invention.

[0029] Figure 11 This is a schematic diagram of a lightweight image super-resolution system based on the frequency domain, provided as an embodiment of the present invention.

[0030] Figure 12 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0032] First, the following explanations are provided for the terms that may be used in this article: The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0033] The following provides a detailed description of a lightweight image super-resolution method, system, device, and medium based on the frequency domain provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they are performed according to conventional conditions in the art or conditions recommended by the manufacturer. Where the manufacturers of reagents or instruments used in the embodiments of this invention are not specified, they are all conventional products that can be purchased commercially.

[0034] Example 1 This invention provides a lightweight image super-resolution method based on the frequency domain, such as... Figure 1 As shown, it mainly includes the following steps: Step 1: Collect the training set.

[0035] In this embodiment of the invention, the training set contains multiple samples, each of which is an image pair, namely: two images with different resolutions. The content of the two images is the same, and the main difference is that the resolutions of the two images are different. The image with the lower resolution is called the low-resolution image, and the other is called the true high-resolution image.

[0036] For example, the high-resolution image has a size of 1024*1024, and the low-resolution image has a size of 256*256.

[0037] Step 2: Construct a frequency domain adaptive super-resolution network and train it using the training set.

[0038] In this embodiment of the invention, the frequency-domain adaptive super-resolution network includes: a feature extractor and a super-resolution image decoder; a low-resolution image is input, and the feature extractor performs initial feature extraction and attention modeling and enhancement to obtain enhanced features. The super-resolution image decoder uses the initial features and enhanced features to output a decoded image, called the predicted high-resolution image; a high-frequency mask is generated using the gradient map of the predicted high-resolution image and the real high-resolution image, and a frequency enhancement loss is constructed. The frequency-domain adaptive super-resolution network is trained based on the frequency enhancement loss; wherein, the feature extractor is provided with a frequency-domain channel attention module and a frequency-domain spatial attention module to realize the interaction between the spatial dimension and the channel dimension, and high-frequency priors are extracted in the frequency-domain channel attention module and the frequency-domain spatial attention module respectively based on the frequency-domain high-low frequency decomposition to guide the attention modeling process of the feature extractor.

[0039] (1) In this embodiment of the invention, the feature extractor includes: a shallow feature extractor and a deep feature extractor; the shallow feature extractor is used to extract initial features; the deep feature extractor is used to perform attention modeling and enhancement on the initial features to obtain enhanced features; wherein, the deep feature extractor includes multiple cascaded frequency domain adaptive feature groups, each frequency domain adaptive feature group includes multiple frequency domain adaptive feature blocks connected in sequence as well as terminal convolutional layers and activation function layers, each frequency domain adaptive feature block includes a frequency domain channel attention module, a channel dimension feature self-attention mechanism processing block, a frequency domain spatial attention module and a spatial attention module arranged in sequence. The dimensional feature self-attention mechanism processing block; the channel dimension feature self-attention mechanism processing block and the spatial dimension feature self-attention mechanism processing block respectively use the high-frequency priors output by the frequency domain channel attention module and the frequency domain spatial attention module to adjust their own input features and fuse them with their own attention output; in each frequency domain adaptive feature group, the output of the spatial dimension feature self-attention mechanism processing block in the last frequency domain adaptive feature block is processed by the terminal convolutional layer and activation function layer, and then added to the input of the frequency domain adaptive feature group as the output of the frequency domain adaptive feature group. The output of the last frequency domain adaptive feature group is the enhanced feature.

[0040] (2) In this embodiment of the invention, the frequency domain channel attention module is used to explicitly model the global dependencies between different channels by performing Fourier transform operations along the channel dimension; the input features are denoted as... Applying a Fourier transform to it yields the channel frequency domain vector. Then, matrix multiplication is performed on the query and key in the frequency domain to obtain a global interactive representation in the channel dimension; an inverse transformation is performed on the global interactive representation to obtain the features. Then, sigmoid activation and average pooling operations are applied sequentially to generate a channel adaptive mask. The channel adaptive mask modulates the values ​​in the spatial domain to obtain the output features. ; The average pooling operation uses high-order global average pooling, which divides the features... Convert to a global vector This global vector This is the channel adaptive mask, and the entire process is represented as follows: ; ; ; Where H and W represent the spatial height and width of the image, respectively. Indicates the feature Features obtained by global average pooling. Features The value at spatial location (h, w) and channel index c; N represents the maximum order of the global statistical feature. k represents the order index of the higher-order global statistical feature. Representation of features The k-th order global statistical feature (higher-order central moment feature) is used to characterize the higher-order statistical information of the feature relative to its mean distribution in the global space.

[0041] (3) In this embodiment of the invention, the frequency domain spatial attention module is used to realize global information modeling in the spatial dimension and fuse it with local spatial features; the frequency domain spatial attention module includes a frequency branch and a spatial branch, and the input features are denoted as... Spatial branching uses deep convolutional layers to process features. Processing is performed to obtain local spatial features; in the frequency branch, the features are... The frequency representation is obtained by performing a two-dimensional Fourier transform. And add location encoding Then, through complex convolutional layers and corresponding activation functions, and then through two-dimensional inverse Fourier transform, a global attention map, i.e., global information, is generated. The global attention map is then fused with local spatial features to obtain the output features of the frequency domain spatial attention module. .

[0042] (4) In this embodiment of the invention, the extraction of high-frequency priors based on high- and low-frequency decomposition in the frequency domain channel attention module and the frequency domain spatial attention module respectively includes: Both the frequency domain channel attention module and the frequency domain spatial attention module obtain the frequency domain representation through corresponding Fourier transforms. d∈{c,s}, where c refers to the frequency domain channel attention module and s refers to the frequency domain spatial attention module. Frequency domain representation The frequency domain vectors are obtained by the channel attention module through Fourier transform operations. Frequency domain representation The frequency representation is obtained through two-dimensional Fourier transform in the frequency domain spatial attention module. ; By using a preset high-frequency mask and low-frequency mask The high-frequency mask, in particular... With low-frequency mask All masks are generated in the frequency domain space based on the frequency distribution location. Specifically, after performing a Fourier transform on the input features, the frequency domain representation corresponds to the zero-frequency (low-frequency) component at the frequency domain center position, and the distance of each frequency component relative to the frequency domain center reflects its corresponding frequency level. In this invention, based on the Euclidean distance metric from each frequency component in the frequency domain to the frequency domain center (low-frequency zero point), the frequency domain space is divided into low-frequency and high-frequency regions: when the distance of a frequency component to the frequency domain center is less than or equal to a preset threshold, it is classified as a low-frequency region, and a corresponding low-frequency mask is generated. When the distance is greater than the threshold, it is divided into high-frequency regions, and a corresponding high-frequency mask is generated. For example, this threshold can be set to... For frequency domain representation Decompose to obtain high-frequency components and low frequency components ; For frequency domain representation By analyzing the corresponding high-frequency components Low-frequency components Perform one-dimensional inverse Fourier transform, normalization mapping, and differentiable discretization to generate channel priors. ; For frequency domain representation By analyzing the corresponding high-frequency components Low-frequency components Perform two-dimensional inverse Fourier transform, normalization mapping, and differentiable discretization to generate spatial priors. ; The channel prior With spatial a priori All of these belong to the aforementioned high-frequency priors.

[0043] (5) In this embodiment of the invention, the channel dimension feature self-attention mechanism processing block and the spatial dimension feature self-attention mechanism processing block respectively use the high-frequency priors output by the frequency domain channel attention module and the frequency domain spatial attention module to adjust their own input features and fuse them with their own attention output, including: The channel-dimensional feature self-attention mechanism processes the high-frequency priors of the block input frequency domain channel attention module output, i.e., the channel priors. The spatial dimension feature self-attention mechanism processes the high-frequency priors of the frequency domain spatial attention module output from the input block, i.e., the spatial priors. ; Memorizing high-frequency priors Representative channel prior or spatial prior , d∈{c,s}; The attention outputs of the channel dimension feature self-attention mechanism processing block and the spatial dimension feature self-attention mechanism processing block are represented as: Where Q, K, and V represent the query, key, and value information in the attention mechanism, Let T be the dimension of the key information, and T be the transpose symbol; then the fusion process can be represented as: ; in, and There are two hyperparameters. The input features represent the self-attention mechanism processing blocks for channel-dimensional features or spatial-dimensional features. The output features represent the fusion of channel dimension feature self-attention mechanism processing blocks or spatial dimension feature self-attention mechanism processing blocks; It is a normalized exponential function. This represents the weight matrix of the output projection. This indicates element-wise multiplication.

[0044] (6) In this embodiment of the invention, the construction of frequency enhancement loss and the training of the frequency domain adaptive super-resolution network based on the frequency enhancement loss include: Constructing frequency enhancement loss , represented as: ; Where H and W represent the spatial height and width of the image, respectively. For true high-resolution images The value at spatial location (h, w) For the prediction of high-resolution images The value at spatial location (h, w) The value of the high-frequency mask at spatial location (h, w); This represents the L1 distance, which can be considered as the absolute value symbol. High-frequency mask calculation methods include: calculating the predicted high-resolution image separately. Compared to true high-resolution images The gradient map is obtained; for each spatial position in each gradient map, the normalized gradient norm is calculated and compared with the set hyperparameter θ. If it is less than θ, the loss weight value is assigned to 1; if it is greater than or equal to θ, the loss weight value is assigned to γ. Finally, the loss weight map corresponding to the two gradient maps is obtained. The two loss weight maps are fused to obtain the high frequency mask. Construct the final loss function Loss based on frequency augmentation loss: ; ; in, For hyperparameters, For L1 loss, Indicates the prediction of high-resolution images Compared to true high-resolution images The difference in gradients; The frequency domain adaptive super-resolution network is trained using the final loss function Loss.

[0045] Step 3: Achieve image super-resolution based on the trained frequency domain adaptive super-resolution network.

[0046] In this embodiment of the invention, the image to be super-resolution is input into the trained frequency domain adaptive super-resolution network to obtain a predicted high-resolution image. The processing involved here is the same as that in the training process, so it will not be described in detail.

[0047] The solution provided in this embodiment of the invention efficiently achieves single-image super-resolution reconstruction from a frequency domain perspective. While ensuring computational efficiency, it effectively models global contextual information and high-frequency detail information, thereby improving the quality of the reconstructed image. The main advantages include: (1) A frequency domain adaptive super-resolution network is proposed, which realizes feature modeling of the global receptive field without significantly increasing the computational complexity.

[0048] (2) By using frequency domain high and low frequency decomposition and high frequency prior guidance mechanism, the model’s ability to express high frequency detail information of the image is enhanced.

[0049] (3) A frequency enhancement loss function was designed to guide the model to focus on learning high-frequency regions from the perspective of optimization objectives, thereby improving the reconstruction quality.

[0050] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0051] I. Overall Overview of the Plan

[0052] Based on the classic SR (Super-Resolution) setting, the goal of this invention is to utilize low-resolution images. Reconstruct a high-resolution image Where s is the scale, Here, H and W represent the image height and width, respectively. This invention considers both reconstruction performance and computational efficiency, proposing a novel, highly efficient super-resolution reconstruction method—FANSR (Frequency Domain Adaptive Super-Resolution Network). The method of this invention mainly includes the following three key innovations: 1) This invention absorbs the design essence of the general transformer architecture and employs two innovative frequency domain token mixers to promote rich interactions in spatial and channel dimensions, providing an alternative to the self-attention mechanism. It can achieve a globally effective receptive field with instance-adaptive weights while maintaining low complexity; 2) By explicitly decomposing features into high and low frequencies in the frequency domain and introducing high-frequency priors into the attention modeling process, the model's ability to express high-frequency details such as image edges and textures is effectively enhanced; 3) A novel frequency-enhanced loss function encourages the model to focus more on learning finer details in the high-frequency regions of the image, rather than treating image regions of different frequency bands equally.

[0053] like Figure 2 As shown, the main structure of the Frequency Domain Adaptive Super-Resolution Network (FANSR) mainly includes the following three parts: (1) Shallow feature extractor It is used to extract initial features (basic features) from the input image. For example, it can be a 3×3 convolutional layer.

[0054] (2) Deep Feature Extractor It is used to further model and enhance the initial features.

[0055] (3) Super-resolution image decoder It is used to generate the final super-resolution reconstructed image (the predicted high-resolution image), which consists of a series of 3×3 convolutional layers and pixel shuffle layers.

[0056] Specifically, given an input image The shallow feature extractor first... Mapping to the feature space to obtain initial features The process can be represented as follows: ; Where C represents the number of channels in the intermediate feature, and H and W represent the spatial height and width of the feature map, respectively.

[0057] Subsequently, in the deep feature extraction stage, a deep feature extractor is used. For the initial features Further processing is performed to output enhanced features. The process can be represented as follows: ; in, include A series of cascaded frequency-domain adaptive feature groups (FAGroups) and convolutional layers. In a preferred embodiment, a classic skip connection structure can be used to combine the initial features... With enhanced features Combined, and then input into the final super-resolution image decoder. To obtain the predicted high-resolution image .

[0058] II. Detailed introduction of the plan.

[0059] This section mainly introduces the frequency domain adaptive feature group architecture, frequency domain channel attention module (FCA), frequency domain spatial attention module (FSA), high frequency prior extraction and injection mechanism, and frequency enhancement loss.

[0060] 1. Frequency domain adaptive feature group architecture.

[0061] Considering the importance of global feature interaction and local feature extraction in super-resolution reconstruction tasks, this invention introduces a frequency-domain adaptive feature group (FAGroup) architecture in the network structure shown in Figure 2. The FAGroup integrates cross-channel information and spatial dimension information in the frequency domain, effectively expanding the effective receptive field of features while ensuring computational efficiency, thereby enhancing feature representation capabilities.

[0062] In one embodiment, each frequency-domain adaptive feature group employs a "Frequency Domain Channel Attention Module (FCA)" and a "Frequency Domain Spatial Attention Module (FSA)" as a token mixer to replace the self-attention module in the traditional Transformer structure, thereby achieving full interaction between the spatial and channel dimensions. The frequency domain channel attention module and frequency domain spatial attention module can achieve feature interaction with a global receptive field with low computational complexity. The specific structure and working mechanism of these two modules will be described in detail in subsequent chapters.

[0063] In one embodiment, consistent with existing related work, this invention introduces a gated-Dconv feed-forward network (GDFN) as a feed-forward network (FFN) module within the FAGroup to enhance nonlinear expressive power. Based on this, this invention arranges an alternating stacked processing sequence within the FAGroup, including FCA, channel-dimensional feature self-attention mechanism (C-MDTA) processing block, FSA, and spatial-dimensional feature self-attention mechanism (W-SA) processing block. Multiple of these processing units are cascaded in a predetermined order to form a frequency-domain adaptive feature block (FABlock). In one embodiment... A series of consecutive combinations of [FCAB, C-MDTAB, FSAB, W-SAB] together constitute a FABlock (frequency domain adaptive feature block).

[0064] Wherein, B represents block, and the block represents a basic feature processing unit built on the Transformer framework, whose structure corresponds to the standard Transformer Block in self-attention networks. In one embodiment, each Block includes at least the following components: 1) a Token-mixer for interactive modeling of features in a specific dimension, wherein the Token-mixer can be any one of Frequency-Based Channel Attention (FCA), Frequency-Based Spatial Attention (FSA), Channel-Dimensional Feature Self-Attention (C-MDTA), or Spatial-Dimensional Feature Self-Attention (W-SA), and the corresponding block can be called FCAB, FSAB, C-MDTAB, or W-SAB; 2) a Layer Normalization (LN) module connected to the Token-mixer, which fuses the output of the Token-mixer with the input features through residual connections; 3) a Gated-DconvFeed-Forward Network (GDFN) for nonlinear mapping and channel expansion of the features processed by the Token-mixer; 4) a Layer Normalization (LN) module connected to the GDFN, which fuses the output of the GDFN with its input features through residual connections. Since the specific calculation process involved in this part can be referred to conventional techniques, it will not be elaborated here.

[0065] For a FABlock, its feature update process can be represented as: ; ; ; ; in, This represents the features input to FABlock. This represents intermediate features. and These represent the channel prior and spatial prior information obtained from frequency domain decomposition, respectively. To provide a clearer picture of the overall process of this invention, the processing steps of LN and GDFN involved in each block are omitted in the following description.

[0066] In one embodiment, after feature processing of multiple FABlocks is completed, a convolutional layer and a nonlinear activation function are set at the end of the FAGroup to complete feature aggregation. The overall calculation process can be represented as follows: ; in, The last one in FAGroup (i.e., the first one) The output of FABlock (number of blocks).

[0067] 2. Frequency Domain Channel Attention Module (FCA).

[0068] This invention proposes a Frequency Domain Channel Attention (FCA) module, which aims to explicitly model the global dependencies between different channels by performing Fourier transform operations along the channel dimension, thereby promoting cross-channel feature interaction. This design is particularly necessary and offers significant advantages for extracting global channel information.

[0069] First, we perform a one-dimensional Fourier transform on the input features along the channel dimension using a one-dimensional Fourier transform. (One-dimensional Fourier transform) The explanation is as follows: For a one-dimensional discrete signal of length N... Its Discrete Fourier Transform (DFT) is defined as follows: .

[0070] The corresponding frequency domain representation is a complex vector. The frequency domain signal can be reconstructed back to the original spatial domain using the inverse discrete Fourier transform (IDFT). .

[0071] Specifically, given a feature The present invention applies a one-dimensional Fourier transform to it along the channel dimension. The channel frequency domain vector is obtained. : .

[0072] To further improve computational efficiency, this invention employs the Fast Fourier Transform (FFT) instead of the standard Discrete Fourier Transform (DFT) and performs matrix multiplication on the query and key in the frequency domain to obtain a global interactive representation in the channel dimension. Subsequently, this interactive representation is mapped back to the spatial domain through an inverse transform to obtain the features. Then, Sigmoid activation and average pooling (avg) operations are applied sequentially to generate a channel-adaptive mask. Ultimately, the mask. Modulating the value branch in the spatial domain can be represented as follows: ; in, Indicates Sigmoid activation. , are the weight matrices for Key, Value, and output projection, respectively. This is the activation function.

[0073] In the above process, the query is... Through the key matrix The key is obtained by projection, and then the value matrix is ​​used to obtain the key. The value is obtained by projection.

[0074] avg indicates that this invention uses high-order global average pooling (H-GP) to pool features. Convert to a global vector To enhance representational capabilities, this is driven by color moments. The integration of higher-order moments, incorporating global properties, offers several advantages; therefore, this invention introduces higher-order moments here. The reasons are as follows: First, higher-order moments contain more detailed information about the color distribution, such as skewness and kurtosis, allowing for a more comprehensive description of the global features of each channel. Furthermore, higher-order moments provide a robust description of the global feature distribution, making the features more resilient to changes in illumination, noise, and other disturbances, thereby enhancing the stability and robustness of the image retrieval system. It can be expressed as: ; ; .

[0075] The meanings of the symbols in the above formula have been explained in detail above, so they will not be repeated here. For example, N=2 can be set to strengthen the expression. The representational power of the global vector obtained here. This refers to the channel adaptive mask mentioned earlier. .

[0076] 3. Frequency Domain Spatial Attention Module (FSA).

[0077] This invention proposes a frequency-based spatial attention module (FSA) for global information modeling in the spatial dimension, which is combined with a local feature extraction mechanism to enhance feature representation capabilities while maintaining low computational complexity.

[0078] Specifically, given a feature The present invention uses two-dimensional FFT to obtain the corresponding frequency representation. ,Right now: We obtain, of which: ; in, Represents the frequency coordinates in Fourier space. From real components and imaginary components Each component is composed of C channels. Considering the conjugate symmetry property of the Fast Fourier Transform (FFT), By retaining only half of the spatial dimensions, we obtain .

[0079] First, to frequency characteristics Add position encoding This is a location encoding used to introduce spatial location information. Its calculation method is based on the normalized coordinates of the input features in the spatial dimension. Specifically, let the height and width of the input features in the spatial dimension be... and First, normalized one-dimensional coordinate sequences are generated along the vertical and horizontal directions respectively: .

[0080] Subsequently, the above one-dimensional coordinates are extended to two-dimensional space to obtain coordinate mappings representing the vertical and horizontal positions respectively: .

[0081] Then, map the vertical coordinates. Mapping with horizontal coordinates The data is stitched together along the channel dimension to form a two-dimensional position code. : .

[0082] Then and The corresponding frequency position fusion features are obtained by splicing. .

[0083] Based on the properties of iFFT (Inverse Fourier Transform), enhancing the feature representation in the frequency domain has a global impact on the original data in the spatial domain. Therefore, the frequency branch can easily introduce global information. To utilize this property, such as... Figure 4 As shown in part (a), this invention comprises two branches: a spatial branch and a frequency branch. In the frequency branch, the frequency domain representation is first extracted using a complex convolution (ℂ-Conv). Multilayer perceptrons (MLPs) incorporate complex convolutions and complex activation functions, cleverly utilizing the feature information of complex features to ensure more comprehensive processing of frequency domain data, such as... Figure 4 Part (b) shows an example of the complex convolution structure. Specifically, it illustrates the fusion of frequency-location features. and the complex kernel of ℂ-Conv Complex convolution can be represented as: ; in, Represents the imaginary unit. express The real part, express The imaginary part. Similarly... Denotes the real part of the complex convolution kernel. This represents the imaginary part of the complex convolution kernel.

[0084] Similarly, the activation ReLU function in FSA is also implemented in complex ReLU (denoted as ℂ-ReLU) to satisfy the Cauchy-Riemann equations as follows: .

[0085] Subsequently, this invention transforms the converted FFT representation back into the spatial domain to generate an attention graph. The spatial branching is accomplished using a simple 3×3 DW-Conv (depthwise convolution). Finally, this invention achieves information fusion of global and local information by multiplying the attention map with points of local spatial features. This process can be described as follows: ; ); ; in, It is a Sigmoid function. For element-wise multiplication, Linear is the linear mapping layer, and GeLU is the activation function. Intuitively, with proper training supervision, the FSA module can learn to adaptively modulate local features using frequency information. Because... Encapsulated Therefore, the FSA module of this invention can obtain a global receptive field by integrating pure convolution operations and FFT transformations, thus providing global information.

[0086] 4. High-frequency prior extraction and injection mechanism.

[0087] As mentioned earlier, in the spatial domain, high-frequency components typically correspond to edges, fine textures, and areas of abrupt brightness changes in an image. This type of information is the most difficult to accurately recover in super-resolution reconstruction tasks. Meanwhile, in the channel domain, high-frequency components reflect the significant differences between different channels, helping to enhance the diversity and discriminative power of channel features. Based on this understanding, this invention decouples features along both the spatial and channel dimensions in the frequency domain to extract high-frequency prior information. This high-frequency prior information is then injected into the subsequent attention modeling process, thereby enhancing the model's ability to express high-frequency detail information.

[0088] In one embodiment, frequency domain feature representations are first obtained from the corresponding frequency domain channel attention module (FCA) or frequency domain spatial attention module (FSA), and high-frequency and low-frequency information are extracted along the channel dimension or spatial dimension, respectively. Specifically, for the frequency domain representation... Where d∈{c,s} represent channel-dimensional frequency domain features or spatial-dimensional frequency domain features, respectively, through a preset high-frequency mask. and low-frequency mask The process of decomposing it can be represented as (i.e., the Filter part in the attached diagram): ; ; Here, ⊙ represents element-wise multiplication. and These represent the high-frequency and low-frequency components in the frequency domain characteristics, respectively.

[0089] For the frequency domain components of the channel dimension (in (representing high-frequency or low-frequency components), by performing a one-dimensional inverse Fourier transform on them. Normalization mapping and differentiable discretization are used to generate channel prior masks. The process can be represented as follows: .

[0090] The output of Gumbel-Softmax (abbreviated as gs) is a tensor in an approximate one-hot form, which retains the discrete selection property while still being differentiable, thus enabling the network to be jointly trained in an end-to-end manner.

[0091] Similarly, for the frequency domain components of the spatial dimension Through two-dimensional inverse Fourier transform It also combines Gumbel-Softmax operations to generate a spatial prior mask. The process can be represented as follows: ; Among them, spatial priors , used to characterize the importance weight of each spatial location.

[0092] In the above calculation process, high-frequency or low-frequency components can be selected as needed and substituted into the above formula to obtain the corresponding a priori mask.

[0093] In the subsequent attention modeling process, such as Figure 5 As shown, the extracted channel priors are... or spatial prior The input features are modulated as a mask to generate a high-frequency enhanced feature representation. Subsequently, this prior-modulated feature is fused with the output of the corresponding attention module to obtain the final output feature. The fusion process can be represented as follows: .

[0094] Through the aforementioned frequency domain prior injection mechanism, this invention can explicitly introduce high-frequency structural information during the attention modeling process, enabling the model to maintain global modeling capabilities while further enhancing attention to high-frequency detail regions such as edges and textures, thereby effectively improving the detail fidelity and structural consistency of the super-resolution reconstruction results.

[0095] 5. Frequency enhancement loss (FE-Loss).

[0096] Frequency enhancement loss (FE-Loss) was adopted with , Similar pixel-level reconstruction loss formats. However, unlike traditional formats that treat each image pixel equally... , Unlike other methods, this invention uses frequency information as a guide and assigns higher priority to image pixels in high-frequency image regions. The intuition behind this invention is quite simple: high-frequency regions typically correspond to complex structures and are more difficult to reconstruct than flat regions; therefore, this invention requires placing higher loss weights. More specifically, as... Figure 6 As shown, this invention calculates a true high-resolution image. and predicted high-resolution images The gradient map is generated by dividing the entire dataset into regions [0-1] by the maximum gradient value. Based on the gradient statistics of the dataset, this invention empirically selects an appropriate threshold to generate the corresponding high-frequency mask.

[0097] Specifically, this invention compares the normalized gradient norm with a high-frequency mask of a predefined hyperparameter θ. If a pixel's normalized gradient norm is greater than θ, this invention considers it a high-frequency pixel and assigns it a higher loss weight. Exemplary It can be set to 2.5. Otherwise, it will be assigned the default loss weight of 1. In form, the present invention has: ; ; in, , These represent the image gradient operators operating in the horizontal and vertical directions, respectively, with image I referring to the true high-resolution image. or predicted high-resolution images After the above calculations, the following results were obtained. and The high-frequency mask is calculated using the following formula: .

[0098] Therefore, the format of the FE-Loss loss is defined as follows: .

[0099] In addition, this invention also calculates the gradient map of the residual image. loss , represented as: .

[0100] Thus, the final loss function is obtained: .

[0101] In the method of the present invention, Emphasizing the core components of the total loss function, and Improve estimation Branching, thereby improving Overall performance. Therefore, maintain The primary position of this invention is the editing hyperparameter. The maximum value is 1, and the default value is 0.2.

[0102] like Figure 7 As shown, this invention performs a comprehensive analysis of the gradient value statistics of the entire training dataset. These values ​​are systematically sorted in ascending order. First, the ratio is adjusted. It indicates that the entire dataset is arranged from largest to smallest at the top (1− The enhanced region is 99.99%. Subsequently, based on the statistical results, the index of the quantile is recorded as θ, corresponding to the ratio based on the statistical results. .

[0103] Figure 7 In the middle, sections (a) and (b) on the right show FANSR at different scales. And the SSIM (structural similarity) performance of ×4 SR on the BSDS dataset with enhanced weight γ. When in part (a) Or in part (b) When it is true, it means that all pixels have equal priority, which is equivalent to . The determination of the region to be enhanced utilizes gradient statistics from the dataset, as shown in the left-hand side. This invention first extracts statistical features from the dataset to obtain a comparison table, and then modulates the proportion of the region requiring enhancement. The corresponding hyperparameter θ can be obtained from the table.

[0104] III. Effect Verification.

[0105] To demonstrate the performance of the above-described solution of the present invention, experiments are conducted.

[0106] This invention trains the model based on the DIV2K dataset and experimental specifications published in the 2017 paper "Ntire 2017 challenge on single image super-resolution: Dataset and study" by E. Agustsson et al. Comparative experiments and performance evaluations were conducted on publicly available datasets such as Set14, BSDS100, Manga109, and HQ-50K proposed or compiled by Dong, Zhang, A. Fujimoto, Yang, and others to verify the advantages and positive effects of the proposed Frequency Domain Adaptive Super-Resolution Network (FANSR) in terms of reconstruction accuracy, detail recovery ability, and overall visual quality. The evaluation metrics used are as follows: (1) Peak Signal to Noise Ratio (PSNR), the higher the better; (2) Structural Similarity Index (SSIM), the higher the better; All the above metrics are calculated only on the luminance channel (Y channel) after converting the RGB image to the YCbCr color space, to ensure that the evaluation results are consistent with existing mainstream methods.

[0107] 1. Experimental details.

[0108] In the experiment, this invention implements FANSR based on the PyTorch open-source framework. The frequency domain adaptive super-resolution network adopts the following parameter configuration: In the network structure, the number of frequency domain adaptive feature groups (FAGroups) is set to... = 3, and each FAGroup consists of 3. = Consists of 3 FABlocks; the number of channels for the intermediate latent features is set to 54. During the model training phase, low-resolution image patches of size 64×64 are selected as training samples, and the AdamW optimization algorithm is used to iteratively update the network parameters. The first-order momentum coefficient in the AdamW optimization algorithm... Set to 0.9, second momentum coefficient The initial learning rate was set to 0.999. The training process consisted of approximately 500,000 iterations. Furthermore, a multi-stage learning rate scheduling strategy is adopted to gradually adjust the learning rate, so as to ensure that the model has a fast convergence speed in the early stage of training and achieves stable optimization results in the later stage.

[0109] Figure 8 The effectiveness analysis results of FCA, FSA, and Frequency Enhancement Loss (FE-Loss) during the training process are presented. The relevant results are based on the "barbara" image from the Set14 dataset.

[0110] Figure 9 This paper demonstrates how Local Attribution Map (LAM), Diffusion Indices (DIs), and the receptive field visualization method proposed in the literature HAT were used to analyze the effective receptive field characteristics of the FANSR proposed in this invention and two existing state-of-the-art methods, ELAN and OmniSR.

[0111] 2. Results on public datasets.

[0112] The FANSR model in this invention is compared and analyzed with several existing representative super-resolution methods, including EDSR-baseline (Enhanced deep residual networks for single image super-resolution), LatticeNet (Towards lightweight image super-resolution with lattice block), and Transformer-based methods such as SwinIR-light (Image restoration using swin transformer), ELAN (Efficient long-range attention network for image super-resolution), Omni-SR (Omni aggregation networks for lightweight image super-resolution), ESRT (Transformer for single image super-resolution), CRAFT (Feature modulation transformer: Cross-refinement of global representation via high-frequency prior for image super-resolution), and CATANet (Catanet: Efficient content-aware token aggregation for lightweight image super-resolution). The comparisons are made from two aspects: intuitive visualization quality and evaluation metrics.

[0113] (1) Visual quality comparison.

[0114] This invention selects two representative reconstruction results at a magnification of ×4 on the Urban100 dataset for visualization comparison. The visualization results are as follows: Figure 10 As shown in the figure, the results demonstrate that, compared to existing methods, the FANSR proposed in this invention achieves clearer and more accurate reconstruction results in complex structural regions.

[0115] In the first example (img004), that is Figure 10In the upper part, the image contains a large number of repeating raster structures. FANSR can recover the regular repeating raster texture relatively completely, with clear edges and accurate alignment. In contrast, convolutional neural network-based methods (such as EDSR and LatticeNet) tend to produce over-smoothing results in this scene, while some Transformer-based methods introduce blurring or structural distortion.

[0116] In the second example (img092), that is Figure 10 The lower half of the image contains rich high-frequency texture information. FANSR is able to maintain the clarity of line intersections and the consistency of the overall geometry during reconstruction, while other contrast methods generally suffer from aliasing or reduced corner accuracy.

[0117] The qualitative results above demonstrate that, thanks to the effective joint modeling of spatial and frequency domain information, the FANSR method proposed in this invention has significant advantages in the recovery of fine-grained structures and high-frequency details.

[0118] (2) Comparison of evaluation indicators.

[0119] As shown in Tables 1 to 3, under the lightweight model settings, the method proposed in this invention was quantitatively compared with a variety of existing representative super-resolution methods, including EDSR-baseline and LatticeNet based on convolutional neural networks (CNN), and SwinIR-light, ELAN, Omni-SR, ESRT, CRAFT and CATANet based on Transformer.

[0120] Table 1: Comparison of evaluation metrics on the HQ-50K, Set14, BSDS100, and Manga109 datasets at ×2 magnification.

[0121] Table 2: Comparison of evaluation metrics on the HQ-50K, Set14, BSDS100, and Manga109 datasets at a magnification of ×3.

[0122] Table 3: Comparison of evaluation metrics on the HQ-50K, Set14, BSDS100, and Manga109 datasets at a magnification of ×4.

[0123] The experimental results show that, at a magnification of ×3, the FANSR method proposed in this invention outperforms existing comparative methods on all benchmark datasets; 2 and 4. At various magnification levels, FANSR also achieved best or near-best reconstruction performance on most test datasets. In particular, on the HQ-50K dataset, which contains 1250 high-quality images from a variety of complex scenes and categories and is one of the most representative and challenging super-resolution test sets currently available, the method of this invention achieved the best quantitative results at all magnification settings.

[0124] Although the recently proposed CATANet has reduced the size of the model parameters to some extent by introducing a learnable token aggregation mechanism, it still introduces a high computational latency during the inference phase, which may limit the real-time performance of the system in practical applications.

[0125] In contrast, the method of this invention effectively avoids redundant computational operations such as Softmax and feature rearrangement required in traditional self-attention structures by performing feature modeling in the frequency domain. While maintaining reconstruction performance comparable to or even better than existing advanced methods, it significantly reduces the inference time of the model, thereby achieving a better balance between reconstruction quality and computational efficiency.

[0126] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0127] Example 2 This invention also provides a lightweight image super-resolution system based on the frequency domain, which is mainly used to implement the methods provided in the foregoing embodiments, such as... Figure 11 As shown, the system mainly includes: The training set collection unit is used to collect the training set. Each sample includes two images with different resolutions, where the image with lower resolution is called the low-resolution image and the other is called the true high-resolution image.

[0128] A network construction and training unit is used to construct a frequency-domain adaptive super-resolution network and train it using a training set. The frequency-domain adaptive super-resolution network includes a feature extractor and a super-resolution image decoder. A low-resolution image is input, and the feature extractor performs initial feature extraction, attention modeling, and enhancement to obtain enhanced features. The super-resolution image decoder uses the initial and enhanced features to output a decoded image, called the predicted high-resolution image. A high-frequency mask is generated using the gradient maps of the predicted high-resolution image and the real high-resolution image, and a frequency enhancement loss is constructed. The frequency-domain adaptive super-resolution network is trained based on the frequency enhancement loss. The feature extractor includes a frequency-domain channel attention module and a frequency-domain spatial attention module to achieve interaction between the spatial and channel dimensions. The frequency-domain channel attention module and the frequency-domain spatial attention module extract high-frequency priors based on high- and low-frequency decomposition in the frequency domain, respectively, to guide the attention modeling process of the feature extractor. The super-resolution unit is used to input the image to be super-resolution into the trained frequency domain adaptive super-resolution network to obtain a predicted high-resolution image.

[0129] Since the main technical details of the above system have been described in detail in the previous embodiments, they will not be repeated here.

[0130] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0131] Example 3 The present invention also provides a processing device, such as Figure 12 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0132] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0133] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example: Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc. The output device can be a display terminal; The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0134] Example 4 The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0135] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0136] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A lightweight image super-resolution method based on the frequency domain, characterized in that, include: Collect a training set, each sample consisting of two images with different resolutions, where the image with lower resolution is called the low-resolution image and the other is called the true high-resolution image; A frequency-domain adaptive super-resolution network is constructed and trained using a training set. The frequency-domain adaptive super-resolution network includes a feature extractor and a super-resolution image decoder. A low-resolution image is input, and the feature extractor performs initial feature extraction, attention modeling, and enhancement to obtain enhanced features. The super-resolution image decoder uses the initial and enhanced features to output a decoded image, called the predicted high-resolution image. A high-frequency mask is generated using the gradient maps of the predicted high-resolution image and the real high-resolution image, and a frequency enhancement loss is constructed. The frequency-domain adaptive super-resolution network is trained based on the frequency enhancement loss. The feature extractor includes a frequency-domain channel attention module and a frequency-domain spatial attention module to achieve interaction between the spatial and channel dimensions. Furthermore, the frequency-domain channel attention module and the frequency-domain spatial attention module extract high-frequency priors based on high- and low-frequency decomposition in the frequency domain, respectively, to guide the attention modeling process of the feature extractor. The image to be super-resolution is input into the trained frequency domain adaptive super-resolution network to obtain the predicted high-resolution image.

2. The lightweight image super-resolution method based on the frequency domain according to claim 1, characterized in that, The feature extractor includes: a shallow feature extractor and a deep feature extractor; The shallow feature extractor is used to extract initial features; The deep feature extractor is used to perform attention modeling and enhancement on the initial features to obtain enhanced features. The deep feature extractor comprises multiple cascaded frequency-domain adaptive feature groups. Each frequency-domain adaptive feature group includes multiple sequentially connected frequency-domain adaptive feature blocks, as well as terminal convolutional layers and activation function layers. Each frequency-domain adaptive feature block includes a frequency-domain channel attention module, a channel-dimensional feature self-attention mechanism processing block, a frequency-domain spatial attention module, and a spatial-dimensional feature self-attention mechanism processing block, arranged sequentially. The channel-dimensional feature self-attention mechanism processing block and the spatial-dimensional feature self-attention mechanism processing block adjust their own input features using the high-frequency priors output by the frequency-domain channel attention module and the frequency-domain spatial attention module, respectively, and fuse them with their own attention outputs. In each frequency-domain adaptive feature group, the output of the spatial-dimensional feature self-attention mechanism processing block within the last frequency-domain adaptive feature block is processed by the terminal convolutional layer and activation function layer, and then added to the input of the frequency-domain adaptive feature group to obtain the output of the frequency-domain adaptive feature group. The output of the last frequency-domain adaptive feature group is the enhanced feature.

3. The lightweight image super-resolution method based on the frequency domain according to claim 1, characterized in that, The frequency domain channel attention module is used to explicitly model the global dependencies between different channels by performing Fourier transform operations along the channel dimension; the input features are denoted as... Applying a Fourier transform to it yields the channel frequency domain vector. Furthermore, matrix multiplication is performed on the query and key in the frequency domain to obtain a global interactive representation in the channel dimension; Perform an inverse transformation on the global interaction representation to obtain features. Then, sigmoid activation and average pooling operations are applied sequentially to generate a channel adaptive mask. The channel adaptive mask modulates the values ​​in the spatial domain to obtain the output features. The query is Through the key matrix The key is obtained by projection, and then the value matrix is ​​used to obtain the key. The value is obtained by projection; The average pooling operation uses high-order global average pooling, which divides the features... Convert to a global vector This global vector This is the channel adaptive mask, and the entire process is represented as follows: ； ； ； Where H and W represent the spatial height and width of the image, respectively. Indicates the feature Features obtained by global average pooling. Features The value at spatial location (h, w) and channel index c; N represents the maximum order of the global statistical feature, and k represents the order index of the global statistical feature. Representation of features The k-th order global statistical feature.

4. The lightweight image super-resolution method based on the frequency domain according to claim 1, characterized in that, The frequency domain spatial attention module is used to model global information in the spatial dimension and fuse it with local spatial features; The frequency-domain spatial attention module includes a frequency branch and a spatial branch, and the input features are denoted as... Spatial branching uses deep convolutional layers to process features. Processing is performed to obtain local spatial features; frequency In the branch, for features The frequency representation is obtained by performing a two-dimensional Fourier transform. And add location encoding Then, through complex convolutional layers and corresponding activation functions, and then through two-dimensional inverse Fourier transform, a global attention map, i.e., global information, is generated. The global attention map is then fused with local spatial features to obtain the output features of the frequency domain spatial attention module. .

5. A lightweight image super-resolution method based on the frequency domain according to claim 3 or 4, characterized in that, The step of extracting high-frequency priors based on high- and low-frequency decomposition in the frequency domain channel attention module and the frequency domain spatial attention module includes: Both the frequency domain channel attention module and the frequency domain spatial attention module obtain the frequency domain representation through corresponding Fourier transforms. d∈{c,s}, where c refers to the frequency domain channel attention module and s refers to the frequency domain spatial attention module. Frequency domain representation The frequency domain vectors are obtained by the channel attention module through Fourier transform operations. Frequency domain representation The frequency representation is obtained through two-dimensional Fourier transform in the frequency domain spatial attention module. ; By using a preset high-frequency mask and low-frequency mask Frequency domain representation Decompose to obtain high-frequency components and low frequency components ; For frequency domain representation By analyzing the corresponding high-frequency components or low-frequency components Perform one-dimensional inverse Fourier transform, normalization mapping, and differentiable discretization to generate channel priors. ; For frequency domain representation By analyzing the corresponding high-frequency components or low-frequency components Perform two-dimensional inverse Fourier transform, normalization mapping, and differentiable discretization to generate spatial priors. ; The channel prior With spatial a priori All of these belong to the aforementioned high-frequency priors.

6. A lightweight image super-resolution method based on the frequency domain according to claim 2, characterized in that, The channel dimension feature self-attention mechanism processing block and the spatial dimension feature self-attention mechanism processing block respectively utilize the high-frequency priors output by the frequency domain channel attention module and the frequency domain spatial attention module to adjust their own input features and fuse them with their own attention output, including: The channel-dimensional feature self-attention mechanism processes the high-frequency priors of the block input frequency domain channel attention module output, i.e., the channel priors. The spatial dimension feature self-attention mechanism processes the high-frequency priors of the frequency domain spatial attention module output from the input block, i.e., the spatial priors. ; Memorizing high-frequency priors Representative channel prior or spatial prior , d∈{c,s}; The attention outputs of the channel dimension feature self-attention mechanism processing block and the spatial dimension feature self-attention mechanism processing block are represented as: Where Q, K, and V represent the query, key, and value information in the attention mechanism, Let T be the dimension of the key information, and T be the transpose symbol; then the fusion process can be represented as: ； in, and There are two hyperparameters. The input features represent the self-attention mechanism processing blocks for channel-dimensional features or spatial-dimensional features. The output features represent the fusion of channel dimension feature self-attention mechanism processing blocks or spatial dimension feature self-attention mechanism processing blocks; For normalized exponential functions, This represents the weight matrix of the output projection. This indicates element-wise multiplication.

7. A lightweight image super-resolution method based on the frequency domain according to claim 2, characterized in that, The construction of the frequency enhancement loss, and the training of the frequency domain adaptive super-resolution network based on the frequency enhancement loss, includes: Constructing frequency enhancement loss , is represented as: ； Where H and W represent the spatial height and width of the image, respectively. For true high-resolution images The value at spatial location (h, w) For the prediction of high-resolution images The value at spatial location (h, w) The value of the high-frequency mask at spatial location (h, w); High-frequency mask calculation methods include: calculating the predicted high-resolution image separately. Compared to true high-resolution images The gradient map is obtained; for each spatial position in each gradient map, the normalized gradient norm is calculated and compared with the set hyperparameter θ. If it is less than θ, the loss weight value is assigned to 1; if it is greater than or equal to θ, the loss weight value is assigned to γ. Finally, the loss weight map corresponding to the two gradient maps is obtained. The two loss weight maps are fused to obtain the high frequency mask. Construct the final loss function Loss based on frequency augmentation loss: ； ； in, For hyperparameters, For L1 loss, Indicates the prediction of high-resolution images Compared to true high-resolution images The difference in gradients; The frequency domain adaptive super-resolution network is trained using the final loss function Loss.

8. A lightweight image super-resolution system based on the frequency domain, characterized in that, To implement the method according to any one of claims 1 to 7, comprising: The training set collection unit is used to collect the training set. Each sample includes two images with different resolutions, where the image with lower resolution is called the low-resolution image and the other is called the true high-resolution image. A network construction and training unit is used to construct a frequency-domain adaptive super-resolution network and train it using a training set. The frequency-domain adaptive super-resolution network includes a feature extractor and a super-resolution image decoder. A low-resolution image is input, and the feature extractor performs initial feature extraction, attention modeling, and enhancement to obtain enhanced features. The super-resolution image decoder uses the initial and enhanced features to output a decoded image, called the predicted high-resolution image. A high-frequency mask is generated using the gradient maps of the predicted high-resolution image and the real high-resolution image, and a frequency enhancement loss is constructed. The frequency-domain adaptive super-resolution network is trained based on the frequency enhancement loss. The feature extractor includes a frequency-domain channel attention module and a frequency-domain spatial attention module to achieve interaction between the spatial and channel dimensions. The frequency-domain channel attention module and the frequency-domain spatial attention module extract high-frequency priors based on high- and low-frequency decomposition in the frequency domain, respectively, to guide the attention modeling process of the feature extractor. The super-resolution unit is used to input the image to be super-resolution into the trained frequency domain adaptive super-resolution network to obtain a predicted high-resolution image.

9. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images