A lightweight image super-resolution method based on trajectory guidance and dual-domain refinement

Abstract

This invention discloses a lightweight image super-resolution method based on trajectory guidance and dual-domain refinement, comprising: acquiring a low-resolution input image and extracting initial spatial features through a shallow feature extraction module; inputting the initial spatial features into a deep feature extraction network, which is composed of multiple trajectory-guided basic blocks stacked together; fusing the deep features output from each level of the deep feature extraction network, and performing spatial resolution amplification through an upsampling reconstruction module, finally reconstructing and outputting a high-resolution image; this invention achieves dual leadership in objective evaluation indicators and subjective visual quality while maintaining low computational overhead.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a lightweight image super-resolution method based on trajectory guidance and dual-domain refinement. Background Technology

[0002] Image super-resolution aims to recover high-resolution images with rich high-frequency details from low-resolution images and is an important research foundation in the fields of image enhancement and computer vision. In recent years, with the rapid development of deep learning technology, super-resolution methods based on the Transformer architecture have achieved remarkable results in many image restoration tasks due to their excellent global context long-distance modeling capabilities.

[0003] However, the computational complexity of the global self-attention mechanism in the standard Vision Transformer is related to the spatial resolution of the input image (number of feature points). It grows in a quadratic manner, that is... This high computational overhead makes it difficult to apply directly to processing high-resolution images, and it is also difficult to meet the needs of lightweight application scenarios with limited computing resources, such as mobile devices, drones, and embedded devices.

[0004] Existing lightweight super-resolution techniques mainly fall into two categories: First, algorithms based on local window attention (such as SwinIR) restrict attention computation to a fixed local window. While this method reduces complexity, it fragments the global structure, easily generating artifacts at window edges and severely limiting the receptive field. Second, algorithms based on sparse attention (such as PFT) attempt to retain only the most strongly responding key feature points for computation. However, existing sparse attention mechanisms generally suffer from the fatal technical limitation of "unidirectional and irreversible hard sparse truncation." In shallow networks, weak high-frequency textures often have weak responses; once rigidly judged as background and discarded, deep networks will permanently lose these feature inputs due to the lack of cross-layer compensation mechanisms, resulting in structural breaks and blurred edges in the final reconstructed image.

[0005] Furthermore, standard feedforward neural networks (FFNs) in existing networks typically exhibit low-pass filter characteristics. Existing frequency domain processing methods often only focus on global spectrum mapping, lacking explicit high-frequency compensation schemes for the low-pass characteristics of feedforward networks, further exacerbating the detail loss problem in dense prediction tasks. In image super-resolution tasks, weak textures or fine edge structures often have weak feature response signals in shallow networks. If these crucial high-frequency features are judged as background or redundant information and filtered out in the shallow stage because of their "weak signal," then deep networks will lose the input of this information and struggle to recover details. This hard-discard feature filtering strategy easily leads to structural breaks, overly smoothed textures, and blurred edges in the final reconstructed image.

[0006] Furthermore, standard feedforward neural networks (FFNs) in existing Transformer networks typically exhibit low-pass filter characteristics during feature mapping, tending to smooth features and suppress high-frequency signals. This further exacerbates the problem of detail loss in dense prediction tasks like super-resolution, which heavily rely on the recovery of high-frequency information. Currently, existing technologies pay little attention to cross-level compensation mechanisms for misselected features during sparse attention processes, and also lack effective frequency domain compensation schemes for the low-pass characteristics of feedforward networks. Summary of the Invention

[0007] Purpose of the Invention: The purpose of this invention is to provide a lightweight image super-resolution method based on trajectory guidance and dual-domain refinement, which solves the problem that existing lightweight image super-resolution methods are prone to loss of image detail information and insufficient structural expression ability during sparse feature modeling. Specifically, it includes: (1) improving the network's ability to retain important feature information by constructing a cross-layer information transmission mechanism, thereby achieving effective restoration of image details; (2) improving the problem of insufficient expression of detail information by introducing a high-frequency feature enhancement structure, thereby achieving explicit modeling and compensation of high-frequency information, and thus improving the structural integrity and texture quality of the reconstructed image.

[0008] Technical solution: The lightweight image super-resolution method based on trajectory guidance and dual-domain refinement described in this invention includes the following steps:

[0009] Step 1: Acquire a low-resolution input image and extract initial spatial features using a shallow feature extraction module;

[0010] Step 2: Input the initial spatial features into the deep feature extraction network, which consists of multiple trajectory-guided preservation blocks stacked together. The processing of each trajectory-guided preservation block is as follows: In the self-attention processing stage, a trajectory-guided dynamic attention mechanism is adopted. By fusing the attention response information of the current layer, the cross-layer historical feature retention information from the shallow layers of the network, and the prior information of the local structure, a dynamic sparse attention mask is generated, and the current layer features are aggregated and compensated based on the dynamic sparse attention mask. In the feedforward processing stage, a dual-domain feature refinement feedforward network is adopted. The input features are divided along the channel dimension and fed into the spatial domain processing branch and the frequency domain processing branch for parallel refinement. Finally, the spatial domain refinement result and the frequency domain refinement result are fused to output the enhanced features.

[0011] Step 3: Fuse the deep features from each level of the deep feature extraction network, and then use the upsampling reconstruction module to amplify the spatial resolution, finally reconstructing and outputting a high-resolution image.

[0012] Furthermore, in step 2, the trajectory-guided dynamic attention mechanism is as follows: Calculate the original global attention score of the current layer's input features; obtain the historical attention trajectory information retained and passed from the previous layer; perform spatial high-pass filtering on the current layer's input features to extract prior information about the local structure representing image contours and edges; weight and fuse the original global attention score, historical attention trajectory information, and prior information about the local structure to generate a fused response map; select the subset of positions with the highest response values ​​from the fused response map according to a preset sparsity ratio, and generate a binarized dynamic sparse attention mask accordingly; mask the original global attention score based on the dynamic sparse attention mask before performing attention calculation to achieve adaptive focusing on key feature regions; update the historical attention trajectory information using the dynamically sparse attention mask generated this time according to a preset momentum update rule, and pass the updated trajectory information to the next processing layer.

[0013] Furthermore, spatial high-pass filtering of the current layer input features specifically involves performing convolution operations on the input feature map using a preset fixed convolution operator to explicitly capture and extract local edge and contour information in the image.

[0014] Furthermore, in step 2, the dual-domain feature refinement feedforward network is specifically as follows: the input features are split into a first branch feature and a second branch feature in the channel dimension according to a preset ratio; in the spatial domain processing branch, the first branch feature is subjected to depthwise convolution and pointwise convolution operations in sequence, and nonlinear context modeling is performed through a gating mechanism to obtain spatial domain enhanced features; in the frequency domain processing branch, a two-dimensional Fourier transform is performed on the second branch feature to map from the spatial domain to the frequency domain to obtain spectral features; learnable complex weights are introduced to adaptively amplify the high-frequency components in the spectral features to generate enhanced spectral features; a two-dimensional inverse Fourier transform is performed on the enhanced spectral features to map them from the frequency domain back to the spatial domain to obtain frequency domain enhanced features; the spatial domain enhanced features and the frequency domain enhanced features are concatenated in the channel dimension, and restored to the original feature dimension through a linear projection layer to complete the complementary fusion of dual-domain features.

[0015] Furthermore, in step 2, the deep feature extraction network is composed of multiple residual groups connected in series. Each residual group contains multiple cascaded trajectory-guided preservation basic blocks, thereby realizing the progressive deep extraction and refinement of features.

[0016] Furthermore, the method also includes: during the model training phase, constructing a training dataset using pairs of high- and low-resolution images, and performing end-to-end iterative training of the network using a loss function based on pixel-level content differences to determine the optimal network weight parameters.

[0017] Furthermore, in step 3, the upsampling reconstruction module achieves spatial resolution magnification of the feature map through subpixel convolution or transposed convolution operations.

[0018] An electronic device according to the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the methods described herein.

[0019] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the methods described herein.

[0020] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: Through dynamic weighted compensation of cross-layer historical trajectories, it effectively avoids the permanent loss of weak textures and thin edges in shallow networks due to rigid sparse classification as background. In complex structural regions such as building grids and text edges, the fragmentation phenomenon in the reconstructed image is significantly reduced, and the structural fidelity is greatly improved. Addressing the texture smoothing problem caused by the inherent low-pass filtering characteristics of traditional feedforward networks, dual-domain decoupling and explicit gain amplification in the frequency domain are used to specifically compensate and enhance high-frequency details such as hair and fabric textures in the image, making the reconstructed image texture sharper and more natural. While significantly improving reconstruction quality, the dynamic sparse masking mechanism suppresses the computational consumption of redundant background regions, keeping the overall model parameter count and computational complexity at a lightweight level. Compared with mainstream models of the same scale, this invention achieves a dual lead in objective evaluation indicators and subjective visual quality while maintaining low computational overhead. Attached Figure Description

[0021] Figure 1 This is a flowchart of the present invention;

[0022] Figure 2 This is a diagram of the overall network structure framework of the TDP-Former of this invention;

[0023] Figure 3 This is a schematic diagram of the decomposition module for local edge extraction and dynamic mask generation in the trajectory-guided dynamic attention mechanism (TGDA) of the present invention;

[0024] Figure 4 This is a schematic diagram of the parallel processing of spatial domain branches and frequency domain branches in the dual-domain feature refinement feedforward network (DDR-FFN) of the present invention. Detailed Implementation

[0025] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0026] Embodiments of the present invention provide a lightweight image super-resolution method based on trajectory guidance and dual-domain refinement, comprising the following steps:

[0027] Step 1: Obtain a low-resolution input image and input it into the shallow feature extraction module to extract the initial high-dimensional spatial feature representation through local convolution operation;

[0028] Step 2: The initial features are fed into a deep feature extraction network composed of multiple Trajectory-Guided Detail-Preserving Transformer basic blocks (hereinafter referred to as TDP-Former). In the self-attention computation stage, a Trajectory-Guided Dynamic Attention (TGDA) mechanism is introduced. By fusing the attention response information of the current layer, the historical feature preservation information across layers, and the prior information of local structure, a dynamic sparse attention mask is generated, thereby achieving adaptive preservation and dynamic compensation of important detail features.

[0029] In the deep feature extraction process, assuming the first The input features of each TDP-Former layer are ,in , , These represent the height, width, and number of channels of the feature map, respectively. First, the feature map is flattened into a sequence. (in (Through the parameter matrix) , , Obtain the query matrix by performing linear projection. Key matrix Sum matrix :

[0030]

[0031] , , These are the learnable parameters in the network, which are optimized and updated during model training using the backpropagation algorithm.

[0032] Further calculate the original attention score matrix of the current layer. :

[0033]

[0034] in, The feature dimensions of the query vector and key vector are represented by a scaling factor of 1. .

[0035] To avoid discarding weakly responding high-frequency textures as background, a historical trajectory weighted fusion mechanism is introduced to generate a dynamic mask. First, a spatial high-pass filter is used to extract local edge prior maps from the input features. This prior mapping can explicitly capture the contours and edge cues of an image. Secondly, the introduction of the... Layer's historical attention-preserving trajectory matrix This is used to convey structural information that has already been focused on in shallow networks. It represents the original attention score of the current layer. Historical trajectory matrix and local structural priors Weighted fusion is performed to generate the evolved dynamic sparse mask. The path is defined as follows:

[0036]

[0037] in, Indices representing spatial locations in the feature map. and These are adjustable weight hyperparameters, their values ​​determined through optimization in ablation experiments on the validation set, used to balance the contributions of the current response, historical trajectory, and local prior. During network training and inference, and It is fixed as a constant and does not participate in backpropagation updates. The function selects the index matrix of the position where the maximum value is located according to a preset sparsity ratio, sets the selected position to 1, and sets the rest to 0, thereby filtering out redundant background information and outputting a binary mask matrix. .

[0038] After obtaining the mask, masking attention is calculated, ensuring the network aggregates features only in key regions activated by the mask. Simultaneously, based on the configured momentum parameters... The updated historical trajectory is passed to the next layer, using an exponential moving average to ensure that deeper layers of the network retain detailed responses. Masked attention calculations are then performed on the current layer's trajectory. The update formula is as follows:

[0039]

[0040] ,

[0041] in This represents the Hadamard product (element-by-element multiplication). To select a relatively large negative number.

[0042] Step 3: In the feature feedforward stage of each TDP-Former, a dual-domain feature refinement feedforward network (DDR-FFN) is used to divide the input features in the channel dimension according to a preset ratio. The features are then processed in parallel by the nonlinear context modeling module in the spatial domain and the high-frequency enhancement module in the frequency domain. Finally, the dual-domain features are fused to output refined features.

[0043] The feature output by the TGDA module is denoted as... When entering DDR-FFN, in order to achieve a reasonable allocation of computational resources and decoupling of different frequency characteristics, the features are first proportionally divided into two parts along the channel dimension: spatial domain branch features. Frequency domain branching features (in ).

[0044] In the spatial domain branch, a deep convolutional structure with non-linear activation mechanism is used for local context modeling:

[0045]

[0046] in, This represents the Gaussian error linear unit activation function, used to enhance the nonlinear expressive power of features.

[0047] In the frequency domain branch, to overcome the technical limitations of feedforward networks in suppressing high-frequency signals, a two-dimensional fast Fourier transform (2D-FFT) is used to... Mapping to the frequency domain:

[0048]

[0049] in, and These represent the horizontal and vertical frequency indices in the frequency domain, respectively. Indicates the channel index.

[0050] Introduce a configurable complex weight tensor This tensor is a built-in learnable parameter vector of the network, whose initial values ​​are randomly generated according to a Gaussian or uniform distribution. During the end-to-end training process described in Step 5, this tensor is updated synchronously with the network weights through backpropagation. Through this adaptive learning, this tensor can amplify the gain of high-frequency regions in the frequency domain that represent details, explicitly compensating for the high-frequency details lost in sparse computation.

[0051]

[0052] Subsequently, the enhanced frequency domain features are mapped back to the spatial domain using a two-dimensional inverse fast Fourier transform (2D-IFFT):

[0053]

[0054] Finally, the refined features of the dual domains and The channels are cascaded and stitched together, and the residual connections are combined with linear projection to restore the original number of channels.

[0055] Step 4: Fuse the deep features of each layer of the network and input them into the upsampling reconstruction module. Spatial resolution is magnified through subpixel convolution or transposed convolution, and the reconstructed high-resolution image is output.

[0056] Step 5: Construct an image super-resolution training dataset, and use a pixel-level content loss function to perform end-to-end iterative optimization training on the above network until the model converges, and obtain the optimal network weights for image inference.

[0057] Example 1:

[0058] like Figure 1 As shown, this embodiment of the invention provides a lightweight image super-resolution method based on trajectory guidance and dual-domain refinement, including the following steps:

[0059] S11: Pair the high-resolution image (HR) and the corresponding downsampled low-resolution image (LR) to construct a training set. In each training batch, a pre-defined image patch (preferably 64×64 pixels) is randomly cropped from the low-resolution image as network input, and the corresponding high-resolution image patch size is preferably 256×256 pixels. Simultaneously, horizontal flipping, vertical flipping, and rotation operations are performed on the input image patch for data augmentation.

[0060] S12: Input the preprocessed low-resolution image patch into the shallow feature extraction module. For example... Figure 2 As shown, an initial feature map is obtained by mapping image information to a high-dimensional feature space through a single 3×3 convolutional layer. Preferably, the number of feature channels... Mapped to 60, but in other embodiments of the present invention, the number of channels can be flexibly adjusted according to actual computing power requirements, and is not limited thereto;

[0061] S13: As Figure 2 As shown, the initial feature map is input into the deep feature extraction network. Preferably, the deep feature extraction network consists of 6 cascaded residual groups, with 6 TDP-Former basic blocks stacked inside each residual group. This invention does not impose an absolute restriction on the specific number of groups and blocks. The features are enhanced sequentially through the self-attention module and the feedforward refinement module in the TDP-Former basic block.

[0062] S14: As Figure 3 As shown, a trajectory-guided dynamic attention mechanism (TGDA) is executed during the self-attention stage of each TDP-Former. First, the input features are high-pass filtered through a two-dimensional convolution operation (preferably the Sobel operator) to extract local edge prior maps. Secondly, obtain the global attention score matrix for the current feature. And introduce the historical attention trajectory matrix of the previous layer. The three path information points are weighted and fused to generate the evolved dynamic sparse mask. Path:

[0063]

[0064] Among them, the fusion coefficient The preferred value is 0.6, guiding weight. The preferred value is 0.1. The sparsity ratio of the function is preferably set to 25%. The generated dynamic mask... With attention score matrix Element-wise multiplication is performed to calculate masked attention. Simultaneously, based on the momentum coefficient... Update trajectory matrix And pass it to the next level;

[0065] S15: Combination Figure 4 The features output from the TGDA module are input into a dual-domain feature refinement feedforward network (DDR-FFN). The input features are then segmented by channel dimension according to a preset ratio, with a total of 60 channels preferably segmented into spatial domain branches. and frequency domain branch The spatial domain branch features undergo gated deep convolution (Gated-Dconv); the frequency domain branch features are transformed to the frequency domain representation using a two-dimensional fast Fourier transform (2D-FFT), where a complex weight matrix tensor is introduced and multiplied element-wise with the spectrum to amplify the high-frequency amplitude gain, and then transformed back to the spatial domain using a two-dimensional inverse fast Fourier transform (2D-IFFT). Finally, the feature channels processed by the two branches are concatenated and fused.

[0066] S16: As Figure 2 As shown, deep refined features are input into the high-resolution image reconstruction module. This module preferably uses sub-pixel convolution (PixelShuffle) to enlarge the feature resolution by a preset factor in the spatial dimension, outputting a reconstructed high-resolution image;

[0067] S17: Deploy the constructed network architecture on an end-to-end training framework. Input the constructed LR / HR image pairs, and calculate the error between the reconstructed image and the real high-resolution image using a preset loss function (preferably L1 loss). Update the network weights using an optimization algorithm (preferably the Adam optimizer), and iteratively train until the model converges;

[0068] S18: Load the optimal weights of the trained network model. Input the test image into the model for inference and output the final high-quality super-resolution image.

Claims

1. A lightweight image super-resolution method based on trajectory guidance and dual-domain refinement, characterized in that, Includes the following steps: Step 1: Acquire a low-resolution input image and extract initial spatial features using a shallow feature extraction module; Step 2: Input the initial spatial features into the deep feature extraction network, which consists of multiple trajectory-guided preservation blocks stacked together. The processing of each trajectory-guided preservation block is as follows: In the self-attention processing stage, a trajectory-guided dynamic attention mechanism is adopted. By fusing the attention response information of the current layer, the cross-layer historical feature retention information from the shallow layers of the network, and the prior information of the local structure, a dynamic sparse attention mask is generated, and the current layer features are aggregated and compensated based on the dynamic sparse attention mask. In the feedforward processing stage, a dual-domain feature refinement feedforward network is adopted. The input features are divided along the channel dimension and fed into the spatial domain processing branch and the frequency domain processing branch for parallel refinement. Finally, the spatial domain refinement result and the frequency domain refinement result are fused to output the enhanced features. Step 3: Fuse the deep features from each level of the deep feature extraction network, and then use the upsampling reconstruction module to amplify the spatial resolution, finally reconstructing and outputting a high-resolution image.

2. The lightweight image super-resolution method based on trajectory guidance and dual-domain refinement according to claim 1, characterized in that, In step 2, the trajectory-guided dynamic attention mechanism is as follows: calculate the original global attention score of the current layer input features; obtain the historical attention trajectory information retained and passed in the previous layer processing; perform spatial high-pass filtering on the current layer input features to extract the local structural prior information representing the image contour and edge; and perform weighted fusion of the original global attention score, historical attention trajectory information and local structural prior information to generate a fused response map. Based on the preset sparsity ratio, a subset of positions with the highest response values ​​is selected from the fused response map, and a binarized dynamic sparse attention mask is generated accordingly. The original global attention score is masked by dynamic sparse attention masking before attention calculation is performed to achieve adaptive focusing on key feature regions. According to the preset momentum update rules, the historical attention trajectory information is updated using the dynamically sparse attention mask generated this time, and the updated trajectory information is passed to the next processing layer.

3. The lightweight image super-resolution method based on trajectory guidance and dual-domain refinement according to claim 2, characterized in that, The spatial high-pass filtering of the current layer input features is specifically performed by using a preset fixed convolution operator to perform convolution operation on the input feature map in order to explicitly capture and extract local edge and contour information in the image.

4. The lightweight image super-resolution method based on trajectory guidance and dual-domain refinement according to claim 1, characterized in that, In step 2, the dual-domain feature refinement feedforward network is implemented as follows: the input features are split into first-branch features and second-branch features in the channel dimension according to a preset ratio; in the spatial domain processing branch, the first-branch features are subjected to depthwise convolution and pointwise convolution operations in sequence, and nonlinear context modeling is performed through a gating mechanism to obtain spatial domain enhanced features; in the frequency domain processing branch, a two-dimensional Fourier transform is performed on the second-branch features to map the frequency domain from the spatial domain to obtain spectral features; learnable complex weights are introduced to adaptively amplify the high-frequency components in the spectral features to generate enhanced spectral features; Perform a two-dimensional inverse Fourier transform on the enhanced spectral features to map them back from the frequency domain to the spatial domain, thus obtaining the frequency domain enhanced features; Spatial domain enhancement features and frequency domain enhancement features are concatenated along the channel dimension and restored to the original feature dimension through a linear projection layer, thus completing the complementary fusion of dual-domain features.

5. A lightweight image super-resolution method based on trajectory guidance and dual-domain refinement according to claim 1, characterized in that, In step 2, the deep feature extraction network is composed of multiple residual groups connected in series. Each residual group contains multiple cascaded trajectory-guided preservation basic blocks, which realizes the progressive deep extraction and refinement of features.

6. The lightweight image super-resolution method based on trajectory guidance and dual-domain refinement according to claim 1, characterized in that, The method further includes: during the model training phase, constructing a training dataset using pairs of high- and low-resolution images, and performing end-to-end iterative training of the network using a loss function based on pixel-level content differences to determine the optimal network weight parameters.

7. A lightweight image super-resolution method based on trajectory guidance and dual-domain refinement according to claim 1, characterized in that, In step 3, the upsampling reconstruction module achieves spatial resolution magnification of the feature map through subpixel convolution or transposed convolution.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images