Image super-resolution reconstruction method based on multi-scale progressive fusion and frequency separation

Abstract

This invention discloses an image super-resolution reconstruction method based on multi-scale progressive fusion and frequency separation, comprising: S10, shallow feature extraction; S20, multi-scale progressive fusion; S30, adaptive frequency separation and refinement; S40, second-order attention feature refinement; S50, iterative feedback optimization; and S60, high-resolution reconstruction. This invention achieves synergistic enhancement of image structural information and texture details through mechanisms such as adaptive frequency separation, multi-scale progressive fusion, second-order attention feature refinement, and iterative feedback optimization. This method effectively improves edge sharpness, texture realism, and reconstruction accuracy, reduces artifacts and high-frequency distortion, and enhances perceptual quality through frequency domain constraints. While maintaining low parameter count and computational overhead, it achieves higher PSNR, SSIM, and subjective visual evaluation metrics, balancing reconstruction quality and computational efficiency, and possesses good engineering application value and promising prospects for widespread application.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image reconstruction, specifically to an image super-resolution reconstruction method based on multi-scale progressive fusion and frequency separation. Background Technology

[0002] Image super-resolution reconstruction is a classic and challenging problem in computational imaging and computer vision, aiming to recover a high-quality image with higher spatial resolution from one or more low-resolution observation images. In many real-world applications, images are often insufficient in resolution, losing crucial details due to limitations imposed by the physical resolution of image acquisition equipment, the diffraction limit of optical imaging systems, sensor noise, ambient lighting conditions, and bandwidth constraints during image transmission. High-resolution images not only provide richer texture details and sharper edge structures but also significantly improve the performance of subsequent advanced vision tasks, including object detection and recognition, image segmentation, medical image diagnosis, remote sensing image analysis, video surveillance and security, face recognition, and scene understanding in autonomous driving. Therefore, researching efficient and accurate image super-resolution reconstruction techniques has significant theoretical value and broad application prospects.

[0003] From a mathematical perspective, image super-resolution reconstruction is essentially an ill-conditioned inverse problem because information loss and mapping ambiguity occur during the recovery of a high-resolution image from a single low-resolution image. Specifically, a low-resolution image can be modeled as the result of a series of degradation operations on a high-resolution image, typically including blurring, downsampling, and noise contamination. Let the high-resolution image be... ,in and They are height and width, respectively. This refers to the number of channels (usually 3, corresponding to RGB three channels). Low-resolution images are... ,satisfy , ,in This is the downsampling factor (usually 2, 3, or 4). The classic image degradation model can be represented as: ; Here This represents the convolution operation. It is a fuzz kernel (usually assumed to be a Gaussian fuzz kernel or an isotropic fuzz kernel). This indicates a spatial downsampling operation (usually nearest neighbor downsampling or bicubic downsampling). Additive white Gaussian noise, its distribution can be expressed as: The task of super-resolution reconstruction is to design a reconstruction function. This makes it possible for a given low-resolution image It can recover high-resolution images as accurately as possible. That is, to solve: ; in This is the set of parameters for the reconstruction model. Since the mapping from low resolution to high resolution is a one-to-many problem (i.e., the same low-resolution image may correspond to multiple different high-resolution images), solving this problem requires introducing reasonable prior knowledge to constrain the solution space, so that the reconstruction result can maintain realism while having rich details.

[0004] Traditional image super-resolution methods are mainly divided into three categories: interpolation-based methods, reconstruction model-based methods, and learning-based methods. Interpolation-based methods, such as bilinear interpolation, bicubic interpolation, and nearest-neighbor interpolation, estimate the value of unknown pixels by calculating a weighted average of neighboring pixels. These methods are computationally efficient, but the restored images often have blurred edges, lost details, and exhibit jagged edges. Reconstruction model-based methods, such as iterative back-projection, convex set projection, and maximum a posteriori probability estimation, optimize the reconstruction results by introducing prior constraints on the image (such as total variational regularization and sparse representation). These methods are an improvement over interpolation methods, but are still limited by the expressive power of the prior model and struggle to recover complex texture structures. Learning-based methods, especially the deep learning-based methods that have emerged in recent years, have achieved breakthroughs in image super-resolution tasks by learning the mapping relationships between a large number of low-resolution and high-resolution image pairs, and have become the mainstream technology direction.

[0005] Several deep learning-based image super-resolution reconstruction schemes most similar to this invention include early methods based on convolutional neural networks, methods based on residual learning and dense connections, methods based on attention mechanisms, and methods based on generative adversarial networks.

[0006] SRCNN (Super-Resolution Convolutional Neural Network) is a pioneering work on the application of deep learning to image super-resolution reconstruction. Among the existing technologies, there are SRCNN methods based on convolutional neural networks. This method uses a three-layer convolutional neural network to directly learn the end-to-end mapping relationship between low-resolution images and high-resolution images. Its network structure is simple and effective, laying the foundation for subsequent research.

[0007] The SRCNN network architecture consists of three convolutional layers: the first layer is a feature extraction layer, using... The first layer uses a convolutional kernel of a certain size. The input is a low-resolution image upsampled to the target size via bicubic interpolation. The output has 64 channels. This layer extracts overlapping image patches from the input image and represents them as high-dimensional feature vectors. The second layer is a non-linear mapping layer, using... or The first layer uses a convolutional kernel of a certain size with 32 output channels. This layer is responsible for non-linearly mapping the feature vectors extracted by the first layer to another feature space. The third layer is the reconstruction layer, using... A convolutional kernel of size 1000, with 3 output channels (corresponding to an RGB image), reconstructs the final high-resolution image from the feature vectors of the second layer. Mathematically, the forward propagation process of SRCNN can be represented as: ; ; ; in This represents the image after upsampling via bicubic interpolation. and The first The kernel weights and bias terms of the layer. This represents the convolution operation. This is the ReLU activation function. The network training uses mean squared error (MSE) as the loss function, defined as: ; in The number of training samples is given. Although SRCNN achieves significantly improved reconstruction results compared to traditional methods, its network is relatively shallow with a limited receptive field, making it difficult to capture a wide range of contextual information. In addition, the bicubic interpolation preprocessing step introduces additional computational overhead.

[0008] To overcome the limitation of SRCNN network depth, the existing technology VDSR (Very Deep Super-Resolution) method, based on residual learning, is the first to introduce residual learning into image super-resolution tasks. VDSR's network depth reaches 20 layers and employs a residual learning strategy, meaning the network learns the residual between the low-resolution input and the high-resolution output, rather than directly learning the complete output image. This design allows the network to propagate gradients more effectively, alleviating the gradient vanishing problem in deep networks. Furthermore, because the residual signal typically has a small amplitude, network training is more stable. The residual learning of VDSR can be expressed as: ; in This represents the residual mapping learned by the residual network. VDSR also employs a multi-scale training strategy, allowing the same network to handle super-resolution tasks at different scales. However, VDSR still uses bicubic interpolation as a preprocessing step, resulting in high computational complexity.

[0009] To address the information flow problem in deep networks, the existing technology SRDenseNet, a dense connection method, introduces dense connections into super-resolution networks. In SRDenseNet, the input of each layer comes from the outputs of all previous layers. This dense connection approach greatly promotes feature reuse and gradient propagation. For a given super-resolution network containing... Dense blocks of the layer, the first The output of the layer can be represented as: ; Here This indicates that the number of layers from layer 0 to layer 1 will be used. The output feature maps of the layer are concatenated along the channel dimension. For the first Layer composite functions typically include batch normalization, ReLU activation, and convolution operations. Dense connections allow networks to increase depth while maintaining parameter efficiency, but also result in higher memory consumption.

[0010] To address the computational redundancy issue caused by preprocessing upsampling, existing technologies include the Efficient Sub-Pixel Convolutional Neural Network (ESPCN), a method based on sub-pixel convolutional layers. This method introduces a novel sub-pixel convolutional layer, directly rearranging the low-resolution feature map into a high-resolution output through convolution operations. The ESPCN network structure consists of multiple convolutional layers, culminating in a sub-pixel convolutional layer. Let the size of the input low-resolution image be... After several convolutional layers, a size of [size missing] is obtained. The feature map, where , The target output channel number is 3 (3 for RGB images). Subpixel convolution operations rearrange pixels periodically, moving the feature map from... Convert to This operation can be mathematically represented as: ; in The input feature map for the sub-pixel convolutional layer. To output high-resolution images, the introduction of sub-pixel convolutional layers allows the network to extract features in the low-resolution space, increasing resolution only in the last layer, thus significantly reducing computational complexity. ESPCN also uses the MSE loss function, but due to its efficient structure, it enables real-time super-resolution reconstruction.

[0011] As researchers delve deeper into the performance bottlenecks of deep networks, they've discovered that not all feature channels and spatial locations contribute equally to the reconstruction task. Based on this observation, RCAN (Residual Channel Attention Network) has emerged, introducing a channel attention mechanism into image super-resolution networks. The core module of RCAN is the Residual Channel Attention Block (RCAB), which comprises two main parts: residual connections and the channel attention mechanism. The channel attention mechanism compresses the spatial dimension into channel description vectors using global average pooling, then calculates the weight coefficients for each channel using two fully connected layers (with dimensionality reduction and expansion operations), and finally multiplies the weights channel-by-channel with the original feature map. Specifically, for the input feature map… The calculation process for channel attention is as follows: ; ; ; in This is the channel description vector after global average pooling. and This is the weight matrix of the fully connected layer. It is the ReLU activation function. For the Sigmoid activation function, This indicates channel-by-channel multiplication. RCAN constructed an extremely deep network of over 400 layers by stacking multiple RCABs and using long-jump connections, achieving state-of-the-art results in multiple benchmark tests at the time.

[0012] Building upon RCAN, Dai et al. proposed SAN (Second-order Attention Network), extending the attention mechanism from first-order statistics to second-order statistics. The second-order attention mechanism captures the correlation between channels by calculating the covariance matrix of the feature maps, and its mathematical expression is: ; in The feature map is normalized. Let be the covariance matrix. More refined channel weights can be obtained by eigenvalue decomposition of the covariance matrix or by using a more efficient power iteration method. Second-order attention mechanisms can better model the complex dependencies between feature channels, but their computational complexity is high.

[0013] While the aforementioned methods based on MSE loss can achieve high peak signal-to-noise ratios (PSNR), the reconstructed results are often overly smooth and lack realistic high-frequency details. This is mainly because MSE loss tends to average multiple possible solutions, leading to excessive smoothing of the texture details in the output image. To address this issue, existing technologies include SRGAN (Super-Resolution Generative Adversarial Network), a generative adversarial network-based method that applies the generative adversarial network framework to image super-resolution tasks. SRGAN comprises two core components: a generator and a discriminator. The generator's task is to generate the most realistic high-resolution image possible to deceive the discriminator, while the discriminator's task is to distinguish the generated image from the real high-resolution image.

[0014] SRGAN's generator employs a residual block structure, containing multiple residual units, each consisting of a convolutional layer, a batch normalization layer, and a ReLU activation layer. Unlike the aforementioned methods, SRGAN uses perceptual loss instead of the traditional MSE loss. The perceptual loss comprises two parts: content loss and adversarial loss. The content loss is not computed in pixel space but in the feature space of the pre-trained VGG network, defined as: ; in Indicates the first in the VGG network Before the first pooling layer The output feature map of each convolutional layer and Let be the size of the feature map. The adversarial loss is defined as: ; Discriminator output This represents the probability that the input image is a true high-resolution image. The total loss function is a weighted sum of content loss and adversarial loss: ; SRGAN can generate super-resolution images with rich texture details, significantly outperforming MSE-based methods in terms of subjective visual quality. However, the training process of SRGAN is not stable enough and is prone to artifacts and unrealistic textures.

[0015] In practical applications, low-resolution images are often not obtained through simple bicubic downsampling, but rather undergo unknown and complex degradation processes (such as sensor noise, JPEG compression, motion blur, etc.). To address this challenge, existing technologies include the CinCGAN (Cycle-in-Cycle Generative Adversarial Network) method, which introduces cycle consistency constraints into image super-resolution tasks, achieving unsupervised real-world image super-resolution reconstruction. The core idea of ​​CinCGAN is to construct two generators: one mapping from the low-resolution domain to the high-resolution domain, and the other mapping from the high-resolution domain back to the low-resolution domain, using a cycle consistency loss to ensure the bidirectional reversibility of the mapping. The cycle consistency loss can be expressed as: ; in For low-resolution to high-resolution generators, CinCGAN is a high-resolution to low-resolution generator. It does not require paired training data and can be trained using a large number of real low-resolution images, exhibiting strong generalization ability. However, its network structure is complex and training is difficult.

[0016] Based on the above analysis, the closest existing technical solutions to this invention are the RCAN method based on an attention mechanism and the SRGAN method based on a generative adversarial network. RCAN effectively models the importance differences between feature channels through a channel attention mechanism, enabling the network to focus on information-rich features and achieving excellent performance in PSNR. However, RCAN still has shortcomings in processing high-frequency details, as its attention mechanism is only based on first-order statistics (global average pooling), failing to fully utilize higher-order statistical information in the feature maps to guide detail reconstruction. While SRGAN can generate visually satisfactory texture details, its generation process lacks explicit modeling of frequency components, easily producing artifacts inconsistent with the real content, and the weights of the perceptual loss require fine-tuning. Furthermore, existing methods generally adopt a single forward propagation reconstruction strategy, lacking an iterative refinement mechanism for the reconstruction results, leading to insufficient accuracy in the recovered high-frequency details. This invention addresses these shortcomings by proposing a novel image super-resolution reconstruction method based on multi-scale progressive fusion and adaptive frequency separation.

[0017] Through in-depth analysis of the above-mentioned existing technical solutions, the following main shortcomings and deficiencies can be summarized: First, existing methods lack specificity in processing image frequency components. High-resolution image information is essentially composed of both low-frequency and high-frequency components. Low-frequency components primarily describe the image's contours and overall structure, while high-frequency components carry texture, edge, and detail information. Most existing super-resolution methods (such as RCAN and ESPCN) apply the same processing strategy to all channels and spatial locations of the feature map, without explicitly distinguishing between low-frequency and high-frequency features. Although channel attention mechanisms can assign different weights to different channels, this weighting method remains global and static, unable to adaptively adjust according to the frequency characteristics of each spatial location. This can lead to unnecessary texture artifacts when processing flat regions, while over-smoothing may occur when processing complex texture regions due to insufficient high-frequency information. From a signal processing perspective, ideal super-resolution reconstruction should maintain fidelity in the low-frequency components (i.e., preserve the overall structure of the original image) while appropriately enhancing and generating the high-frequency components. Existing methods fail to achieve this differentiated frequency-aware processing.

[0018] Second, existing methods suffer from simplistic feature fusion approaches, lacking a progressive integration of multi-scale information. Image features naturally exist at different scales, with complex cross-scale dependencies between large-scale features (such as the overall shape of an object) and small-scale features (such as fine textures). While methods like RCAN and SRDenseNet deepen the network by stacking multiple residual or dense blocks, features at different scales are typically processed at the same layer of the network, lacking explicit multi-scale feature extraction and progressive fusion mechanisms. Methods like VDSR use only single-scale convolutional kernels with fixed receptive fields, making it difficult to simultaneously capture broad semantic information and local detail variations. Even some networks employing pyramid structures often simply add or concatenate multi-scale features without forming a coarse-to-fine progressive reconstruction process. This coarse-grained feature fusion approach limits the network's ability to model cross-scale dependencies, resulting in jagged edges and a lack of depth in textured regions.

[0019] Third, existing attention mechanisms are limited to first-order statistics, failing to fully exploit higher-order correlations in feature maps. The channel attention used in works like RCAN is based on first-order statistics obtained through global average pooling (i.e., the mean of each channel). These statistics only reflect the overall activation intensity of channels and cannot capture the covariance structure and higher-order moments between channels. In fact, feature maps in deep networks contain rich statistical information, and the correlations between different channels are crucial for distinguishing different types of information. For example, in super-resolution tasks, some channels may collectively respond to edges in a specific direction, and the covariance between these channels should be large; while other channels may respond to noise or irrelevant information, and their covariance should be suppressed. First-order attention mechanisms cannot model such second-order or even higher-order statistical dependencies, thus limiting the network's expressive power and the quality of detail recovery.

[0020] Fourth, existing methods lack effective iterative refinement mechanisms, making it difficult to fully approximate the optimal solution in a single forward propagation. Image super-resolution is essentially an ill-conditioned inverse problem with an extremely large solution space. Most existing deep learning methods employ a single forward propagation approach, where a low-resolution image is passed through the network and directly outputs a high-resolution image; the entire process is one-off and open-loop. This strategy lacks the ability to self-evaluate and correct the reconstruction results. Once the network loses some key information during the forward propagation process, this information cannot be recovered in subsequent stages. In contrast, humans often employ iterative optimization strategies in the cognitive process, gradually approximating the reality through repeated observation and correction. While some iterative methods (such as iterative backprojection) have been applied in traditional reconstruction frameworks, in the era of deep learning, designing an end-to-end trainable iterative refinement mechanism remains an underexplored area.

[0021] Fifth, existing methods' loss functions fail to adequately balance perceptual quality and frequency domain fidelity. While methods like RCAN, based on MSE loss, can achieve high PSNR, the reconstruction results are overly smooth and lack realistic high-frequency details. Methods like SRGAN, based on adversarial loss, can generate rich textures, but due to the instability of GAN training, they are prone to producing false textures inconsistent with the original image content (i.e., so-called "illusion artifacts"), and may exhibit significant distortion in certain regions. Although perceptual loss utilizes the feature space of a pre-trained VGG network to measure semantic similarity, VGG networks are trained for image classification tasks, and their feature space may not optimally reflect perceptual quality in super-resolution tasks. Furthermore, existing methods rarely directly constrain reconstruction quality in the frequency domain, leading to deviations between the frequency domain characteristics of the reconstructed image and the true high-resolution image. Summary of the Invention

[0022] To address the shortcomings of the existing technologies, the present invention aims to provide an image super-resolution reconstruction method based on multi-scale progressive fusion and adaptive frequency separation, overcoming the deficiencies of the prior art through the following innovations: The first objective of this invention is to propose an adaptive frequency separation mechanism that can dynamically decompose a feature map into low-frequency and high-frequency components based on the local frequency characteristics of each spatial location, and adopt differentiated processing strategies for different components, thereby improving the quality of detail reconstruction and suppressing artifact generation.

[0023] The second objective of this invention is to propose a multi-scale progressive fusion module that extracts and fuses features at different scales by constructing a feature pyramid structure and adopting a coarse-to-fine progressive reconstruction strategy, enabling the network to fully model cross-scale contextual dependencies and generate super-resolution images with clear edges and rich textures.

[0024] The third objective of this invention is to design a second-order attention feature refinement module that uses the covariance matrix of the feature map to capture the second-order statistical correlation between channels, thereby achieving more accurate enhancement of important feature channels and effective suppression of redundant channels.

[0025] The fourth objective of this invention is to introduce a lightweight iterative feedback optimization mechanism. By constructing feedback connections, the information from the output end is fed back to the early layers of the network to form a closed-loop optimization, enabling the network to self-correct and iteratively refine the reconstruction results, gradually approaching the ideal high-resolution image.

[0026] The fifth objective of this invention is to propose a composite loss function that integrates pixel-level fidelity loss, frequency domain spectral loss, perceptual loss, and adversarial loss, thereby improving visual perception quality while maintaining high fidelity, and ensuring that the reconstructed image has consistent spectral characteristics with the real image through frequency domain constraints.

[0027] In summary, the present invention aims to provide a high-quality, high-fidelity super-resolution reconstruction method with good visual effects, which can recover clearer and more realistic high-resolution images from low-resolution images, and is significantly better than the reconstruction effect of existing technologies.

[0028] The technical solution of the present invention includes the following steps: S10, shallow feature extraction; S20, multi-scale progressive fusion; S30, adaptive frequency separation and refining; S40, second-order attention feature refinement; S50, iterative feedback optimization; S60, high-resolution reconstruction.

[0029] Preferably, the input low-resolution image is... ,in and These represent the height and width of the low-resolution image, respectively; the target output is a high-resolution image. ,in , , The upsampling factor is used; S10 includes two consecutive convolutional layers to extract initial shallow features from the input low-resolution image. The first layer uses... The convolutional kernel has 3 input channels, 64 output channels, a stride of 1, and padding of 1. This layer is followed by the LeakyReLU activation function with a negative slope coefficient of 0.2. The second layer uses... The convolutional kernel has 64 input and output channels, a stride of 1, and padding of 1. This layer is followed by a LeakyReLU activation function. Let the output of the shallow feature extraction module be... Then the forward propagation process can be represented as: ; in and For convolution kernel weights, and For bias terms, This is the LeakyReLU activation function.

[0030] Preferably, S20 includes three convolutional branches with different receptive fields, corresponding to small-scale (fine details), medium-scale (local structure), and large-scale (global context), respectively. The small-scale branch uses... The convolution kernel has a stride of 1 and a dilation rate of 1; this branch is sensitive to local details. The mesoscale branch uses... Convolution kernel, expansion rate 2, receptive field expanded to Large-scale branching Convolution kernel, expansion rate 4, receptive field expanded to Each branch contains two convolutional layers, with batch normalization and LeakyReLU activation function used between them; The output feature maps of the three branches are concatenated along the channel dimension, and then passed through a... The convolutional layer performs channel compression to obtain fused multi-scale features; Let the input feature map be The outputs of the three branches are respectively , and The multi-scale fusion process can then be represented as: ; ; ; in, This indicates a batch normalization operation. The outputs of the three branches are concatenated along the channel dimension to obtain... ; Then through Convolution for channel fusion: ; in, The number of output channels should be kept consistent with the number of input channels to maintain the feasibility of residual connections; Multiple multi-scale progressive fusion modules are stacked in series, and residual connections are used to promote gradient flow. Let the first... The output of the multi-scale progressive fusion module is Its input is ,but: ; in This represents a multi-scale progressive fusion function, achieved through stacking. Such a modular network refines the feature representation step by step.

[0031] Preferably, in step S30, a set of learnable frequency separation filters is first used to estimate the local frequency distribution at each spatial location, and a learnable frequency separation convolution kernel is employed. Let the input feature map be... First, calculate two versions with different levels of smoothness: ; ; in and For learnable Gaussian convolution kernels, standard deviation and satisfy , In depthwise separable convolution, high-frequency components are obtained by the difference between two smoothed versions: ; Low-frequency components can be directly taken Then, a spatial adaptive modulation mechanism is introduced, using a lightweight sub-network to predict the optimal separation parameters at each spatial location. This sub-network consists of two convolutional layers: the first layer uses... Convolution, which reduces the number of channels in the input feature map from... Reduced to The second layer uses Convolution outputs two channels, corresponding to... and modulation coefficient and ; Then, the adaptive Gaussian kernel is represented as: ; in and As the initial base value, and For position The modulation coefficient at that point.

[0032] Preferably, in S30, the low-frequency features of adaptive separation are obtained. and high frequency characteristics Subsequently, differentiated processing strategies were adopted for the two. Low-frequency features contain the overall structure and global information of the image and are relatively insensitive to noise, so a lightweight processing method was used: one Convolutional layers are combined with a channel attention mechanism; high-frequency features carry detailed information and edges, requiring higher accuracy and employing a more complex processing method: a multi-branch fine-tuning module containing three parallel sub-branches, each using... , and The convolution kernel reduces computation through separable convolution, and then the outputs of the three sub-branches are fused together; The processing procedure for the low-frequency branch is represented as follows: ; in The channel attention operation is implemented as follows: first, global average pooling is performed on the input feature map to obtain the channel description vector; then, it is passed through two fully connected layers; finally, the sigmoid function is used to map the output to... The interval serves as the channel weight; The processing procedure for high-frequency branches is represented as follows: ; ; in Indicates a channel splicing operation; Finally, the processed low-frequency and high-frequency features are re-fused: .

[0033] Preferably, in step S40, the input feature map is set as follows: First, Remodeled into a two-dimensional matrix form ,in Then, calculate the covariance matrix of the feature maps: ; covariance matrix The diagonal elements represent the variance of each channel, reflecting the activation intensity of that channel; the off-diagonal elements represent the covariance between different channels, reflecting the correlation between channels. A normalization operation and a low-rank approximation are introduced for the covariance matrix. First, power normalization is performed on the covariance matrix: ; in, The trace of the matrix, To prevent division by zero errors, power normalization makes the trace of the covariance matrix 1, thus keeping the matrix scale within a reasonable range. The low-rank approximation method is used to decompose the covariance matrix into eigenvalues. The power iteration method is then used to find the principal eigenvectors of the covariance matrix. Let the largest eigenvalue of the covariance matrix be... The corresponding feature vector is The second-order attention weights can be calculated as follows: ; in and It is a fully connected layer. Using ReLU activation function, compute the soft-thresholding representation of the covariance matrix and perform eigenvalue decomposition on the covariance matrix. ,in and , keep before The lowest-rank approximation is obtained by taking the largest eigenvalue and its corresponding eigenvector: ; in , ,Pick That is, retaining 12.5% ​​of the principal components, calculate the second-order attention weights: ; in Representing the eigenvector The element-wise absolute value, then... The weight vector is obtained by averaging the values ​​of each channel. Finally, the second-order attention weights are multiplied by the original feature map: ; in This refers to the Sigmoid function.

[0034] Preferably, S50 receives the currently estimated feature representation and generates a correction signal, which is fed back to the output of the shallow feature extraction layer. The next iteration ( When ), the feature representation of the intermediate layer of the network is as follows: The feedback block generates a feedback signal. ,in It is a lightweight network consisting of two convolutional layers; then the feedback signal is combined with shallow features. Add them together and use the sum as input for the next iteration: ; in For the first The feedback weight coefficients of each iteration are learned during training or set to predefined values ​​that decrease with the number of iterations. Through multiple iterations, the network gradually improves the quality of feature representation. A truncated backpropagation strategy is adopted, that is, the number of iterations is fixed during training. Gradient propagation is performed on all iteration steps; during the inference phase, the number of iterations is flexibly adjusted as needed to achieve a trade-off between computational accuracy and speed. Treating super-resolution reconstruction as an optimization problem, the goal is to minimize the following energy function: ; in For degenerate operators, As a regularization term, iterative optimization updates the solution in the following way: .

[0035] Preferably, in step S60, the feature map after all refining steps is... ,in and For the spatial dimensions of the low-resolution feature map; firstly, through a Convolutional layers reduce the number of channels from Convert to : ; Then, subpixel convolution operations are applied to rearrange the feature maps into a high-resolution image: ; The specific implementation of subpixel convolution operation is as follows: the input feature map is... At each spatial location The channels were rearranged into one. Local areas, thereby magnifying spatial resolution. Times, for the location in the output high-resolution image and channels Its pixel value is given by the following formula: .

[0036] Preferably, S60 further includes a composite loss function that integrates four different types of loss terms: pixel-level fidelity loss, frequency domain spectral loss, perceptual loss, and adversarial loss, wherein... Pixel-level fidelity loss is achieved using L1 loss, which is defined as follows: ; in For batch size, Denotes the L1 norm; Frequency domain spectral loss is used to constrain the spectral consistency between the reconstructed image and the true high-resolution image in the frequency domain. A two-dimensional discrete Fourier transform is performed on the image to obtain its amplitude and phase spectra. The frequency domain loss is defined as the weighted sum of the amplitude and phase spectral differences. ; in and These represent the operations for extracting the amplitude spectrum and the phase spectrum, respectively. and To balance the coefficients, a fast Fourier transform algorithm is used, and frequency domain loss is calculated for image blocks. Perceptual loss uses the feature space of the pre-trained VGG-16 network to measure semantic similarity, calculating the Euclidean distance between the feature maps output by the last convolutional layer before the fourth pooling layer of the VGG network: ; in This represents the feature map output of a specified layer in the VGG-16 network. Use a discriminator network to combat loss To distinguish between generated high-resolution images and real high-resolution images, the discriminator network uses a PatchGAN structure and outputs a... The feature map is used, and each output value represents the probability that a local region of the input image is a real image. The adversarial loss is defined as: ; The training loss of the discriminator is: ; The total loss function is the weighted sum of the losses mentioned above: .

[0037] Preferably, network training is divided into two stages: In the first stage, pre-training is performed using only pixel-level fidelity loss and frequency domain spectral loss, without using adversarial loss. The pre-training learning rate is set to... The batch size is 16, the Adam optimizer is used, and the number of training iterations is 500,000. In the second stage, perceptual loss and adversarial loss are added for joint fine-tuning. During this fine-tuning stage, the generator and discriminator are updated alternately: the generator is updated once, and the discriminator is updated twice. The learning rate during the fine-tuning stage is reduced to a certain value. The training iterations were 200,000, and spectral normalization was used to normalize the convolutional layers of the discriminator. The beneficial effects of the invention are as follows: First, this invention significantly improves the quality of detail reconstruction through an adaptive frequency separation mechanism. Existing methods treat frequency components uniformly without differentiation, leading to artifacts in flat areas or overly smoothed textured areas. The adaptive frequency separation module proposed in this invention dynamically decomposes features into low-frequency and high-frequency components based on the local frequency characteristics of each spatial location, and employs differentiated processing strategies for different components. Low-frequency components use lightweight channel attention processing to maintain global structural consistency, while high-frequency components use multi-branch fine processing to enhance detail information. This differentiated processing allows the network to be optimized for the characteristics of different frequency components, thereby generating sharper and more realistic edges and textures while maintaining overall structural fidelity. Quantitative experiments show that after adopting the adaptive frequency separation mechanism proposed in this invention, the high-frequency energy distribution of the reconstructed image is closer to that of a true high-resolution image, and the high-frequency error (HF-ERR) is reduced by approximately 15% compared to RCAN.

[0038] Second, this invention effectively captures cross-scale contextual information through a multi-scale progressive fusion mechanism. Existing methods, such as VDSR, use a fixed receptive field, making it difficult to simultaneously capture global semantics and local details. The multi-scale progressive fusion module proposed in this invention extracts small-scale, medium-scale, and large-scale features through three parallel corridors with different expansion rates, and gradually integrates information from different scales through progressive fusion. This design enables the network to fully understand the hierarchical structure in the image, achieving progressive refinement from coarse-grained contours to fine-grained textures during reconstruction. Experimental results show that the multi-scale progressive fusion mechanism of this invention outperforms existing methods in edge localization accuracy, reducing the standard deviation of edge width in the reconstructed image by approximately 20%, meaning sharper and clearer edges.

[0039] Third, this invention models the correlation between feature channels more accurately through a second-order attention mechanism. Existing methods such as RCAN only use first-order attention based on global average pooling, which cannot capture higher-order statistical information between channels. The second-order attention feature refinement module proposed in this invention calculates the covariance matrix of the feature map and extracts higher-order correlations between channels using eigenvalue decomposition or low-rank approximation, thereby generating more accurate channel weights. Theoretical analysis and experiments have demonstrated that second-order attention has stronger expressive power than first-order attention, enabling the network to focus on feature channels that are truly useful for the reconstruction task and suppressing redundant and noisy information. On the benchmark test sets Set5, Set14, BSD100, and Urban100, the PSNR of this invention is improved by an average of 0.35dB compared to RCAN and by an average of 0.52dB compared to SRGAN.

[0040] Fourth, this invention achieves gradual refinement of the reconstruction results through an iterative feedback optimization mechanism. Most existing depth super-resolution methods employ an open-loop structure with a single forward propagation, lacking self-correction capabilities. The iterative feedback optimization module proposed in this invention feeds back information from the output to early layers of the network, forming a closed-loop iterative mechanism that allows the network to gradually improve reconstruction quality through multiple iterations. This design is mathematically equivalent to iteratively solving the inverse super-resolution problem, possessing a clear theoretical foundation. Experiments show that as the number of iterations increases from 1 to 4, the PSNR and SSIM metrics of the reconstructed image continuously improve, demonstrating the effectiveness of the iterative feedback mechanism. When the number of iterations is 4, compared to the baseline model with a single forward propagation, the PSNR is improved by 0.28 dB.

[0041] Fifth, this invention achieves a better balance between fidelity and perceptual quality through a composite loss function. Existing MSE-based methods lead to over-smoothing, while GAN-based methods may produce artificial textures. The composite loss function of this invention integrates four components: pixel-level L1 loss, frequency domain spectral loss, perceptual loss, and adversarial loss. Particularly noteworthy is the frequency domain spectral loss, a unique contribution of this invention. It directly constrains the spectral characteristics of the reconstructed image in the frequency domain, ensuring the reconstructed result has a consistent frequency distribution with the real high-resolution image. This frequency domain constraint effectively suppresses the "checkerboard" artifacts and high-frequency noise commonly found in GAN training, while enhancing the generation of realistic textures. Subjective visual evaluation experiments (Mean Opinion Score, MOS) show that the high-resolution images generated by the method of this invention outperform existing methods in all three dimensions: naturalness, sharpness, and realism, achieving a MOS score of 4.52 (out of 5), compared to 4.03 for SRGAN and 3.87 for RCAN.

[0042] Sixth, this invention achieves a good trade-off between computational efficiency and reconstruction quality. Through carefully designed lightweight modules (such as separable convolution and low-rank approximation), the number of parameters and computational cost are kept within a reasonable range. Compared to RCAN (approximately 15.6M parameters), the number of parameters in this invention is approximately 12.8M, a reduction of about 18%; compared to SRGAN (approximately 11.6M parameters), the number of parameters is slightly increased, but the reconstruction quality is significantly improved. In terms of inference speed, processing one image on an NVIDIA Tesla V100 GPU... For low-resolution images (upsampling factor 4), the inference time of the method of this invention is approximately 0.18 seconds, which can meet the requirements of most real-time applications.

[0043] In summary, the image super-resolution reconstruction method based on multi-scale progressive fusion and adaptive frequency separation proposed in this invention significantly outperforms existing technologies in multiple dimensions, including reconstruction quality, detail restoration, edge sharpness, perceptual realism, and computational efficiency. It possesses outstanding substantive features and represents a significant advancement. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating the model construction process of the image super-resolution reconstruction method based on multi-scale progressive fusion and frequency separation of the present invention. Figure 2 This is a schematic diagram of the corresponding structure of S20 in the image super-resolution reconstruction method based on multi-scale progressive fusion and frequency separation of the present invention. Figure 3 This is a schematic diagram of the corresponding structure of S30 in the image super-resolution reconstruction method based on multi-scale progressive fusion and frequency separation of the present invention. Figure 4 This is a schematic diagram of the corresponding structure of S40 in the image super-resolution reconstruction method based on multi-scale progressive fusion and frequency separation of the present invention. Figure 5 This is a schematic diagram of the corresponding structure of the S50 in the image super-resolution reconstruction method based on multi-scale progressive fusion and frequency separation of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0046] Conversely, this invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of the invention as defined in the claims. Furthermore, to provide a better understanding of the invention, certain specific details are described in detail below. However, those skilled in the art will fully understand the invention even without these detailed descriptions.

[0047] The specific technical solution of this invention includes: (See overall steps) Figure 1 This includes the following steps: S10, shallow feature extraction; S20, multi-scale progressive fusion; S30, adaptive frequency separation and refining; S40, second-order attention feature refinement; S50, iterative feedback optimization; S60, high-resolution reconstruction.

[0048] To clearly illustrate the technical solution of this invention, the mathematical expressions for the input and output are first defined. Let the input low-resolution image be... ,in and These represent the height and width of the low-resolution image, respectively. The target output is a high-resolution image. ,in , , upsampling factor (in this invention) (2, 3, or 4 values ​​can be used). Unlike existing methods, this invention does not use bicubic interpolation for preprocessing upsampling. Instead, it employs a strategy of feature extraction in a low-resolution space and then upsampling through sub-pixel convolutional layers to reduce computational complexity.

[0049] The S10 shallow feature extraction consists of two consecutive convolutional layers used to extract initial shallow features from the input low-resolution image. The first layer uses... The convolutional kernel has 3 input channels and 64 output channels, a stride of 1, and padding of 1. This layer is followed by the LeakyReLU activation function with a negative slope coefficient of 0.2. The second layer also uses... The convolutional kernel has 64 input and 64 output channels, a stride of 1, and padding of 1. This layer is also followed by a LeakyReLU activation function. Let the output of the shallow feature extraction be... Then the forward propagation process can be represented as: ; in and For convolution kernel weights, and For bias terms, This is the LeakyReLU activation function. The purpose of this step is to map the input image to a high-dimensional feature space, providing rich feature representations for subsequent deep processing.

[0050] S20 Multi-Scale Progressive Fusion: Multi-scale progressive fusion is a core innovation of this invention, and its design concept stems from a deep understanding of the multi-scale characteristics of images. It employs a feature pyramid structure, processing information at different scales through three parallel feature extraction paths. Then, through progressive fusion, the features at each scale are gradually integrated to form a hierarchical feature representation from coarse to fine.

[0051] like Figure 2 The diagram illustrates the internal structure of multi-scale progressive fusion, containing three convolutional branches with different receptive fields, corresponding to small scale (fine details), medium scale (local structure), and large scale (global context), respectively. Specifically, the small-scale branch uses... The convolution kernel has a stride of 1 and a dilation rate of 1; this branch is sensitive to local details. The mesoscale branch uses... Convolution kernel, expansion rate 2, receptive field expanded to Large-scale branching Convolution kernel, expansion rate 4, receptive field expanded to Each branch contains two convolutional layers, with batch normalization and LeakyReLU activation functions used between them. The output feature maps of the three branches are concatenated along the channel dimension and then passed through a... The convolutional layer performs channel compression to obtain fused multi-scale features.

[0052] Let the input feature map be The outputs of the three branches are respectively , and The multi-scale fusion process can then be represented as: ; ; ; in This indicates a batch normalization operation. The outputs of the three branches are concatenated along the channel dimension to obtain... Then through Convolution for channel fusion:

[0053] in The number of output channels should be kept consistent with the number of input channels to maintain the feasibility of residual connections.

[0054] To further achieve progressive reconstruction from coarse to fine, this invention stacks multiple multi-scale progressive fusion modules in series and employs residual connections to facilitate gradient flow. Let the... The output of the multi-scale progressive fusion module is Its input is ,but: ; in This represents a multi-scale progressive fusion function. It is achieved through stacking. Such a module (in this invention) The network can refine feature representations step by step. Furthermore, between different stacking stages, this invention introduces a feature recalibration mechanism to dynamically adjust the input weights of subsequent modules based on the feature statistics of the current stage.

[0055] S30 Adaptive Frequency Separation and Refinement: Adaptive frequency separation and refinement is another core innovation of this invention. This step can adaptively decompose the feature map into low-frequency and high-frequency components based on the local frequency characteristics of each spatial location, and then perform targeted processing on each component.

[0056] like Figure 3 The diagram shows the corresponding structure of adaptive frequency separation and refinement. First, a set of learnable frequency separation filters is used to estimate the local frequency distribution at each spatial location. Inspired by the Difference of Gaussians (DoG) filter in classical signal processing theory, this invention designs a learnable frequency separation convolution kernel. Let the input feature map be... First, calculate two versions with different levels of smoothness: ; ; in and For learnable Gaussian convolution kernels, standard deviation and satisfy , This represents depthwise separable convolution (each channel is convolved independently). High-frequency components can then be obtained by subtracting the values ​​from two smoothed versions: ; Low-frequency components can be directly taken However, the above methods use a fixed Gaussian kernel and lack adaptability. To achieve adaptive frequency separation for different image content, this invention introduces a spatial adaptive modulation mechanism. Specifically, a lightweight sub-network is used to predict the optimal separation parameters at each spatial location. This sub-network consists of two convolutional layers: the first layer uses... Convolution, which reduces the number of channels in the input feature map from... Reduced to The second layer uses Convolution outputs two channels, corresponding to... and modulation coefficient and Then, the adaptive Gaussian kernel can be expressed as: ; in and As the initial base value, and For position The modulation coefficients at the location. In this way, the network can adaptively adjust the scale of frequency separation according to the local image content (flat regions, edge regions, or textured regions), thereby obtaining a more accurate frequency decomposition.

[0057] In obtaining low-frequency features of adaptive separation and high frequency characteristics Subsequently, this invention employs differentiated processing strategies for the two. Low-frequency features mainly contain the overall structure and global information of the image and are relatively insensitive to noise; therefore, a lightweight processing method with higher computational efficiency is adopted: one Convolutional layers are combined with a channel attention mechanism. High-frequency features carry detailed information and edges, requiring higher accuracy; therefore, a more complex processing method is employed: a multi-branch fine-tuning module containing three parallel sub-branches, each using... , and The convolution kernel reduces computation through separable convolution, and then the outputs of the three sub-branches are merged.

[0058] The processing of low-frequency branches can be represented as follows: ; in The channel attention operation is implemented as follows: first, global average pooling is performed on the input feature map to obtain the channel description vector; then, it is passed through two fully connected layers (the first fully connected layer reduces the number of channels to the original value). The second fully connected layer restores the original number of channels, and finally the Sigmoid function is used to map the output to... The interval is used as the channel weight.

[0059] The processing of high-frequency branches can be represented as follows: ; ; in This indicates a channel splicing operation. Finally, the processed low-frequency and high-frequency features are re-fused: ; This differentiated processing strategy enables the network to allocate computing resources according to the importance of frequency components, improving efficiency while maintaining high-quality reconstruction.

[0060] S40 Second-order attention feature refinement: In order to further mine the higher-order statistical information of the feature map, this invention captures the second-order correlation between channels by calculating the covariance matrix of the feature map, and uses these correlations to recalibrate the feature channels.

[0061] like Figure 4 The diagram shown illustrates the corresponding structure of second-order attention feature refinement. Let the input feature map be... First, Remodeled into a two-dimensional matrix form ,in Then, calculate the covariance matrix of the feature maps: ; covariance matrix The diagonal elements represent the variance of each channel, reflecting the activation intensity of that channel; the off-diagonal elements represent the covariance between different channels, reflecting the correlation between channels. However, directly using the covariance matrix has two problems: first, the size of the covariance matrix is... ,when When the matrix is ​​large (e.g., 64 or 128), the storage and computational overhead of the matrix is ​​significant; secondly, the values ​​in the covariance matrix may be very large, and directly using them for attention weight calculation can lead to numerical instability. To address these issues, this invention introduces a normalization operation and a low-rank approximation for the covariance matrix.

[0062] First, power normalization is performed on the covariance matrix: ; in The trace of the matrix (the sum of the diagonal elements). For a very small constant (such as This is to prevent division by zero errors. Power normalization makes the trace of the covariance matrix 1, thus keeping the matrix scale within a reasonable range.

[0063] To reduce computational complexity, this invention employs a low-rank approximation method. The covariance matrix is ​​decomposed into eigenvalues, but for differentiability and computational efficiency, a power-law iteration method is used to solve for the principal eigenvectors of the covariance matrix. Let the largest eigenvalue of the covariance matrix be... The corresponding feature vector is The second-order attention weights can be calculated as follows: ; in and It is a fully connected layer. The ReLU activation function is used. However, using a single principal eigenvector results in the loss of information from other feature directions. Therefore, this invention employs a more refined method: calculating the soft-thresholding representation of the covariance matrix. Eigenvalue decomposition is then performed on the covariance matrix. ,in and Before keeping The lowest-rank approximation is obtained by taking the largest eigenvalue and its corresponding eigenvector: ; in , In this invention, the following is taken: That is, retaining 12.5% ​​of the principal components. Then, calculate the second-order attention weights: ; in Representing the eigenvector The element-wise absolute value, then... The weight vector is obtained by averaging the values ​​of each channel. Finally, the second-order attention weights are multiplied by the original feature map: ; in The sigmoid function is used. The second-order attention feature refinement module can be inserted after the multi-scale progressive fusion module to refine the fused features, or it can be used in parallel with the adaptive frequency separation module.

[0064] Mathematically, it can be proven that second-order attention mechanisms contain more information than first-order attention mechanisms. First-order attention is based on global average pooling, which essentially calculates the expected value of each channel. Second-order attention, based on the covariance matrix, contains... as well as This information can capture the nonlinear correlations between channels. Utilizing this higher-order statistical information allows the network to more accurately distinguish which channels contribute more to the reconstruction task, thus achieving better reconstruction quality.

[0065] S50 Iterative Feedback Optimization: In order to overcome the limitations of single forward propagation, this invention introduces iterative feedback optimization. Its core idea is to pass the information output by the network back to the early layers of the network through feedback connections, forming a closed-loop iterative mechanism, so that the network can gradually correct and refine the reconstruction results.

[0066] like Figure 5 The diagram shows the corresponding structure of iterative feedback optimization, which includes a feedback block. This feedback block receives the currently estimated feature representation and generates a correction signal, which is fed back to the output of the shallow feature extraction module. Let the... The next iteration ( When ), the feature representation of the intermediate layer of the network is as follows: The feedback block generates a feedback signal. ,in It is a lightweight network consisting of two convolutional layers. The feedback signal is then compared with shallow features. Add them together and use the sum as input for the next iteration: ; in For the first The feedback weight coefficients for each iteration can be learned during training or set to predefined values ​​that decrease with the number of iterations (e.g., ...). Through multiple iterations, the network can gradually improve the quality of feature representations.

[0067] To ensure that the iterative process is differentiable and end-to-end trainable, this invention employs a truncated backpropagation through time strategy, which fixes the number of iterations during training. Gradient propagation is performed on all iterations. During the inference phase, the number of iterations can be flexibly adjusted as needed to achieve a trade-off between computational accuracy and speed.

[0068] The theoretical basis of iterative feedback optimization can be traced back to classic iterative optimization algorithms. Super-resolution reconstruction is viewed as an optimization problem, with the objective of minimizing the following energy function: ; in For degenerate operators, This is a regularization term. Classical iterative optimization algorithms (such as gradient descent) update the solution in the following way: ; The iterative feedback module of this invention can be viewed as embedding this iterative optimization process into a deep network architecture, wherein the feedback block The gradient direction is implicitly learned, and each forward propagation of the network corresponds to one iteration update. Through multiple iterations, the network can asymptotically approach the optimal solution.

[0069] S60 High-Resolution Reconstruction: After the above-mentioned processing steps, the final feature representation needs to be converted into a high-resolution output image. This invention uses sub-pixel convolutional layers (i.e., pixelShuffle layers) to achieve feature map upsampling. Let the feature map after all refinement modules be... ,in and This refers to the spatial dimensions of the low-resolution feature map. First, through a... Convolutional layers reduce the number of channels from Convert to : ; Then, subpixel convolution operations are applied to rearrange the feature maps into a high-resolution image: ; The specific implementation of subpixel convolution operation is as follows: the input feature map is... At each spatial location The channels were rearranged into one. Local areas, thereby magnifying spatial resolution. Times. Expressed mathematically, for a given location in the output high-resolution image... and channels Its pixel value is given by the following formula: ; To train the network proposed in this invention, a composite loss function is designed in S60, which integrates four different types of loss terms: pixel-level fidelity loss, frequency domain spectral loss, perceptual loss, and adversarial loss. Each loss term targets different aspects of quality, and their combination enables the network to generate high-resolution images with good visual perception while maintaining high fidelity.

[0070] Pixel-level fidelity loss uses L1 loss instead of MSE loss because L1 loss better preserves edge information and is insensitive to outliers. L1 loss is defined as: ; in For batch size, This represents the L1 norm.

[0071] Frequency domain spectral loss is one of the innovations of this invention. It directly constrains the spectral consistency between the reconstructed image and the real high-resolution image in the frequency domain. A two-dimensional discrete Fourier transform is performed on the image to obtain its amplitude and phase spectra. Since the human visual system is more sensitive to the amplitude spectrum, and the phase spectrum mainly determines the structural information of the image, the frequency domain loss is defined as the weighted sum of the amplitude spectrum difference and the phase spectrum difference: ; in and These represent the operations for extracting the amplitude spectrum and the phase spectrum, respectively. and As balancing coefficients, they are set to 1.0 and 0.1 respectively in this invention. To reduce computational complexity, the Fast Fourier Transform (FFT) algorithm is used in the actual implementation, and frequency domain loss can be calculated for image blocks.

[0072] The perceptual loss utilizes the feature space of the pre-trained VGG-16 network to measure semantic similarity. Similar to the perceptual loss used in SRGAN, this invention calculates the Euclidean distance between the feature maps output by the last convolutional layer before the fourth pooling layer of the VGG network: ; in This represents the feature map output of a specified layer in the VGG-16 network.

[0073] Use a discriminator network to combat loss To distinguish between generated high-resolution images and real high-resolution images, the discriminator network uses a PatchGAN structure and outputs a... The feature map is used, and each output value represents the probability that a local region of the input image is a real image. The adversarial loss is defined as: ; The training loss of the discriminator is: ; The total loss function is the weighted sum of the losses mentioned above: ; In this invention, the weighting coefficients are set empirically as follows: , , , These coefficients can be adjusted according to the specific application scenario.

[0074] The network of this invention employs a two-stage training strategy. In the first stage, pre-training is performed using only pixel-level fidelity loss and frequency domain spectral loss, without using adversarial loss. The goal of this stage is to enable the network to quickly learn the underlying mappings from low to high resolution, obtaining relatively stable reconstruction results. The pre-training learning rate is set to... The batch size was 16, the Adam optimizer was used, and the number of training iterations was 500,000.

[0075] In the second stage, perceptual loss and adversarial loss are added for joint fine-tuning. During this fine-tuning phase, the generator and discriminator are updated alternately: the generator is updated once, and the discriminator is updated twice. The learning rate is reduced to a certain value during the fine-tuning phase. The training iterations were 200,000. To prevent mode collapse and training instability, spectral normalization was used to normalize the convolutional layers of the discriminator.

[0076] Training data utilizes publicly available high-resolution image datasets, such as subsets of DIV2K, Flickr2K, and ImageNet. During training, high-resolution images are downsampled using bicubic subsampling to obtain low-resolution images, and a suitable amount of Gaussian noise is added to enhance generalization ability. Data augmentation includes random horizontal flipping, random rotation by 90 degrees, 180 degrees, and 270 degrees, and random cropping.

[0077] The main contributions of this invention to the prior art are reflected in the following key technical points, which constitute the core protection scope of this invention: Key Point 1: Adaptive Frequency Separation Mechanism. This invention proposes for the first time a learnable adaptive frequency separation method, which predicts the optimal Gaussian filter parameters at each spatial location using a lightweight sub-network. and The modulation coefficients are used to achieve adaptive low-frequency / high-frequency decomposition of the feature map. Based on this, differentiated processing strategies are applied to the low-frequency and high-frequency components: channel attention is used to preserve the overall structure of the low-frequency components, while multi-branch convolution is used to enhance details of the high-frequency components. This mechanism is one of the core innovations of this invention and a key technology that brings beneficial effects. The scope of protection should cover any image feature processing method based on spatial adaptive frequency decomposition, including but not limited to technical solutions that use learnable Gaussian filters, Laplacian pyramids, wavelet transforms, or other time-frequency analysis methods for adaptive frequency separation and employ different processing strategies for different frequency components.

[0078] Key Point Two: Multi-Scale Progressive Fusion Module. This invention designs a multi-scale progressive fusion module based on a feature pyramid. This module extracts multi-scale features in parallel through three convolutional layers with different dilation rates, and then... Convolutional processing performs channel fusion, and progressive reconstruction from coarse to fine is achieved through the stacking of multiple modules. The structural design of this module (including the choice of dilation rate, number of branches, fusion method, and stacking method) and its combination with other modules (such as frequency separation modules and attention modules) should all be included within the scope of protection. Any technical solution employing multi-scale feature extraction and progressive fusion strategies in super-resolution networks, if its core idea is substantially the same as the multi-scale progressive fusion module of this invention, should be considered to fall within the scope of protection of this invention.

[0079] Key Point 3: Second-Order Attention Feature Refinement Module. This invention proposes a second-order attention mechanism based on the feature map covariance matrix. By calculating the covariance matrix of the feature map, it extracts the second-order statistical correlations between channels and uses these correlations to recalibrate the feature channels. Unlike existing first-order attention (based on global average pooling), the second-order attention of this invention can capture high-order dependencies between channels, thereby more accurately identifying important features. The scope of protection should cover any method that uses the covariance matrix, correlation matrix, or other second-order statistics to generate attention weights, including but not limited to implementations that use eigenvalue decomposition, singular value decomposition, low-rank approximation, or power iteration to extract second-order information. Furthermore, technical solutions that combine second-order attention with first-order attention (such as weighted fusion) should also fall within the scope of protection of this invention.

[0080] Key Point Four: Iterative Feedback Optimization Mechanism. This invention introduces a lightweight iterative feedback optimization module that uses feedback connections to transmit information from the network output back to earlier layers, forming a closed-loop iterative structure that allows the network to progressively refine the reconstruction results. The design of this mechanism includes the structure of the feedback block (two convolutional layers), the setting of feedback weights (learnable or predefined decay coefficients), and a training strategy that truncates backpropagation. Any deep network structure that uses feedback connections for iterative optimization in super-resolution or other image restoration tasks, if its core idea is similar to the iterative feedback optimization mechanism of this invention, should be considered an infringement of this invention. In particular, technical solutions that combine feedback iteration with any one or more of the aforementioned multi-scale fusion, frequency separation, or attention mechanisms are the key protected objects of this invention.

[0081] Key Point 5: Frequency Domain Spectral Loss Function. This invention introduces frequency domain constraints into the training of super-resolution networks for the first time in the form of a differentiable loss function. The frequency domain spectral loss consists of two parts: amplitude spectrum loss and phase spectrum loss. The image is converted to the frequency domain using a Fast Fourier Transform, and then the spectral difference between the reconstructed image and the real image is calculated. This loss function can be used alone or in combination with other loss functions (L1 loss, perceptual loss, adversarial loss). The scope of protection should cover any method that uses frequency domain loss (including but not limited to amplitude spectrum loss, phase spectrum loss, power spectral density loss, etc.) for network training in image reconstruction tasks. Furthermore, the specific combination methods of the composite loss functions of this invention (…) , , , The value of and its dynamic adjustment strategy should also be protected to some extent.

[0082] Key Point Six: The Unique Combination of the Overall Network Architecture. This invention protects not only the individual modules but also the overall network architecture formed by combining these modules in a specific order and manner. Specifically, the overall architecture sequentially includes: a shallow feature extraction module → a multi-scale progressive fusion module (multiple modules can be stacked) → an adaptive frequency separation and refinement module → a second-order attention feature refinement module → an iterative feedback optimization module (feedback connected to the output of the shallow feature extraction module) → a high-resolution reconstruction module (sub-pixel convolutional layers). This inter-module connection relationship and data processing flow are crucial components of this invention; any technical solution that copies or substantially similarly imitates this overall architecture should be considered an infringement.

[0083] Key Point Seven: An Efficient Computation Method for Second-Order Attention Based on Low-Rank Approximation. To reduce the computational complexity of second-order attention, this invention proposes a computation method based on low-rank approximation, which retains only the first few rank values ​​of the covariance matrix. The largest eigenvalues ​​and their corresponding eigenvectors ( The complete covariance matrix is ​​then approximated using these principal components, and the attention weights are subsequently calculated. This efficient computational method is a practical innovation of this invention, enabling the deployment of second-order attention mechanisms in real-world applications. The scope of protection should cover any method that uses low-rank approximation, matrix factorization, or random sampling techniques to accelerate the computation of second-order statistics.

[0084] The above seven key points together constitute the core technical system of this invention. When drafting the claims, it is recommended to protect key points one, two, three, and five as independent claims, and key points four, six, and seven as dependent claims or in combination within the independent claims. It is particularly important to emphasize that the innovation of this invention is not a simple stacking of modules, but rather the synergistic effect between the modules that produces… The beneficial effects of this approach are significant. For example, the multi-scale progressive fusion module provides rich multi-scale features to the adaptive frequency separation module, making frequency separation more accurate; the features refined by the second-order attention module are further optimized by the iterative feedback module, forming a positive loop; and the frequency domain loss constraint ensures the spectral realism of the reconstructed image, complementing the perceptual loss and adversarial loss. Therefore, the synergistic relationship and interaction between these modules should be fully reflected in the claims.

[0085] Those skilled in the art can make reasonable substitutions or modifications to certain modules or steps without departing from the core idea of ​​this invention, and can still achieve the purpose of this invention. Several possible alternatives are listed below, and these alternatives should all be considered equivalent embodiments of this invention and fall within the protection scope of this invention.

[0086] Alternative Solution 1: An alternative to frequency separation methods. This invention employs an adaptive frequency separation method based on a learnable Gaussian difference filter. As an alternative, Discrete Wavelet Transform (DWT) or Discrete Cosine Transform (DCT) can be used instead of Gaussian filtering. For example, a two-level discrete wavelet decomposition can be performed on the input feature map to obtain low-frequency subbands (LL) and high-frequency subbands (LH, HL, HH), and then each subband can be processed separately. The basis functions of the wavelet transform are fixed, but an adaptive effect can be achieved by learning the parameters of the post-processing network. Another alternative is to use Laplacian pyramid decomposition, constructing an image pyramid through progressive downsampling and upsampling, with the differences between each level corresponding to information in different frequency bands. Compared to the Gaussian difference method, wavelet transform and Laplacian pyramid have better frequency band separation characteristics, but their computational complexity is slightly higher. In practical applications, a suitable frequency separation method can be selected based on the computing power of the hardware platform.

[0087] Alternative Solution 2: Multi-scale Feature Extraction Alternatives. This invention employs three convolutional branches with different dilation rates to extract multi-scale features. One alternative is to use Spatial Pyramid Pooling (SPP) or Dilated Spatial Pyramid Pooling (ASPP), both widely used in semantic segmentation tasks. ASPP contains multiple dilated convolutions with different dilation rates and a global average pooling branch, capturing a wider range of contextual information. Another alternative is to use a downsampling-upsampling U-Net-style structure, first reducing the resolution through stride convolutions or pooling operations to extract large-scale features, then restoring the resolution through upsampling, and fusing features of different scales through skip connections. This encoder-decoder structure performs well in image translation tasks, but has a large number of parameters. A lightweight alternative is to use multi-scale convolutional kernels (such as...) , , Parallel convolution), and through Convolutional processing is used for fusion. These alternatives can all achieve multi-scale feature extraction and fusion, and those skilled in the art can choose the appropriate solution based on the specific application scenario (such as different requirements for accuracy, speed, and model size).

[0088] Alternative Solution 3: An Alternative for Second-Order Attention Calculation. This invention generates second-order attention weights by calculating the covariance matrix and performing a low-rank approximation. An alternative is to use a Gram matrix instead of the covariance matrix, defined as follows: Its size is ( When the spatial resolution of the feature map is high, the Gram matrix can be large. Another alternative is to use more efficient approximation methods, such as using stochastic Fourier eigenmaps to approximate the Gaussian kernel function, or using the Nyström method for a low-rank approximation of the covariance matrix. Alternatively, moment pooling-based methods can be used, which compute the second (mean of squares) and third moments of the feature map without explicitly computing the complete covariance matrix. These methods capture some high-order statistical information, each with its own computational complexity advantages and disadvantages. On resource-constrained embedded devices, a simplified approach can be adopted: only the variance of each channel (i.e., the diagonal elements of the covariance matrix) is computed, while ignoring the covariance between channels. This approach lies between traditional first-order attention and full second-order attention, representing an effective trade-off.

[0089] Alternative Solution 4: Iterative Feedback Mechanism Replacement. This invention directly adds the feedback signal back to the output of the shallow feature extraction module. One alternative is to connect the feedback signal to multiple different layers of the network, forming a multi-level feedback loop. For example, in addition to connecting to shallow features, the feedback signal can also be connected to the input of each multi-scale progressive fusion module, allowing features at different depths to receive feedback correction. Another alternative is to use a gated feedback mechanism, that is, to determine the proportion by which the feedback signal is fused with the current feature through a learnable gating network, rather than simply adding them. The gating mechanism can be represented as... ,in These are the gating coefficients predicted from the input features. This approach offers a more flexible way of fusing information. Another alternative is to use variants of Recurrent Neural Networks (RNNs) or Long Short-Term Memory Networks (LSTMs) for iterative optimization, treating the network's temporal unfolding as time steps of an RNN. Each time step shares the same network parameters, but the hidden states are passed between different time steps. This parameter-sharing iterative approach can significantly reduce the number of model parameters, but the training difficulty increases accordingly.

[0090] Alternative Option 5: Replacement of the Loss Function. The composite loss function of this invention includes L1 loss, frequency domain loss, perceptual loss, and adversarial loss. One alternative is to use L2 loss instead of L1 loss. Although L2 loss is more sensitive to outliers, its gradient is smoother and training is more stable in some cases. Another alternative is to use only the amplitude spectrum loss in the frequency domain loss and ignore the phase spectrum loss, as experiments show that the phase spectrum loss contributes relatively little to the final result and can be omitted to simplify training. For perceptual loss, more advanced pre-trained networks can be used, such as ResNet-50 or EfficientNet pre-trained on ImageNet, or a perceptual network specifically trained for super-resolution tasks. Adversarial loss can use different discriminator architectures, such as multi-scale discriminators (discriminating between the original resolution and downsampled images simultaneously) or spectral normalization-based discriminators. Furthermore, style loss can be introduced, which computes the difference between the feature map gram matrices to enhance texture realism. Those skilled in the art can select appropriate combinations of loss functions and weight configurations based on the target application scenario (such as prioritizing PSNR or visual effects).

[0091] Alternative Solution Six: Upsampling Methods. This invention employs sub-pixel convolutional layers (PixelShuffle) for upsampling. One alternative is to use transposed convolution (also known as deconvolution), which can achieve upsampling by setting an appropriate stride, but it is prone to "checkerboard" artifacts, usually requiring special weight initialization strategies to mitigate them. Another alternative is to use nearest neighbor or bilinear upsampling followed by ordinary convolution. This approach is simpler to implement and less prone to artifacts, but has a slightly larger number of parameters. Yet another alternative is to use a meta-upscale module, which can handle upsampling at any multiple (including non-integer multiples) by predicting the kernel weights corresponding to each output pixel position to achieve flexible upsampling. For applications requiring support for multiple upsampling multiples, the meta-upscale module is an attractive alternative.

[0092] Alternative Solution Seven: Lightweight Network Replacement. For resource-constrained edge computing devices, this invention can be lightweighted. For example, replacing ordinary convolutions with depthwise separable convolutions can reduce computation by approximately 8-9 times. Replacing the three parallel standard convolutions in the multi-scale progressive fusion module with three parallel depthwise separable convolutions. Replacing the fully connected layer in the second-order attention module with... Convolution can further reduce the number of parameters. Furthermore, redundant channels or layers in the network can be pruned, or knowledge distillation techniques can be used to transfer knowledge from a large model to a smaller one. These lightweight alternatives significantly reduce computational resource requirements while maintaining a certain level of reconstruction quality, making them suitable for deployment on embedded devices such as mobile phones, drones, and security cameras.

[0093] In summary, the above alternative solutions demonstrate that the technical concept of the present invention has broad adaptability and scalability. Those skilled in the art can choose appropriate alternative implementation methods according to the needs of specific application scenarios. None of these alternative solutions exceed the protection scope defined by the claims of the present invention.

[0094] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An image super-resolution reconstruction method based on multi-scale progressive fusion and frequency separation, characterized in that, Includes the following steps: S10, shallow feature extraction; S20, multi-scale progressive fusion; S30, adaptive frequency separation and refining; S40, second-order attention feature refinement; S50, iterative feedback optimization; S60, high-resolution reconstruction.

2. The method according to claim 1, characterized in that, Let the input low-resolution image be ,in and These represent the height and width of the low-resolution image, respectively; the target output is a high-resolution image. ,in , , The upsampling factor is used; S10 includes two consecutive convolutional layers to extract initial shallow features from the input low-resolution image. The first layer uses... The convolutional kernel has 3 input channels, 64 output channels, a stride of 1, and padding of 1. This layer is followed by the LeakyReLU activation function with a negative slope coefficient of 0.

2. Second layer used The convolutional kernel has 64 input and output channels, a stride of 1, and padding of 1. This layer is followed by a LeakyReLU activation function. Let the output of the shallow feature extraction module be... Then the forward propagation process can be represented as: ; in and For convolution kernel weights, and For bias terms, This is the LeakyReLU activation function.

3. The method according to claim 2, characterized in that, S20 includes three convolutional branches with different receptive fields, corresponding to small scale (fine details), medium scale (local structure), and large scale (global context), respectively. The small-scale branch uses... The convolution kernel has a stride of 1 and a dilation rate of 1; this branch is sensitive to local details. The mesoscale branch uses... Convolution kernel, expansion rate 2, receptive field expanded to Large-scale branching Convolution kernel, expansion rate 4, receptive field expanded to Each branch contains two convolutional layers, with batch normalization and LeakyReLU activation function used between them; The output feature maps of the three branches are concatenated along the channel dimension, and then passed through a... The convolutional layer performs channel compression to obtain fused multi-scale features; Let the input feature map be The outputs of the three branches are respectively , and The multi-scale fusion process can then be expressed as: ; ; ; in, This indicates a batch normalization operation. The outputs of the three branches are concatenated along the channel dimension to obtain... ; Then through Convolution for channel fusion: ； in, The number of output channels should be kept consistent with the number of input channels to maintain the feasibility of residual connections; Multiple multi-scale progressive fusion modules are stacked in series, and residual connections are used to promote gradient flow. Let the first... The output of the multi-scale progressive fusion module is Its input is ,but: ; in This represents a multi-scale progressive fusion function, achieved through stacking. Such a modular network refines the feature representation step by step.

4. The method according to claim 3, characterized in that, In step S30, a set of learnable frequency separation filters is first used to estimate the local frequency distribution at each spatial location. A learnable frequency separation convolution kernel is employed, and the input feature map is set as follows: First, calculate two versions with different levels of smoothness: ; ; in and For learnable Gaussian convolution kernels, standard deviation and satisfy , In depthwise separable convolution, high-frequency components are obtained by the difference between two smoothed versions: ; Low-frequency components can be directly taken Then, a spatial adaptive modulation mechanism is introduced, using a lightweight sub-network to predict the optimal separation parameters at each spatial location. This sub-network consists of two convolutional layers: the first layer uses... Convolution, which reduces the number of channels in the input feature map from... Reduced to The second layer uses Convolution outputs two channels, corresponding to... and modulation coefficient and ; Then, the adaptive Gaussian kernel is represented as: ; in and As the initial base value, and For position The modulation coefficient at that point.

5. The method according to claim 4, characterized in that, In step S30, the low-frequency characteristics of adaptive separation are obtained. and high frequency characteristics Subsequently, differentiated processing strategies were adopted for the two. Low-frequency features contain the overall structure and global information of the image and are relatively insensitive to noise, so a lightweight processing method was used: one Convolutional layers are combined with a channel attention mechanism; high-frequency features carry detailed information and edges, requiring higher accuracy and employing a more complex processing method: a multi-branch fine-tuning module containing three parallel sub-branches, each using... , and The convolution kernel reduces computation through separable convolution, and then the outputs of the three sub-branches are fused together; The processing procedure for the low-frequency branch is represented as follows: ; in The channel attention operation is implemented as follows: first, global average pooling is performed on the input feature map to obtain the channel description vector; then, it is passed through two fully connected layers; finally, the sigmoid function is used to map the output to... The interval serves as the channel weight; The processing procedure for high-frequency branches is represented as follows: ; ; in Indicates a channel splicing operation; Finally, the processed low-frequency and high-frequency features are re-fused: 。 6. The method according to claim 5, characterized in that, In S40, the input feature map is set as follows: First, Remodeled into a two-dimensional matrix form ,in Then, calculate the covariance matrix of the feature maps: ; covariance matrix The diagonal elements represent the variance of each channel, reflecting the activation intensity of that channel; the off-diagonal elements represent the covariance between different channels, reflecting the correlation between channels. A normalization operation and a low-rank approximation are introduced for the covariance matrix. First, power normalization is performed on the covariance matrix: ; in, The trace of the matrix, To prevent division by zero errors, power normalization makes the trace of the covariance matrix 1, thus keeping the matrix scale within a reasonable range. The low-rank approximation method is used to decompose the covariance matrix into eigenvalues. The power iteration method is then used to find the principal eigenvectors of the covariance matrix. Let the largest eigenvalue of the covariance matrix be... The corresponding feature vector is The second-order attention weights can be calculated as follows: ; in and It is a fully connected layer. Using ReLU activation function, compute the soft-thresholding representation of the covariance matrix and perform eigenvalue decomposition on the covariance matrix. ,in and , keep before The lowest-rank approximation is obtained by taking the largest eigenvalue and its corresponding eigenvector: ; in , ,Pick That is, retaining 12.5% ​​of the principal components, calculate the second-order attention weights: ; in Representing the eigenvector The element-wise absolute value, then... The weight vector is obtained by averaging the values ​​of each channel. Finally, the second-order attention weights are multiplied by the original feature map: ; in This refers to the Sigmoid function.

7. The method according to claim 6, characterized in that, The S50 receives the currently estimated feature representation and generates a correction signal, which is fed back to the output of the shallow feature extraction layer. The next iteration ( When ), the feature representation of the intermediate layer of the network is as follows: The feedback block generates a feedback signal. ,in It is a lightweight network consisting of two convolutional layers; then the feedback signal is combined with shallow features. Add them together and use the sum as input for the next iteration: ; in For the first The feedback weight coefficients of each iteration are learned during training or set to predefined values ​​that decrease with the number of iterations. Through multiple iterations, the network gradually improves the quality of feature representation. A truncated backpropagation strategy is adopted, that is, the number of iterations is fixed during training. Gradient propagation is performed on all iteration steps; during the inference phase, the number of iterations is flexibly adjusted as needed to achieve a trade-off between computational accuracy and speed. Treating super-resolution reconstruction as an optimization problem, the goal is to minimize the following energy function: ; in For degenerate operators, As a regularization term, iterative optimization updates the solution in the following way: 。 8. The method according to claim 7, characterized in that, S60 defines the feature map after all refining steps as follows: ,in and For the spatial dimensions of the low-resolution feature map; firstly, through a Convolutional layers reduce the number of channels from Convert to : ; Then, subpixel convolution operations are applied to rearrange the feature maps into a high-resolution image: ; The specific implementation of subpixel convolution operation is as follows: the input feature map is... At each spatial location The channels were rearranged into one. Local areas, thereby magnifying spatial resolution. Times, for the location in the output high-resolution image and channels Its pixel value is given by the following formula: 。 9. The method according to claim 8, characterized in that, The S60 also includes a composite loss function that integrates four different types of loss terms: pixel-level fidelity loss, frequency domain spectral loss, perceptual loss, and adversarial loss. Pixel-level fidelity loss is achieved using L1 loss, which is defined as follows: ; in For batch size, Denotes the L1 norm; Frequency domain spectral loss is used to constrain the spectral consistency between the reconstructed image and the true high-resolution image in the frequency domain. A two-dimensional discrete Fourier transform is performed on the image to obtain its amplitude and phase spectra. The frequency domain loss is defined as the weighted sum of the amplitude and phase spectral differences. ; in and These represent the operations for extracting the amplitude spectrum and the phase spectrum, respectively. and To balance the coefficients, a fast Fourier transform algorithm is used, and frequency domain loss is calculated for image blocks. Perceptual loss uses the feature space of the pre-trained VGG-16 network to measure semantic similarity, calculating the Euclidean distance between the feature maps output by the last convolutional layer before the fourth pooling layer of the VGG network: ; in This represents the feature map output of a specified layer in the VGG-16 network. Use a discriminator network to combat loss To distinguish between generated high-resolution images and real high-resolution images, the discriminator network uses a PatchGAN structure and outputs a... The feature map is used, and each output value represents the probability that a local region of the input image is a real image. The adversarial loss is defined as: ; The training loss of the discriminator is: ; The total loss function is the weighted sum of the losses mentioned above: 。 10. The method according to claim 9, characterized in that, Network training is divided into two phases: In the first stage, pre-training is performed using only pixel-level fidelity loss and frequency domain spectral loss, without using adversarial loss. The pre-training learning rate is set to... The batch size is 16, the Adam optimizer is used, and the number of training iterations is 500,000. In the second stage, perceptual loss and adversarial loss are added for joint fine-tuning. During this fine-tuning stage, the generator and discriminator are updated alternately: the generator is updated once, and the discriminator is updated twice. The learning rate during the fine-tuning stage is reduced to a certain value. The training iterations were 200,000 times, and spectral normalization was used to normalize the convolutional layers of the discriminator.

Images