Image super-resolution method of lightweight mixed path model based on content awareness

Abstract

The invention discloses an image super-resolution method of a lightweight mixed path model based on content awareness, and the method comprises the following steps: providing matched recovery paths for different types of image degradation by introducing a grouped multipath strategy; in combination with a content awareness mechanism, processing branches are adaptively selected according to image region complexity, and refined reconstruction of a complex region and efficient processing of a simple region are realized; a progressive calibration module is further adopted, the fusion process of simple and complex region features in the content perception module is optimized, artifacts are reduced, and reconstruction consistency is improved. According to the method, the image super-resolution reconstruction quality in a real complex degraded scene can be remarkably improved on the premise of keeping relatively low calculation overhead, and the method is suitable for the fields of remote sensing image enhancement, medical image analysis, high-definition display and the like.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and digital image processing, specifically to an image super-resolution method based on deep learning technology, and particularly to a lightweight and efficient image super-resolution method for real-world complex degradation scenarios that combines a content-aware mechanism and a hybrid path model. Background Technology

[0002] With the widespread application of image acquisition equipment and the improvement of resolution, the demand for high-quality images continues to grow. However, images acquired in real-world scenarios often degrade to low resolution due to sensor limitations, transmission compression, motion blur, and other reasons, directly affecting the accuracy of subsequent visual analysis and understanding tasks. To address this challenge, image super-resolution technology has emerged, aiming to recover high-resolution details from degraded low-resolution images. This technology has wide applications in fields such as medical image processing, satellite remote sensing, video surveillance, and target detection.

[0003] Since its inception, image super-resolution technology has undergone rapid development from traditional methods to deep learning-based methods. Early methods mainly relied on non-deep learning techniques, such as interpolation-based algorithms (e.g., bilinear interpolation, bicubic interpolation) and reconstruction-based algorithms (e.g., iterative back projection). While these methods are computationally simple and fast, they are limited by their feature representation capabilities and struggle to handle complex degradation scenarios.

[0004] In recent years, the rise of deep learning has brought revolutionary breakthroughs to the field of image super-resolution, with super-resolution methods undergoing a leapfrog development from deep convolutional neural networks (CNNs) to the Transformer architecture. Stacking more convolutional layers and introducing channel / spatial attention mechanisms based on CNNs has significantly improved the model's representational power, achieving substantial progress compared to traditional methods on multiple benchmark datasets. Although convolutional networks are relatively simple and efficient, they are limited by the inherent local receptive field characteristics of CNNs. To overcome the aforementioned problems of CNNs, the Transformer architecture has been introduced into image restoration tasks. Through window self-attention and a hierarchical structure, it can model a longer receptive field, achieving better performance than convolutional networks, and further optimizing the attention mechanism to improve the model's detail recovery ability. However, the high computational cost of the Transformer limits its practical application to some extent. Therefore, how to significantly reduce its computational complexity while maintaining or even improving the powerful restoration capabilities of the Transformer has become a key research challenge.

[0005] Building on this foundation, researchers have begun to focus on more dynamic and adaptive lightweight strategies, with the Mixture of Experts (MoE) mechanism being a prime example. This mechanism achieves a dynamic balance between model capacity and inference cost by sparsely activating some expert subnetworks. For instance, IM-LUT utilizes an expert selection mechanism to optimize the interpolation lookup table structure, achieving a good trade-off between efficiency and expressive power; GLDFN and SSIU, respectively, design lightweight MoE modules from the perspectives of degenerate modeling and structural sensitivity to improve deployability; and CasArbi effectively enhances reconstruction consistency at arbitrary magnification levels through adaptive routing at regional scales.

[0006] Despite this, existing methods still face fundamental challenges in tackling real-world image super-resolution tasks. First, real-world image degradation patterns are extremely complex, diverse, and spatially non-uniform, representing a non-linear superposition of various blurs, noises, and compression artifacts, making them difficult to accurately describe with simple analytical models. Most existing methods, whether based on CNNs or Transformers, are typically trained and optimized under assumed, relatively singular degradation models, resulting in severely insufficient generalization ability when facing complex real-world degradation, and the restored images are prone to artifacts, blurring, or texture distortion. Second, even with the introduction of dynamic sparse computation, existing MoE models often have relatively simple routing mechanisms in super-resolution tasks, failing to fully incorporate the semantic information of the image content to guide path selection. This leads to potentially unoptimal allocation of computational resources, making it difficult to achieve optimal performance restoration under extremely lightweight constraints.

[0007] In summary, developing a super-resolution network architecture that can adapt to image content and efficiently and accurately model complex real-world degradation is crucial for advancing this technology towards practical application. Against this backdrop, this invention proposes a content-aware, lightweight hybrid path model aimed at achieving high-quality, high-efficiency reconstruction of realistically degraded images. Summary of the Invention

[0008] The purpose of this invention is to design an image super-resolution method based on a content-aware, lightweight hybrid path model. This method is capable of handling realistic, complex, and degraded images by adaptively selecting processing paths of varying complexity based on the content complexity and degradation characteristics of different regions of the image. This improves super-resolution reconstruction quality, reduces artifacts, and enhances the consistency and stability of the reconstruction results while controlling computational overhead and model size. The proposed image super-resolution method based on a content-aware, lightweight hybrid path model mainly includes the following steps: (1) Construct a content-aware hybrid path super-resolution network model, which can adaptively select computational paths of different complexities for different regions based on the local content complexity and content features of the input image. (2) The network model is trained under supervision using a real degraded image pairing dataset to learn the mapping relationship from low resolution to high resolution.

[0009] Compared with the prior art, the present invention has the following significant beneficial technical effects: (1) By setting up a grouped multi-path structure, the present invention enables the network to match different processing paths for different types and levels of degradation features, which is beneficial to improving the modeling ability of real complex degradation. (2) By introducing a content-aware path selection mechanism, the present invention enables the network to adaptively allocate computing resources according to the complexity of local content in the image, enhance feature representation in complex regions, and adopt a lighter processing path in relatively simple regions, thereby balancing reconstruction quality and inference efficiency. (3) By setting up a progressive content-aware fusion module, the present invention performs coarse and fine calibration on different path features, which can reduce statistical mismatch between different branches, reduce local brightness and darkness drift and edge artifacts, and improve the consistency and visual quality of reconstruction results. (4) While maintaining the overall lightweight model, this invention can improve the super-resolution reconstruction performance of real degraded images and is suitable for application scenarios such as remote sensing image enhancement, medical image analysis, video surveillance and high-definition display. (5) According to the experimental results in the specification, the present invention has achieved better objective evaluation index on the real image super-resolution test set, indicating that the present invention has a better reconstruction effect on real complex degradation scenes. Attached Figure Description

[0010] Figure 1 This is a block diagram illustrating the overall architecture of the super-resolution method based on a content-aware lightweight hybrid path model proposed in this invention.

[0011] Figure 2 This is a schematic diagram of the specific structure of the grouped multi-path construction in this invention.

[0012] Figure 3 This is a diagram of the internal structure of the content adjustment node in this invention.

[0013] Figure 4 This is a schematic diagram illustrating the working principle of the path selection mechanism in this invention.

[0014] Figure 5 This is a detailed structural diagram of the Progressive Content-Aware Fusion Module (PCAF) in this invention.

[0015] Figure 6 A bar chart comparing the ablation experimental performance under different module configurations in the invention. Detailed Implementation

[0016] The present invention will be further described below with reference to the accompanying drawings: Figure 1 In this paper, the image super-resolution method based on a content-aware lightweight hybrid path model can be divided into the following steps: (1) Construct a content-aware hybrid path super-resolution network model; (2) Using the network model constructed in step (1), the input low-resolution image to be processed is reconstructed and a high-resolution image is output.

[0017] Specifically, in step (1), the overall architecture of the network model follows the workflow of "shallow feature extraction → progressive content-aware fusion → multi-path node feedforward → dynamic path routing → high-resolution reconstruction", which takes into account both expressive power and computational efficiency in real complex degradation scenarios.

[0018] Given a low-resolution input image, shallow features are first obtained through a shallow feature extraction module. These features then flow into a deep feature extraction section, which consists of multiple cascaded Transformer modules. Each module contains: a Progressive Content-Aware Fusion (PCAF) module that progressively refines features based on content complexity through coarse and fine calibration stages; and a Modulation Feedforward Network-based Path Node Type Architecture (MFFN), which internally sets up multiple parallel paths and combines content awareness and normalization operations, outputting intermediate representations after layer normalization. Finally, the features are processed by a high-resolution image reconstruction module to output a high-resolution result.

[0019] In parallel with the backbone, the predictor analyzes the input and generates a mask, which is then fed into the Path Routing Network (PRN) after pooling. The PRN selects appropriate Top-K path nodes for different regions based on image characteristics, activating more path nodes in complex regions to enhance expressiveness, while using lightweight paths in simple regions to improve efficiency. Finally, the deep features are upsampled and reconstructed through convolutional layers and pixel rearrangement operations, outputting a high-resolution result.

[0020] A. Constructing a content-aware hybrid path super-resolution network model – multi-path content conditioning control: 1) Grouped multipath structure. For example... Figure 2As shown, at the path level, the original single FFN is upgraded to a grouped multi-path structure. The backbone is divided into several path groups according to network depth. Each group contains several path nodes with different functions (which can be regarded as FFN branches of different styles / capacities), and different numbers and structures of nodes are configured for each group. This design of "lighter shallow paths and stronger deep paths" enables the network to provide matching modeling paths for different types and complexities of degradation patterns at different levels, realizing on-demand capacity deployment in the depth dimension and structurally addressing the diversity and hierarchy of degradation.

[0021] 2) Content adjustment node construction. For example... Figure 3 As shown, at the branch selection level, a content adjustment control mechanism is introduced, enabling the network to adaptively select branches of different complexities based on local content, achieving a dynamic balance between reconstruction quality and inference efficiency. Given input features... Gated network (Based on content features and degradation-related features) (Commonly driven) outputs the weight vector of each path node. : Top-k sparse selection is used during the training phase: in For the first Each path node corresponds to an FFN branch with different complexities / inductive biases. This indicates selecting the largest value from the input vector. The index of each element is used. By forwarding and backwarding through only a small number of nodes in the set, diverse representations are preserved while maintaining training stability and efficiency. During the inference phase, Top-1 is used, where only a single path node is activated at each position, significantly reducing computational cost.

[0022] 3) Path selection mechanism. For example... Figure 4As shown, the path routing mechanism proposed in this section follows a main framework of "dual-source statistics—light projection—temperature-controlled normalization—sparse selection—weighted fusion": On one hand, contextual features and content features respectively extract and calculate three types of statistical features: maximum / min / mean, forming a stable description of degradation intensity and content complexity; on the other hand, convolution is used to map six statistical channels to the path node dimension, combined with temperature-controlled normalization to obtain the probability distribution of location-dependent path nodes, and activated during training to reduce invalid computation. Finally, the outputs of the selected path nodes are linearly weighted and fused, so that computational resources are automatically concentrated on the most relevant path nodes, avoiding the discontinuity caused by hard switching, and balancing expressiveness, stability, and efficiency. This router works in conjunction with the subsequent "complex / simple paths" of PCAF and gated FFN to strengthen contextual modeling in complex degradation regions and maintain lightweight reconstruction in simple regions.

[0023] Operation 1: Dual-Source Statistical Sensing. The router simultaneously receives two representations: an upper branch from PCAF and a lower branch from the predictor's content features. The two features are statistically analyzed at the window / pixel scale using three categories: max / min / mean. After aligning the two statistical analyses, they are merged along the channel dimension to obtain the final result. The six statistical channels, among which B This indicates the number of data samples passed to the program for training in a single session. H Image height, W This represents the image width.

[0024] Operation 2: Concatenation, fusion, and projection onto the path node dimension. After concatenating the six statistical channels along the channel dimension, non-linear fusion and dimensionality enhancement are achieved through two levels of 1×1 convolutions: .in For the baseline number of channels, This represents the number of path nodes. This step corresponds to " The mapping of "".

[0025] Operation 3: Normalization and Sparsity Selection. For the obtained original weight tensor... Execute based on path node dimension Function normalization: This yields the path node weight distribution for each location. , Indicates spatial location.

[0026] Operation 4: Weighted fusion of path node outputs. Let the... The output of each path node is Stack all path nodes into Multiply by the gate weights pixel by pixel and sum them: This fusion can be integrated with PCAF. , Contextual branches, in conjunction with the "heavy / light paths" of gated FFN, can focus computing power on the most relevant path nodes and complex regions, thereby achieving efficient, stable, and generalizable super-resolution reconstruction of real-world scenes.

[0027] B. Constructing a content-aware hybrid path super-resolution network model—Progressive calibration for content-aware hybrid paths. like Figure 5 As shown, to expand the context without increasing query overhead, an asymmetric window and a learnable offset are used: maintaining the query... Small window, zoom in key / value , The window radius is determined, and the offset is predicted by the predictor, with alignment achieved using bilinear resampling. Based on this, the channel attention mixer is restructured into a two-stage "pre-coarse calibration – post-fine calibration" structure to reduce statistical mismatch between the two sub-paths and suppress artifacts.

[0028] 1) Feature coarse calibration. Maintain [this calibration] before entering the PCAF backbone. Small window, zoom in , With the asymmetric window and learnable offset settings unchanged, the learnable offset is generated by the predictor. This allows for the use of the warp function to align features through spatial deformation. To reduce the statistical mismatch between the light and heavy processing paths, the features of the light path are first processed... Contextual reference Window-level mean-variance alignment for the baseline Thus, the coarse calibration result is obtained. ,in It is the mean. It is the standard deviation. It is a scaling factor. It is the offset.

[0029] Subsequently Parallel modeling of sparse attention and convolutional branches; rectified gating (Generated by the statistical branch of the predictor) Determines the proportion of complex / simple regions within the block, thereby aggregating more distant contexts and stabilizing subsequent fusions without increasing query overhead.

[0030] 2) Fine-tuning of features. After the backbone calculation is completed, to further eliminate local brightness shifts and edge narrowband artifacts, fine-tuning is added to the output: This is performed on the light path residuals. Depthwise separable convolutions with learnable fine-tuning of channel scaling / bias, followed by affine backinjection, yield... : Finally, during the fusion, As a spatially adaptive mask, the outputs of the two parallel paths are weighted and summed to form the final output: in For the output of the attention path, This is the output after fine calibration.

[0031] C constructs a super-resolution model based on a content-aware, lightweight hybrid path model. The purpose of this invention is to construct a lightweight hybrid path super-resolution model framework based on a content-aware mechanism to reconstruct low-resolution images of the real world more intelligently and efficiently. This framework follows a processing flow of "shallow feature extraction → deep feature extraction (content-aware hybrid path processing) → high-resolution reconstruction". In the deep feature extraction stage, the network analyzes the complexity of the input image content through a parallel content predictor and drives the path routing network to dynamically select and activate the most relevant node paths in the multi-path feedforward network. Simultaneously, a progressive content-aware fusion module is used to finely calibrate and fuse features, thereby adaptively allocating appropriate computational resources to different degradation regions in the image, achieving "on-demand computation". Finally, while maintaining the overall model's lightweight nature, high-quality and high-efficiency super-resolution reconstruction of complex real-world degraded images is achieved.

[0032] In step (2), high- and low-resolution image pairing data from real-world degradation datasets are used, and the aforementioned network model is used as the architecture. The model parameters are trained through supervised learning, thereby significantly improving its reconstruction performance and generalization ability in real image super-resolution tasks.

[0033] To better illustrate the effectiveness of this invention, comparative experiments were conducted on commonly used real-world image super-resolution test sets "Real-Canon", "Real-Nikon", and "DRealSR". Four typical image super-resolution algorithms were selected as comparison methods in the experiments.

[0034] The super-resolution algorithm being compared is: Method 1: EDSR, reference "Lim-B, Son-S, Kim-H, et al. Enhanced deepresidual networks for single image super-resolution[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2017:136-144." Method 2: RCAN, reference "Zhang Y, Li K, Li K, et al. Image super-resolution using very deep residual channel attention networks[C] / / Proceedings of the European conference on computer vision (ECCV). 2018: 286-301." Method 3: MWDIN, reference "Wang S, Liu G, Liu X, et al. Efficient image super resolution via Mixed Window and Dimension Interaction[J].Neurocomputing, 2025, 620: 129211." Method 4: CAMixer, reference "Wang Y, Liu Y, Zhao S, et al. CAMixerSR:Only Details Need More" Attention"[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024: 25837-25846." The comparative experiment is as follows: Experiment 1 used methods 1 to 4, as well as this invention, to perform 4x, 3x, and 2x super-resolution reconstructions on low-resolution images in the commonly used test sets "RealSR-Canon", "RealSR-Nikon", and "DRealSR". The objective evaluation parameters for the super-resolution reconstruction results are shown in Table 1. The objective evaluation parameters used are Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). Higher PSNR and SSIM values ​​indicate better reconstructed image quality.

[0035] Table 1 Clearly, in terms of quantitative evaluation, this invention, compared with other methods, performs better across all magnifications in the three datasets. , , It achieved optimal performance on all evaluation metrics (PSNR, SSIM).

[0036] Experiment 2, to verify the actual contributions of each core module in this invention, such as... Figure 6 As shown, in RealSR-Nikon ( A systematic ablation experiment was conducted on the dataset. The objective evaluation parameters of the actual contribution results of each core module in this invention are shown in Table 2. The objective evaluation parameters are Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). The higher the PSNR and SSIM values, the better the quality of the reconstructed image under the given conditions.

[0037] Table 2 Experiments show that: (1) The contribution of the multi-path node module: The introduction of the multi-path node mechanism alone can bring about a PSNR improvement of about 0.08dB, which verifies the effectiveness of the sparse path node mechanism for modeling diverse degradation. (2) The contribution of the content-aware module: The introduction of content-aware routing alone can bring about a PSNR improvement of about 0.03dB. This module provides key guidance for dynamic path selection. (3) The contribution of the progressive calibration strategy: Both coarse calibration and fine calibration have slight improvements when used alone, while the complete progressive calibration strategy combining the two contributes the main performance gain, proving its effectiveness in eliminating feature mismatch and suppressing artifacts. (4) The contribution of the path node distribution strategy: The default "front-heavy, back-light" path node distribution strategy (i.e., deploying more path nodes in the shallow layer) achieves the best performance, which is better than uniform distribution and intermediate centralized distribution. This verifies the design assumption of this invention: shallow networks need more diverse path nodes to handle the varied low-level degradation features, while deep semantic features are relatively abstract and the requirement for path nodes is reduced. (5) Synergistic effect: The PSNR of the complete model is improved by 0.16 dB compared with the baseline, which is slightly higher than the sum of the improvements of only the multi-path node module and only the content-aware module. This indicates that there is a positive synergistic effect between the multi-path node module, the content-aware module and the progressive calibration module, which together achieve efficient and high-precision reconstruction of real complex degradation scenarios.

[0038] In summary, compared to the comparative methods, the content-aware lightweight hybrid path model proposed in this invention has significant advantages in the objective evaluation of reconstruction results. Therefore, this invention is an image super-resolution method that can effectively address complex real-world degradation while balancing high performance and high computational efficiency.

Claims

1. An image super-resolution method based on a content-aware lightweight hybrid path model, characterized in that... Includes the following steps: Step 1: Construct a content-aware hybrid path super-resolution network model. The network model can adaptively select computational paths of different complexities for different regions based on the local content complexity and content features of the input image. Step 2: Supervised training of the network model using a real degraded image pairing dataset, enabling it to learn the mapping relationship from low resolution to high resolution.

2. The image super-resolution method based on a content-aware lightweight hybrid path model according to claim 1, characterized in that... Step one describes the construction of a content-aware hybrid path super-resolution network model. The overall architecture of this network model follows a processing flow of "shallow feature extraction → progressive content-aware fusion → multi-path node feedforward → dynamic path routing → high-resolution reconstruction," outputting high-resolution results. Furthermore, the model employs a grouped multi-path strategy, which dynamically routes the input degraded image to the corresponding restoration path. A progressive content-aware feature extraction module (PCAF) is introduced during the content-aware fusion process to assist in selecting decision paths and achieve differentiated processing of complex and simple regions. A unified lightweight architecture is also constructed to significantly reduce the number of model parameters.

3. The image super-resolution method based on a content-aware lightweight hybrid path model according to claim 1, characterized in that... The construction of the content-aware hybrid path super-resolution network model described in step one involves a grouped multi-path strategy in the multi-path node feedforward step. This strategy divides the backbone into several path groups according to the network depth. Each group is configured with multiple path nodes with the same structure but different model parameters. The number of nodes in the path group satisfies the hierarchical distribution of N1≥N2≥N3. The design principle of "shallow paths are lighter and deep paths are stronger" is adopted, so that the network can provide matching modeling paths for different types and complexities of degradation modes at different levels, realize the on-demand deployment of capacity in the depth dimension, and structurally address the diversity and hierarchy of degradation.

4. The image super-resolution method based on a content-aware lightweight hybrid path model according to claim 1, characterized in that... Step one describes the construction of a content-aware hybrid path super-resolution network model. This model involves constructing content adjustment nodes in the multi-path node feedforward step and introducing a content adjustment control mechanism at the branch selection level. This allows the network to adaptively select branches of varying complexity based on local content, achieving a dynamic balance between reconstruction quality and inference efficiency. Given input features... Gated network (Based on content features and degradation-related features) (Commonly driven) outputs the weight vector of each path node. : Top-k sparse selection is used during the training phase: in For the first Each path node corresponds to an FFN branch with different complexities / inductive biases. This indicates selecting the largest value from the input vector. The index of each element is obtained by forwarding and backward propagating only a small number of nodes in the set, which preserves the diversity of representations and maintains the stability and efficiency of training. In the inference stage, it switches to Top-1, where only a single path node is activated at each position, which significantly reduces the amount of computation.

5. The image super-resolution method based on a content-aware lightweight hybrid path model according to claim 1, characterized in that... Step one describes the construction of a content-aware hybrid path super-resolution network model. During dynamic path routing, a routing selection mechanism is employed, based on the principle of "dual-source statistics—light projection—temperature-controlled normalization—sparse selection—weighted fusion." On one hand, contextual features and content features extract the maximum, minimum, and average statistical features respectively, forming a stable description of degradation intensity and content complexity. On the other hand, convolution maps six statistical channels to the node dimension, combined with temperature-controlled normalization to obtain the probability distribution of location-dependent path nodes, and activates this during training to reduce invalid computation. Finally, the outputs of the selected nodes are linearly weighted and fused, automatically concentrating computational resources on the most relevant nodes, avoiding discontinuities caused by hard switching, and balancing expressiveness, stability, and efficiency. This router works in conjunction with the subsequent PCAF and gated FFN "complex / simple paths," strengthening contextual modeling in complex degradation regions and maintaining lightweight reconstruction in simple regions.

6. The image super-resolution method based on a content-aware lightweight hybrid path model according to claim 1, characterized in that... The construction of the content-aware hybrid path super-resolution network model described in step one; wherein the progressive content-aware fusion module PCAF, based on asymmetric windows and learnable offsets, restructures the channel attention mixer into a two-stage structure of "pre-coarse calibration - post-fine calibration": Step 1: Feature coarse calibration: Before entering the PCAF backbone, maintain Small window, zoom in , With the asymmetric window and learnable offset settings unchanged, the learnable offset is generated by the predictor. Then, the warp function is used to align the features through spatial deformation. To reduce the statistical mismatch between the light and heavy processing paths, the features of the light path are first aligned. Contextual reference Window-level mean-variance alignment is performed based on the baseline. To obtain coarse calibration results , It is the mean. It is the standard deviation. It is a scaling factor. It is the offset; Subsequently Parallel modeling of sparse attention and convolutional branches; rectified gating. The proportion of complex or simple regions is determined within a block, thereby aggregating more distant contexts and stabilizing subsequent fusion without increasing query overhead. Generated by the statistical branch of the predictor, it is a spatially adaptive gated weight graph; Step 2: Feature Fine-Calibration: After the backbone calculation is completed, to further eliminate local brightness and darkness shifts and edge narrowband artifacts, fine-calibration is added at the output. This involves performing a 3×3 depthwise separable convolution and learnable fine-tuning of channel scaling / bias on the light path residuals, and back-injecting the residuals to obtain... : Finally, during the fusion, As a spatially adaptive mask, the outputs of the two parallel paths are weighted and summed to form the final output: in For the output of the attention path, This is the output after fine calibration.

Images