A cross-domain applicable lightweight super-resolution reconstruction method
By constructing a lightweight super-resolution reconstruction network and combining local and non-local feature exploration branches, the problems of high model complexity and poor cross-domain adaptability in existing technologies are solved, achieving efficient image reconstruction and cross-domain applicability, suitable for resource-constrained devices.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIAN INST OF OPTICS & PRECISION MECHANICS CHINESE ACAD OF SCI
- Filing Date
- 2025-08-06
- Publication Date
- 2026-05-15
AI Technical Summary
Existing super-resolution technologies suffer from insufficient local information modeling capabilities, high model complexity, and poor cross-domain adaptability, leading to resource waste and inefficiency, and making them difficult to deploy on resource-constrained devices.
We design a lightweight super-resolution reconstruction method applicable across domains. By constructing a network model that includes a dual-branch feature extraction and aggregation module and a parallel feature enhancement module, and combining local and non-local feature exploration branches, we adopt an efficient module structure and information selection mechanism to achieve lightweight model and cross-domain adaptability.
It improves reconstruction accuracy and visual quality, reduces the number of model parameters and computational complexity, and has good cross-domain generalization ability, making it suitable for resource-constrained devices such as mobile devices, UAV image processing, and satellite remote sensing image enhancement.
Smart Images
Figure CN121235904B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of super-resolution reconstruction, specifically relating to a lightweight super-resolution reconstruction method applicable across fields. Background Technology
[0002] Image super-resolution (SR) is an image processing technique in the field of computer vision. It is the process of reconstructing a high-resolution image (HR) with better visual quality and detail from one or more low-resolution images (LR) with coarse details.
[0003] To fully utilize the multi-layered information in images, image super-resolution reconstruction methods have gradually evolved towards more complex network structures in recent years. Many studies have attempted to combine convolutional neural networks (CNNs) with self-attention mechanisms (such as Transformers) to achieve joint modeling of local details and global contextual information.
[0004] However, to further improve model performance, these methods often employ stacking a large number of modules or introducing multi-layered attention mechanisms, continuously increasing the depth and width of the network. While achieving some improvement in reconstruction accuracy, this results in a surge in the number of network parameters, a massive model size, and high computational complexity in the inference process. This not only leads to significant memory and GPU memory consumption, reducing image processing efficiency, but also limits the practical deployment capability of the model on resource-constrained devices (such as mobile devices, edge computing platforms, and remote sensing payload platforms), resulting in wasted computing resources and limited application scenarios.
[0005] To fully utilize both local and non-local features, many scholars have developed a series of networks that combine local and non-local feature extraction, proposing different modules to optimize the feature extraction process. Sen et al. proposed a local-non-local combined network, LGCNet (Super-Resolution for Remote Sensing Images via Local–Global Combined Network), to improve the representational power of SR models. Subsequently, more and more scholars began to study the impact of combining local and non-local features on model capabilities. Hou et al. used stacked local and non-local residual blocks for non-local and local feature extraction of images (Image Super-Resolution Reconstruction Method Based on Global and Local Residual Learning). To enhance edge detail information in images, Li et al. designed a local-non-local context-aware generative adversarial network (LGC-GDAN), utilizing a dual-region discriminator and generator to enhance edge detail information (Local-Global Context-Aware Generative Dual-Region Adversarial Networks for Remote Sensing Scene Image Super-Resolution). However, these models typically consume significant amounts of memory and computational resources, which not only reduces image processing efficiency but also leads to resource waste. Furthermore, many models are suitable for super-resolution remote sensing images, but their effectiveness is not significant when applied to natural images.
[0006] To reduce the number of parameters in models, many researchers have designed lightweight models. For example, Hui et al. proposed IMDN, which uses multi-distillation information networks to refine and distill features, saving computation. Liu et al. proposed the RFDN architecture and added residual blocks to achieve more refined feature extraction and utilization through feature distillation, while reducing model complexity and the number of parameters. Although these models adopt a compact architecture and achieve better visual results, they struggle to fully utilize local and non-local information and find it difficult to balance model size and performance. Furthermore, the complexity and computational cost of using these models for image super-resolution reconstruction remain high, making it difficult to meet the demand for lightweight designs. Summary of the Invention
[0007] The purpose of this invention is to address the significant limitations of existing super-resolution technologies, such as insufficient local information modeling capabilities, high model complexity, and poor cross-domain adaptability, and to provide a lightweight super-resolution reconstruction method applicable across multiple domains.
[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0009] A lightweight super-resolution reconstruction method applicable across multiple domains, characterized by the following steps:
[0010] Step 1: Obtain the dataset, which includes natural image data and remote sensing image data; the dataset is divided into a training set and a test set;
[0011] Step 2: Construct a super-resolution reconstruction network model. This model includes a first convolutional layer, multiple feature aggregation blocks, a first adder, a second convolutional layer, and a pixel reconstruction layer connected in sequence. The input of the first convolutional layer is connected to the second input of the first adder, serving as the input of the super-resolution reconstruction network model. The feature aggregation block includes a dual-branch feature extraction and aggregation module, a second adder, and a parallel feature enhancement module connected in sequence. The input of the dual-branch feature extraction and aggregation module is connected to the second input of the second adder, serving as the input of the feature aggregation block. The output of the parallel feature enhancement module serves as the output of the feature aggregation block.
[0012] Step 3: Input the training set into the super-resolution reconstruction network model for training to obtain the trained super-resolution reconstruction network model;
[0013] Step 4: Input the test set into the trained super-resolution reconstruction network model to achieve cross-domain super-resolution reconstruction.
[0014] Furthermore, in step 2, the dual-branch feature extraction and aggregation module includes a third convolutional layer, a channel splitting layer, a local feature exploration branch, a non-local feature exploration branch, a third adder, and a fourth convolutional layer;
[0015] The input of the third convolutional layer is the input of the dual-branch feature extraction and aggregation module, and the output is connected to the input of the channel splitting layer.
[0016] The inputs of the local feature exploration branch and the non-local feature exploration branch are connected to the two outputs of the channel splitting layer, respectively, and the outputs are connected to the two inputs of the third adder, respectively.
[0017] The output of the third adder is connected to the input of the fourth convolutional layer, and the output of the fourth convolutional layer is the output of the dual-branch feature aggregation module.
[0018] Furthermore, in step 2, the local feature exploration branch includes a normalization layer, a first GELU layer, a partial convolutional channel mixer, a fourth adder, and a fifth convolutional layer connected in sequence.
[0019] The input of the normalization layer and the second input of the fourth adder serve as the input of the local feature exploration branch, connecting to the first output of the channel splitting layer; the output of the fifth convolutional layer serves as the output of the local feature exploration branch, connecting to the first input of the third adder.
[0020] The nonlocal feature exploration branch consists of an adaptive max pooling layer, a deep convolutional layer, an approximately depthwise separable convolutional block, a fifth adder, a sixth convolutional layer, a second GELU layer, and a multiplier, connected in sequence.
[0021] The second input of the fifth adder is connected to the second input of the channel separation layer via the global variance calculation unit, the second input of the multiplier, the input of the adaptive max pooling layer, and the second input of the channel separation layer; the output of the multiplier is connected to the second input of the third adder.
[0022] Furthermore, the first and second convolutional layers are both 3×3 convolutional layers, while the third, fourth, fifth, and sixth convolutional layers are all 1×1 convolutional layers.
[0023] Furthermore, in step 2, the parallel feature enhancement module includes a branch that preserves the original information and a local information enhancement module;
[0024] The original information is preserved by sequentially connecting the seventh convolutional layer, the third GELU layer, the cascaded layer, the eighth convolutional layer, and the sixth adder;
[0025] The input of the seventh convolutional layer and the second input of the sixth adder serve as the input of the parallel feature enhancement module, and are connected to the output of the first adder; the output of the sixth adder is the output of the parallel feature enhancement module.
[0026] The input of the local information enhancement module is connected to the output of the third GELU layer, and the output is connected to the second input of the cascaded layer.
[0027] Furthermore, the local information enhancement module includes a ninth convolutional layer, a tenth convolutional layer, and a fourth GELU layer connected in sequence;
[0028] The input of the ninth convolutional layer is connected to the output of the third GELU layer, and the output of the fourth GELU layer is connected to the second input of the cascaded layer.
[0029] Furthermore, the seventh, eighth, and tenth convolutional layers are all 1×1 convolutional layers, and the ninth convolutional layer is a 3×3 convolutional layer.
[0030] The beneficial effects of this invention are:
[0031] 1. Synergistic Improvement in Performance and Efficiency: This invention fully integrates local and non-local feature modeling mechanisms (local feature exploration branch and non-local feature exploration branch) in its overall architecture, effectively improving the model's ability to recover texture details, edge information, and structural features. Experiments on multiple authoritative datasets demonstrate that this method significantly outperforms existing mainstream methods in mainstream image quality evaluation metrics such as PSNR and SSIM, exhibiting stronger reconstruction accuracy and visual quality.
[0032] 2. Lightweight Model Structure Design: This invention introduces an efficient modular structure and information selection mechanism, achieving a significant reduction in model parameters and computational load while maintaining or even improving reconstruction performance. Compared to traditional deep network structures, the proposed model effectively reduces both the number of parameters and FLOPs, resulting in lower storage and computational consumption, meeting the real-time operation requirements of edge computing devices. This invention utilizes the characteristics of reference depth separable convolutions to reduce computational redundancy and achieve lightweight design. Furthermore, the parallel structure within the parallel feature enhancement module can process local information while preserving original information, ensuring image reconstruction quality while achieving a lightweight model.
[0033] 3. Excellent cross-domain generalization ability: This invention incorporates a task-independent feature extraction strategy (preserving the original information branch) during the design process, significantly enhancing the model's adaptability to different image scenarios. Experimental results show that this model not only exhibits excellent performance in remote sensing image super-resolution tasks but also demonstrates high transferability and robustness in natural image applications, overcoming the overfitting problem of traditional models in specific domains.
[0034] 4. High practical application and promotion value: With its excellent performance and lightweight architecture, this invention has broad application prospects in resource-constrained or real-time-critical scenarios such as mobile devices, UAV image processing, satellite remote sensing image enhancement, and medical image preprocessing. Attached Figure Description
[0035] Figure 1 This is a schematic diagram of the structural principle of the super-resolution reconstruction network model in this embodiment of the invention;
[0036] Figure 2 This is a schematic diagram of the principle of the dual-branch feature extraction and aggregation module in this embodiment of the invention;
[0037] Figure 3 This is a schematic diagram illustrating the principle of the parallel feature enhancement module in this embodiment of the invention. Detailed Implementation
[0038] To make the objectives, advantages, and features of the present invention clearer, the following detailed description of a lightweight super-resolution reconstruction method applicable across various fields, in conjunction with the accompanying drawings and specific embodiments, will further illustrate the advantages and features of the present invention. The advantages and features of the present invention will become clearer from the following detailed embodiments.
[0039] See Figure 1 The lightweight super-resolution reconstruction method applicable across domains in this embodiment mainly includes a first convolutional layer (Conv), multiple feature aggregation blocks, a first adder, a second convolutional layer, and a pixel reconstruction layer connected in sequence; the input end of the first convolutional layer is connected to the second input end of the first adder, which together serve as the input of the super-resolution reconstruction network model.
[0040] In this embodiment, both the first and second convolutional layers are 3×3 convolutional layers. In this embodiment, the number of feature aggregation modules is set to 8.
[0041] The feature aggregation block comprises a dual-branch feature extraction and aggregation module, a second adder, and a high-efficiency parallel feature enhancement module connected in sequence. The input of the dual-branch feature extraction and aggregation module is connected to the second input of the second adder, serving as the input of the feature aggregation block. Residual connections are made through the dual-branch feature extraction and aggregation module, and then it is connected in series with the parallel feature enhancement module to further enhance the information flow.
[0042] See Figure 2 The dual-branch feature extraction and aggregation module includes a third convolutional layer, a channel split layer, a local feature exploration branch, a non-local feature exploration branch, a third adder, and a fourth convolutional layer.
[0043] In this embodiment, both the third and fourth convolutional layers are 1×1 convolutional layers. The input of the third convolutional layer is the input of the dual-branch feature extraction and aggregation module, and the output is connected to the input of the channel splitting layer. The channel splitting layer processes the input feature map F. in The process is divided into two branches. The inputs of the local feature exploration branch and the non-local feature exploration branch are connected to the two outputs of the channel splitting layer, respectively. The local feature exploration branch enhances local modeling capabilities, while the non-local feature exploration branch strengthens non-local information extraction capabilities. The outputs of the local and non-local feature exploration branches are connected to the two inputs of the third adder, which fuses the features from the two branches to refine the detailed information. The output of the third adder is connected to the input of the fourth convolutional layer, and the output of the fourth convolutional layer is the output of the dual-branch feature aggregation module.
[0044] Specifically, the local feature exploration branch consists of a normalization layer (Layer Norm), a first GELU (activation function) layer, a partially convolutional channel mixer, a fourth adder, and a fifth convolutional layer connected in sequence.
[0045] The input of the normalization layer and the second input of the fourth adder serve as the inputs of the local feature exploration branch, connecting to the first output of the channel splitting layer. The model training is stabilized through the normalization layer. The fifth convolutional layer is specifically a 1×1 convolutional layer used for deep feature extraction. Its output serves as the output of the local feature exploration branch, connecting to the first input of the third adder.
[0046] Meanwhile, in the local feature exploration branch, a partial convolutional channel mixer is used to better preserve image details and edge information. Its internal structure consists of a convolutional channel mixer (CCM), GELU, and PConv (partial convolution) connected in sequence. Specifically, PConv's main function is to reduce unnecessary computation and improve parameter utilization in local information extraction, while also enhancing texture reconstruction capabilities. By effectively combining PConv and CCM, not only is network complexity reduced, but the reconstructed image is also very clear and detailed, better preserving image details and edge information, thus improving the overall image quality and visual effect.
[0047] The non-local feature exploration branch consists of an adaptive max-pooling layer, a depthwise convolutional layer (DWConv), an approximately depthwise separable convolutional block, a fifth adder, a sixth convolutional layer, a second GELU layer, and a multiplier, connected in sequence. The sixth convolutional layer is a 1×1 convolutional layer. The second input of the fifth adder is connected to the second input of the channel separation layer via the global variance calculation unit, the second input of the multiplier, and the input of the adaptive max-pooling layer, respectively; the output of the multiplier is connected to the second input of the third adder.
[0048] The approximate depth-separable convolutional block consists of a series of interconnected 1×1 convolutional layers, GELU, and DWConv. The 1×1 convolutional layers facilitate information exchange between channels, followed by GELU to enhance nonlinearity and improve model fitting ability. Finally, depthwise convolution is performed to extract more detailed spatial information while reducing computational cost. This approximate depth-separable convolutional block is then merged with global variance calculation to further improve global modeling while maintaining a lightweight model.
[0049] The nonlocal feature exploration branch utilizes global variance calculation to enhance the exploration of nonlocal information and employs max pooling to obtain low-frequency information from the image. This information is then input into DWConv and approximately depthwise separable convolutional blocks to further improve feature representation capabilities. The output features from the two branches are summed and then passed through a fourth convolutional layer to serve as the final output F of the dual-branch feature extraction and aggregation module. a ′ .
[0050] It should be noted that in this embodiment... Figure 2 and Figure 3 The blue cubes of the same size in the diagram all represent feature maps, while Figure 2 The smaller blue cube representing the feature at the second input of the fifth adder is input to the fifth adder after global variance calculation.
[0051] See Figure 3 The efficient parallel feature enhancement module, inspired by feedforward neural networks, is improved to allow the model to effectively utilize image information and achieve a lightweight design. Specifically, the parallel feature enhancement module consists of a seventh convolutional layer, a third GELU layer, a cascaded layer (Channel Concatenation), an eighth convolutional layer, and a sixth adder, all connected in sequence, as well as a local information enhancement module. The seventh and eighth convolutional layers are specifically 1×1 convolutional layers. The seventh convolutional layer, the third GELU layer, the cascaded layer, the eighth convolutional layer, and the fifth adder form a branch that preserves the original information.
[0052] The input of the seventh convolutional layer and the second input of the sixth adder serve as the input of the parallel feature enhancement module and are connected to the output of the first adder; the output of the sixth adder is the output of the parallel feature enhancement module; the input of the local information enhancement module is connected to the output of the third GELU layer, and the output is connected to the second input of the cascaded layer.
[0053] The local information enhancement module includes a ninth convolutional layer, a tenth convolutional layer, and a fourth GELU layer connected in sequence; the ninth convolutional layer is a 3×3 convolutional layer, and the tenth convolutional layer is a 1×1 convolutional layer. The input of the ninth convolutional layer is connected to the output of the third GELU layer, and the output of the fourth GELU layer is connected to the second input of the cascaded layers.
[0054] Input feature map F aAfter the seventh convolutional layer and the third GELU layer, the number of channels is doubled using a 1×1 convolution, resulting in a dimension of H*W*2C. Channel separation is then used, selecting half the number of channels for a local information enhancement module (H*W*C). The ninth convolutional layer captures the feature relationships between adjacent pixels, and the tenth convolutional layer performs channel fusion to integrate the feature relationships between different channels. The fourth GELU layer enhances the model's non-linear fitting ability. Finally, half the number of channels is selected to retain the original information (H*W*C). The local information enhancement module and the original information retention branch are cascaded and fed into the eighth convolutional layer for further feature mixing, reducing the dimension to the original H*W*C, resulting in the final output feature map F. out .
[0055] This model not only effectively enhances the extraction and restoration of local details, improving the texture fidelity and edge sharpness of reconstructed images, but also significantly reduces computational complexity and memory overhead through structural optimization and parameter compression, making it suitable for deployment on resource-constrained edge devices. Furthermore, it possesses good versatility, adapting to various image types, including remote sensing and natural images, enabling more efficient cross-domain super-resolution reconstruction and meeting diverse practical application needs.
[0056] When using the above model to achieve lightweight super-resolution reconstruction applicable across different domains, the specific method involves the following steps:
[0057] Step 1: Obtain the dataset, which includes natural image data and remote sensing image data; the dataset is divided into training set, validation set and test set in an 8:1:1 ratio.
[0058] Step 2: Input the training set into the super-resolution reconstruction network model for training to obtain the trained super-resolution reconstruction network model.
[0059] Step 3: Input the validation set to validate the super-resolution reconstruction network model and evaluate its reconstruction performance in different image domains. Validate the effectiveness and versatility of the model in multi-domain image super-resolution reconstruction tasks through quantitative indicators (such as PSNR, SSIM, etc.) and visualization results, and evaluate the model's weight by calculating the number of model parameters and flops.
[0060] Step 4: Input the test set into the trained super-resolution reconstruction network model to achieve cross-domain super-resolution reconstruction.
Claims
1. A lightweight super-resolution reconstruction method applicable across multiple domains, characterized in that, Includes the following steps: Step 1: Obtain the dataset, which includes natural image data and remote sensing image data; the dataset is divided into a training set and a test set; Step 2: Construct a super-resolution reconstruction network model. This model includes a first convolutional layer, multiple feature aggregation blocks, a first adder, a second convolutional layer, and a pixel reconstruction layer connected in sequence. The input of the first convolutional layer is connected to the second input of the first adder, serving as the input of the super-resolution reconstruction network model. The feature aggregation block includes a dual-branch feature extraction and aggregation module, a second adder, and a parallel feature enhancement module connected in sequence. The input of the dual-branch feature extraction and aggregation module is connected to the second input of the second adder, serving as the input of the feature aggregation block. The output of the parallel feature enhancement module serves as the output of the feature aggregation block. The dual-branch feature extraction and aggregation module includes a third convolutional layer, a channel splitting layer, a local feature exploration branch, a non-local feature exploration branch, a third adder, and a fourth convolutional layer; The input of the third convolutional layer is the input of the dual-branch feature extraction and aggregation module, and the output is connected to the input of the channel splitting layer. The inputs of the local feature exploration branch and the non-local feature exploration branch are respectively connected to the two outputs of the channel splitting layer, and the outputs are respectively connected to the two inputs of the third adder. The output of the third adder is connected to the input of the fourth convolutional layer, and the output of the fourth convolutional layer is the output of the dual-branch feature aggregation module. The parallel feature enhancement module includes a branch that preserves original information and a local information enhancement module; The branch that preserves the original information is sequentially connected to the seventh convolutional layer, the third GELU layer, the cascaded layer, the eighth convolutional layer, and the sixth adder; The input of the seventh convolutional layer and the second input of the sixth adder serve as the input of the parallel feature enhancement module and are connected to the output of the first adder; the output of the sixth adder is the output of the parallel feature enhancement module. The input of the local information enhancement module is connected to the output of the third GELU layer, and the output is connected to the second input of the cascaded layer. The local information enhancement module includes a ninth convolutional layer, a tenth convolutional layer, and a fourth GELU layer connected in sequence; The input of the ninth convolutional layer is connected to the output of the third GELU layer, and the output of the fourth GELU layer is connected to the second input of the cascaded layer. Step 3: Input the training set into the super-resolution reconstruction network model for training to obtain the trained super-resolution reconstruction network model; Step 4: Input the test set into the trained super-resolution reconstruction network model to achieve cross-domain super-resolution reconstruction.
2. The lightweight super-resolution reconstruction method applicable across multiple domains according to claim 1, characterized in that: In step 2, the local feature exploration branch includes a normalization layer, a first GELU layer, a partial convolutional channel mixer, a fourth adder, and a fifth convolutional layer connected in sequence. The input of the normalization layer and the second input of the fourth adder serve as the input of the local feature exploration branch and are connected to the first output of the channel splitting layer; the output of the fifth convolutional layer serves as the output of the local feature exploration branch and is connected to the first input of the third adder. The nonlocal feature exploration branch includes an adaptive max pooling layer, a deep convolutional layer, an approximately depth-separable convolutional block, a fifth adder, a sixth convolutional layer, a second GELU layer, and a multiplier connected in sequence. The second input of the fifth adder is connected to the second input of the channel separation layer through the global variance calculation unit, the second input of the multiplier, and the input of the adaptive maximum pooling layer, respectively; the output of the multiplier is connected to the second input of the third adder.
3. The lightweight super-resolution reconstruction method applicable across multiple domains according to claim 2, characterized in that: The first and second convolutional layers are both 3×3 convolutional layers, and the third, fourth, fifth and sixth convolutional layers are all 1×1 convolutional layers.
4. A lightweight super-resolution reconstruction method applicable across domains according to any one of claims 1-3, characterized in that: The seventh, eighth, and tenth convolutional layers are all 1×1 convolutional layers, and the ninth convolutional layer is a 3×3 convolutional layer.