Lightweight image super-resolution method based on multi-scale local and global feature fusion network
By using the Multi-Scale Local and Global Feature Fusion Network (MLGFFN), the problems of feature fusion mismatch and high computational resource consumption in image super-resolution models are solved, achieving a balance between high image reconstruction performance and low resource consumption.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- WUYI UNIV
- Filing Date
- 2026-01-06
- Publication Date
- 2026-04-17
AI Technical Summary
Existing image super-resolution models suffer from mismatch issues when fusing global and local features, and consume excessive computational resources, failing to effectively capture multi-scale information and detailed features of images.
A multi-scale local and global feature fusion network (MLGFFN) is adopted. By combining a composite feature distillation block and a feature enhancement feedforward network, along with a multi-scale heterogeneous feature fusion block and a local feature extraction block, and using cascaded residual connections and channel attention modules, efficient fusion and interaction of feature information are achieved.
It achieves efficient reconstruction of high-quality images with low resource consumption, significantly improves image reconstruction performance, maintains the lightweight features of the model, and effectively captures multi-scale and local detail information.
Smart Images
Figure CN121883259A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer vision technology, and in particular to a lightweight image super-resolution method based on a multi-scale local and global feature fusion network. Background Technology
[0002] Image super-resolution (SR) refers to the task of reconstructing a high-resolution (HR) image from an input low-resolution (LR) image. Since the SRCNN model first introduced deep learning into super-resolution, SR image reconstruction research has made groundbreaking progress. With the advent of the Transformer model, which emphasizes global features, the image reconstruction performance of Transformer-based SR models has achieved a significant leap. However, this performance improvement often comes with extremely high computational resource consumption, and the Self-Attention mechanism in the Transformer model is somewhat lacking in its ability to explore fine local details.
[0003] The SMFANet model proposes a self-modulating feature aggregation module to collaboratively utilize the interaction between local and global feature branches. The local detail branch introduces a 3 × 3 depthwise separable convolution to compensate for the lack of capturing high-frequency local information, achieving more accurate image reconstruction. However, edge details and contours in local image details are directional, and a single convolutional kernel is insufficient to represent these features. MAN proposes an MLKA module that uses a multi-scale large kernel attention mechanism to obtain rich attention maps at different granularity levels. However, parallel large kernel convolution operations can only coarsely extract multi-scale features and cannot fully utilize contextual information at different scales. In visual tasks, the size of image information features varies greatly, and feature representations within a single receptive field cannot distinguish and express feature information from regions of different sizes.
[0004] Feature fusion can simultaneously utilize global and local information to understand the global contextual structure of feature maps, preserving fine edge details while maintaining the stability of the overall structure. Recent studies using dual-branch methods to explore local and global features have employed simple element-wise addition fusion. However, the receptive fields of global and local features are significantly different, and basic additive fusion operations cannot address this inherent mismatch. Furthermore, it ignores the dependencies between feature channels, lacking inter-channel information interaction and failing to effectively distinguish information from different blocks. Summary of the Invention
[0005] The purpose of this invention is to disclose a lightweight image super-resolution method based on a multi-scale local and global feature fusion network in order to simultaneously obtain the ability to capture global and local features, solve the feature fusion mismatch problem, and maintain the low resource consumption advantage of CNN models.
[0006] The above-mentioned objective of this application is achieved through the following technical solution: S1: Obtain a low-resolution image dataset; S2: Construct the MLGFFN model using a composite feature distillation block and a feature enhancement feedforward network; S3: Input the low-resolution image dataset into the MLGFFN model for training; S4: Obtain the low-resolution image to be super-resolution reconstructed, input the low-resolution image into the trained MLGFFN model, and obtain the high-resolution image reconstructed by super-resolution.
[0007] Furthermore, the MLGFFN model includes: Composite Feature Distillation Block (CFDB) and Feature Enhancement Feedforward Network (FEFN).
[0008] Furthermore, the Composite Feature Distillation Block (CFDB) includes: Multi-Scale Heterogeneous Feature Fusion Block (MAFFB), a distillation branch convolutional layer, and an output convolutional layer; Three multi-scale heterogeneous feature fusion blocks (MAFFB) main branches and four distillation branch convolutional layers are arranged sequentially as four layers through residual connections, and finally combined through channel splicing operations. The formula is as follows: ; in Indicates input, Indicates the output. This represents the main branch at level i. This represents the distillation branch of the i-th layer. This represents the multi-scale heterogeneous feature fusion block MAFFB at the i-th layer. represents the i-th convolutional layer, and represents the channel splicing operation.
[0009] Furthermore, the multi-scale heterogeneous feature fusion block MAFFB includes: a multi-scale large kernel convolution block MLKB, a local feature extraction block LFEB, and a fusion module; A multi-scale large kernel convolution block (MLKB) and a local feature extraction block (LFEB) are combined in parallel, followed by a fusion module.
[0010] Furthermore, the multi-scale large kernel convolutional block MLKB includes: an input convolutional layer, three efficient large kernel units ELKU, and an output convolutional layer; The efficient large kernel unit ELKU consists of two asymmetric large kernel depth separable convolutional layers, one convolutional layer, and a GELU Gaussian error linear unit activation function; The kernel sizes of the asymmetric large kernel depth separable convolutional layers of the three efficient large kernel units ELKU are 5, 11 and 19, respectively.
[0011] Furthermore, the Local Feature Extraction Block (LFEB) includes: two asymmetric depthwise separable convolutional layers, two convolutional layers, a GELU Gaussian error linear unit activation function, and an output convolutional layer.
[0012] Furthermore, the fusion module includes: a channel shuffling operation layer, a channel attention module (CA), and an output convolutional layer; The Channel Attention Module (CA) consists of an adaptive average pooling layer, an adaptive max pooling layer, and two Sigmoid activation functions.
[0013] Furthermore, the Feature Enhancement Feedforward Network (FEFN) includes: an input convolutional layer, a GELU Gaussian error linear unit activation function, four branches combined through channel concatenation operations, and an output convolutional layer; The four branches combined through channel splicing operations include: A convolutional layer; A convolutional layer and GELU Gaussian error linear unit activation function; A depthwise separable convolutional layer and a GELU Gaussian error linear unit activation function; A convolutional layer.
[0014] The beneficial effects of the technical solution provided in this application are: This invention relates to the MLGFFN model, whose core modules are the Multi-Scale Heterogeneous Feature Fusion Block (MAFFB) and the Feature Enhancement Feedforward Network (FEFN). The MAFFB module uses the local feature branch LFEB and the global feature branch MLKB for feature extraction, and then the fusion module endows the model with the ability to interact with channel information. MLKB effectively expands the receptive field by using multiple large-kernel asymmetric depthwise separable convolutions of different sizes with cascaded residual connections, while simultaneously representing features of information regions of different sizes separately. LFEB adopts an asymmetric convolution construction method heterogeneous with MLKB, independently extracting directional local detail textures horizontally and vertically. FEFN further refines the representative features extracted by the previous stage module. Based on the above design, compared with other lightweight super-resolution models, the proposed MLGFFN model achieves superior image reconstruction performance while maintaining relatively low model complexity. Extensive experiments show that the MLGFFN model proposed in this invention achieves a favorable balance between computational efficiency and reconstruction performance. Attached Figure Description
[0015] The present application will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a framework diagram of the Multi-Scale Local and Global Feature Fusion Network (MLGFFN), which includes the structure of the Composite Feature Distillation Block (CFDB), the Feature Enhancement Feedforward Network (FEFN), and the Multi-Scale Heterogeneous Feature Fusion Block (MAFFB). Figure 2 It consists of the Local Feature Extraction Block (LFEB), the Channel Attention (CA) module, and the High-Efficiency Large Kernel Unit (ELKU). Figure 3 The visualization results of each model on the image019 image of the Urban100 dataset with a magnification factor of 4; Figure 4 The visualization results of each model on the image084 image of the Urban100 dataset with a magnification factor of 4. Detailed Implementation
[0016] To provide a clearer understanding of the technical features, objectives, and effects of this application, the specific embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0017] The embodiments of this application provide a lightweight image super-resolution method based on a multi-scale local and global feature fusion network.
[0018] Please refer to Figure 1 , Figure 1 This is a framework diagram of the MLGFFN model, a lightweight image super-resolution method based on a multi-scale local and global feature fusion network, as described in this application embodiment, including: S1: Obtain a low-resolution image dataset; S2: Construct the MLGFFN model using a composite feature distillation block and a feature enhancement feedforward network; S3: Input the low-resolution image dataset into the MLGFFN model for training; S4: Obtain the low-resolution image to be super-resolution reconstructed, input the low-resolution image into the trained MLGFFN model, and obtain the high-resolution image reconstructed by super-resolution.
[0019] The MLGFFN model includes: Composite Feature Distillation Block (CFDB) and Feature Enhancement Feedforward Network (FEFN).
[0020] As one embodiment, the DIV2K dataset was used as both the training and validation sets for training the MLGFFN model. This dataset contains high-resolution images of various complex scenes, with 800 training images and 100 validation images. To effectively evaluate the capabilities of the MLGFFN model, five commonly used test datasets were selected: Set5, Set14, BSD100, Urban100, and Manga109. The Peak Signal-to-Noise Ratio (PSNR) and the Structural Similarity Index (SSIM) were used to evaluate the quality of the reconstructed images. Both metrics were calculated on the Y channel of the image converted to the YCbCr color space.
[0021] As one embodiment, in order to simultaneously aggregate local and global information, this invention designs a composite feature distillation block (CFDB), see [link to CFDB]. Figure 1 (b) The CFDB module embeds multiple multi-scale heterogeneous feature fusion blocks (MAFFB) into the main branch of feature distillation, and performs channel splicing and aggregation of the global and local fusion information extracted from the multi-scale heterogeneous feature fusion blocks (MAFFB) and the global information extracted through multiple 1×1 convolutional layers.
[0022] The CFDB module includes: Multi-Scale Heterogeneous Feature Fusion Block (MAFFB), Distillation Branch Convolutional Layer, and Output Convolutional Layer.
[0023] As one embodiment, the Multi-Scale Heterogeneous Feature Fusion Block (MAFFB) employs a dual-branch structure to extract global and local features. Figure 1 As shown in (c), the multi-scale large kernel convolutional block (MLKB) is a global feature branch used to capture a wide range of contextual dependencies, while the local feature extraction block (LFEB) is a local feature branch focused on extracting high-frequency details. Figure 2 (a). The outputs of these two branches are integrated through a feature fusion module to obtain a feature map that adaptively fuses global and local information.
[0024] The multi-scale heterogeneous feature fusion block MAFFB includes: a multi-scale large kernel convolution block MLKB, a local feature extraction block LFEB, and a fusion module.
[0025] As one embodiment, the multi-scale large kernel convolutional block (MLKB) utilizes multiple cascaded, high-efficiency large kernel units (ELKU) of varying kernel sizes, as shown in [link to example]. Figure 2 (c) Extracting rich global contextual features. Cascaded residual connections construct an information flow channel, allowing features at different scales to enhance and fuse with each other, providing the model with richer feature information representation. Large kernel convolutions can provide a larger receptive field, helping to understand the long-term dependencies and structural information of feature maps and maintaining the overall consistency of the reconstructed image. Meanwhile, the efficient large kernel unit ELKU, using cascaded asymmetric depthwise separable large kernel convolutions, can effectively reduce the number of model parameters. This invention employs three ELKU modules with sizes of 5, 11, and 19.
[0026] The High Efficiency Large Kernel Unit (ELKU) comprises: two asymmetric large kernel depth-separable convolutional layers, one convolutional layer, and a GELU Gaussian error linear unit activation function. The size of the ELKU refers to the kernel size of the two asymmetric large kernel depth-separable convolutional layers.
[0027] As one example, the Local Feature Extraction Block (LFEB) uses an asymmetric convolutional structure heterogeneous with the efficient large kernel unit ELKU. LFEB employs a parallel asymmetric convolutional design, independently focusing on texture information in both the horizontal and vertical directions, demonstrating good performance in capturing directional details. Finally, a 1 × 1 convolution with a GELU activation function is used to enhance the informative local feature details.
[0028] The Local Feature Extraction Block (LFEB) comprises: two asymmetric depthwise separable convolutional layers, two convolutional layers, a GELU Gaussian error linear unit activation function, and an output convolutional layer.
[0029] As one embodiment, the fusion module adaptively fuses global and local feature maps from the output of the previous layer, including a channel shuffling operation layer, a channel attention module (CA), and an output convolutional layer. The CA module, for example... Figure 2 As shown in (b), adaptive average pooling and adaptive max pooling operations are introduced without introducing additional parameters. Dual pooling gives higher weights to channels with important details (such as sharp points and precise edges) in the feature map, while suppressing channels containing noisy or redundant background information.
[0030] The channel attention module (CA) includes: an adaptive average pooling layer, an adaptive max pooling layer, and two Sigmoid activation functions.
[0031] As one example, the Feature Enhancement Feedforward Network (FEFN) further refines features in both spatial and channel dimensions, emphasizing important information and suppressing noise.
[0032] The Feature Enhancement Feedforward Network (FEFN) comprises: an input convolutional layer, a GELU Gaussian error linear unit activation function, four branches combined through channel concatenation operations, and an output convolutional layer.
[0033] To fully evaluate the model's performance, this application compares it with SMSR, ShuffleMixer, SAFMN, SMFANet, CFM, MAN-tiny, MSWSR, and ESDAN models. Table 1 shows a quantitative comparison of the various models at ×2, ×3, and ×4 scaling factors on the benchmark dataset. In addition to PSNR and SSIM metrics, this application also lists the number of parameters (#Params) and the number of floating-point operations (#FLOPs) used to evaluate the model complexity of the method. For fair comparison, all model complexity calculations were performed with LR images super-resolutiond to 1280 × 720 pixels.
[0034] In one specific embodiment of this application, SMSR: Sparse Mask Super-Resolution; ShuffleMixer: Channel Shuffle Mixer Network; SAFMN Spatial Adaptive Feature Modulation Network; SMFANet Lightweight Self-Modulation Feature Aggregation Network; CFM Convolutional Filter Hybrid Network; MAN-tiny: Miniature Multi-Scale Attention Network; MSWSR: Lightweight Multi-Scale Feature Selection Network; ESDAN: Sharpening Enhancement and Dual Attention Network.
[0035] Table 1. Performance of different models on different datasets
[0036] As shown in Table 1, the MLGFFN model proposed in this invention achieved the best performance metrics on most benchmark datasets at all magnification levels in the experiments, fully demonstrating its excellent image reconstruction capabilities. For example, on the Set14 dataset with a ×4 magnification factor, the MLGFFN model outperformed the SMSR model by 0.21 dB, while reducing the number of parameters and FLOPs by only 37.38% and 40%, respectively. This indicates that the MLGFFN model can acquire more feature information while effectively reducing the number of parameters and computational costs. Compared with the CFM model, the MLGFFN model achieved better performance at the cost of only an 18.99% increase in the number of parameters, outperforming CFM by an average of 27.6 dB on all datasets with a ×4 magnification factor, while reducing FLOPs by 26.96%. The MLGFFN model achieves an excellent balance between image reconstruction performance and resource consumption.
[0037] This application conducted extensive ablation experiments to analyze and evaluate the performance of each module in the proposed MLGFFN model. All ablation experiments were implemented based on a ×4 MLGFFN model and trained using the DIV2K dataset for fair comparison. The quantitative ablation results in Table 2 were measured on the Set5 and Manga109 datasets.
[0038] Table 2 Ablation experimental results of the MLGFFN model on the Set5 and Manga109 datasets with a magnification factor of ×4.
[0039] The core modules of the MLGFFN model proposed in this invention are the Multi-Scale Heterogeneous Feature Fusion Block (MAFFB) and the Feature Enhancement Feedforward Network (FEFN). To verify its effectiveness, this application removed both modules and compared their performance with the MLGFFN model. Table 2 shows that removing the MAFFB module reduced performance by 1.38 dB and 1.28 dB on the ×4 Set5 and Manga109 datasets, respectively, while removing FEFN reduced performance by 0.38 dB and 0.44 dB on the ×4 Set5 and Manga109 datasets, respectively. Experimental results demonstrate the necessity of the Multi-Scale Heterogeneous Feature Fusion Block (MAFFB) and the Feature Enhancement Feedforward Network (FEFN) in the MLGFFN network. Disabling the MLKB or LFEB modules leads to a significant drop in network performance, while replacing the fusion module with a simple feature addition fusion reduces the model's accuracy by 0.19 dB and 0.15 dB on the ×4 Set5 and Manga109 datasets, respectively. Table 2 shows that replacing MLK-K with 3 × 3 convolutions resulted in a 0.26 dB and 0.21 dB decrease in accuracy on the × 4 Set5 and Manga109 datasets, respectively. This is attributed to MLK-K's ability to capture feature information from regions of different sizes. Replacing the FEFN module with the FFN module reduced network performance by 0.11 dB and 0.14 dB on the × 4 Set5 and Manga109 datasets, respectively. This is mainly because the FFN module performs the same operation on feature information at each location, lacking the ability to effectively filter important information.
[0040] Visualization results analysis: This application selected the Urban100 dataset with rich details and textures, and compared the visualization results of the MLGFFN model with those of SAFMN, CFM, ShuffleMixer, MAN-tiny, SMFANet and ESDAN methods at a magnification factor of ×4. Figure 3 and Figure 4 The visualization compares the results of two images, img019 and img084, from the Urban100 dataset. For the image reconstruction of img019, other models showed poor ability to recover regions of different sizes and failed to adequately understand multi-scale information. The MLGFFN model effectively captured multi-scale information through cascaded large-kernel convolutions of different sizes, correctly recovering information from regions of different sizes. Its channel attention and channel shuffling operations effectively understood the interactions between regions, avoiding block adhesion. Regarding the detail and texture recovery of img084, other models' reconstructed images showed intersecting lines not present in the HR image, while the MLGFFN model's reconstructed image did not exhibit erroneous lines. This is attributed to the LFEB local feature extraction block's ability to understand directional detail textures.
[0041] Finally, it should be noted that the above description only depicts some embodiments of the present invention. For those skilled in the art, various changes, modifications, substitutions, and variations can be conceived of these embodiments without departing from the principles and spirit of the present invention. The scope of protection of the present invention is defined by the appended claims and their equivalents, and all the above-mentioned behaviors should be covered within the scope of protection of the present invention.
Claims
1. A lightweight image super-resolution method based on a multi-scale local and global feature fusion network, characterized in that, Includes the following steps: S1: Obtain a low-resolution image dataset; S2: Construct the MLGFFN model using a composite feature distillation block and a feature enhancement feedforward network; S3: Input the low-resolution image dataset into the MLGFFN model for training; S4: Obtain the low-resolution image to be super-resolution reconstructed, input the low-resolution image into the trained MLGFFN model, and obtain the high-resolution image reconstructed by super-resolution.
2. The lightweight image super-resolution method based on a multi-scale local and global feature fusion network according to claim 1, characterized in that, The MLGFFN model includes: Composite Feature Distillation Block (CFDB) and Feature Enhancement Feedforward Network (FEFN).
3. The lightweight image super-resolution method based on a multi-scale local and global feature fusion network according to claim 2, characterized in that, The Composite Feature Distillation Block (CFDB) includes: Multi-Scale Heterogeneous Feature Fusion Block (MAFFB), a distillation branch convolutional layer, and an output convolutional layer; Three multi-scale heterogeneous feature fusion blocks (MAFFB) main branches and four distillation branch convolutional layers are arranged sequentially as four layers through residual connections, and finally combined through channel splicing operations. The formula is as follows: in Indicates input, Indicates the output. This represents the main branch at level i. This represents the distillation branch of the i-th layer. This represents the multi-scale heterogeneous feature fusion block MAFFB at the i-th layer. represents the i-th convolutional layer, and represents the channel splicing operation.
4. The lightweight image super-resolution method based on a multi-scale local and global feature fusion network according to claim 3, characterized in that, The multi-scale heterogeneous feature fusion block MAFFB specifically includes: a multi-scale large kernel convolutional block MLKB, a local feature extraction block LFEB, and a fusion module. A multi-scale large kernel convolution block (MLKB) and a local feature extraction block (LFEB) are combined in parallel, followed by a fusion module.
5. The lightweight image super-resolution method based on a multi-scale local and global feature fusion network according to claim 4, characterized in that, The multi-scale large kernel convolutional block MLKB specifically includes: an input convolutional layer, three efficient large kernel units ELKU, and an output convolutional layer; The efficient large kernel unit ELKU consists of two asymmetric large kernel depth separable convolutional layers, one convolutional layer, and a GELU Gaussian error linear unit activation function; The kernel sizes of the asymmetric large kernel depth separable convolutional layers of the three efficient large kernel units ELKU are 5, 11 and 19, respectively.
6. The lightweight image super-resolution method based on a multi-scale local and global feature fusion network according to claim 4, characterized in that, The Local Feature Extraction Block (LFEB) specifically includes: two asymmetric depthwise separable convolutional layers, two convolutional layers, a GELU Gaussian error linear unit activation function, and an output convolutional layer.
7. The lightweight image super-resolution method based on a multi-scale local and global feature fusion network according to claim 4, characterized in that, The fusion module specifically includes: a channel shuffling operation layer, a channel attention module (CA), and an output convolutional layer; The Channel Attention Module (CA) consists of an adaptive average pooling layer, an adaptive max pooling layer, and two Sigmoid activation functions.
8. The lightweight image super-resolution method based on a multi-scale local and global feature fusion network according to claim 2, characterized in that, The Feature Enhancement Feedforward Network (FEFN) specifically includes: an input convolutional layer, a GELU Gaussian error linear unit activation function, four branches combined through channel concatenation operations, and an output convolutional layer. The four branches combined through channel splicing operations include: A 1 × 1 convolutional layer; A 3 × 3 convolutional layer and GELU Gaussian error linear unit activation function; A depthwise separable convolutional layer and a GELU Gaussian error linear unit activation function; A 1 × 1 convolutional layer.