An image super-resolution construction method based on a multi-scale feature refinement network

Abstract

The application discloses a kind of based on multi-scale feature refinement network's image super-resolution construction method, comprising the following steps: step one: using DIV2K dataset constructs high-resolution and low-resolution image pair, and the generalization ability of network model to diversified image features is improved by data enhancement technology;Step two: construct multi-scale feature refinement network: the network includes detail extraction engine, global multi-scale context module and feature refinement module;Step three: the parameter of multi-scale feature refinement network model is optimized by minimizing L1 loss function to train;Step four: using the network model trained to low-resolution image carries out super-resolution reconstruction, generates high-resolution image output;The model is evaluated on test dataset.The application introduces detail extraction engine (DEE), global multi-scale context module (GMCM) and feature refinement module (FRM), enhances cross-region feature interaction and the limitation in the aspect of detail in single image super-resolution recovery.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image super-resolution construction method based on a multi-scale feature thinning network. Background Technology

[0002] Single-image super-resolution (SISR) is a method for recovering high-resolution (HR) images from low-resolution (LR) images, and its importance in the field of computer vision is self-evident. Traditional super-resolution methods, such as interpolation techniques and reconstruction algorithms, while improving image resolution to some extent, have significant limitations in restoring image details. In recent years, with the rapid development of deep learning technology, super-resolution methods based on convolutional neural networks (CNNs) have gradually become mainstream. CNNs, with their powerful feature extraction capabilities, have made significant progress in the field of SISR.

[0003] However, existing CNN models, while pursuing high-resolution images, also face the problem of excessive computational complexity and storage requirements. This limits the application of CNN models in resource-constrained environments, such as mobile devices or embedded systems. To address this issue, researchers began exploring lightweight CNN models, reducing computational complexity by decreasing the number of network layers and using small convolutional kernels. While these methods have achieved some success, it remains difficult to significantly improve computational efficiency while maintaining image reconstruction quality.

[0004] On the other hand, the successful application of Transformers in Natural Language Processing (NLP) has inspired researchers to explore their applications in computer vision. The Transformer's self-attention mechanism can effectively model long-range dependencies, which is crucial for image detail recovery. However, directly applying Transformers to image super-resolution tasks is unreasonable because CNNs have irreplaceable advantages in extracting local features and preserving local invariance.

[0005] Therefore, building network models that combine CNNs and Transformers has become a research hotspot. While existing methods combining CNNs and Transformers attempt to compensate for the shortcomings of single structures, they often fail to effectively improve the interactivity of cross-regional features in practice, merely resulting in a simple superposition of local and global features. Furthermore, during feature extraction, high-frequency information such as edges and textures is easily diluted or lost, leading to a lack of clarity and naturalness in the generated images, thus affecting the final visual quality. Summary of the Invention

[0006] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0007] In view of the problems existing in the above-mentioned image super-resolution construction methods based on multi-scale feature refinement networks, this invention is proposed.

[0008] Therefore, the purpose of this invention is to provide an image super-resolution construction method based on a multi-scale feature refinement network, which enhances cross-region feature interaction and recovers details in single image super-resolution by introducing a detail extraction engine (DEE), a global multi-scale context module (GMCM), and a feature refinement module (FRM).

[0009] To address the aforementioned technical problems, this invention provides the following technical solution: an image super-resolution construction method based on a multi-scale feature thinning network, comprising the following steps:

[0010] Step 1: Construct high-resolution and low-resolution image pairs using the DIV2K dataset. Combine data augmentation methods (such as random cropping, rotation, flipping, etc.) and design diverse input features to cover different types of degradation modes and resolution requirements to initialize the network model and improve its generalization ability during training.

[0011] Step 2: Construct a multi-scale feature refinement network: The network includes a detail extraction engine, a global multi-scale context module, and a feature refinement module; wherein, the detail extraction engine focuses on extracting edge features and detailed textures of the input image; the global multi-scale context module captures local and global information through a multi-scale window grouping strategy; and the feature refinement module further optimizes and fuses features through upsampling, downsampling, and residual learning mechanisms to improve the image reconstruction quality;

[0012] Step 3: During training, we optimize the parameters of the multi-scale feature refinement network model by minimizing the L1 loss function and dynamically adjust the learning rate using the Adam optimization algorithm to reduce the difference between the reconstructed image and the real high-resolution image, thereby enhancing the model's detail recovery ability. The specific optimization objective is:

[0013]

[0014] Where F θ (I LR ) = I SR This represents the reconstructed image generated by the model parameters θ, where N is the number of images in the training set.

[0015] Step 4: The trained multi-scale feature refinement network model is used to perform super-resolution processing on the low-resolution image to produce a high-resolution image as output. Subsequently, we evaluate the model on test datasets (Set5, Set11, B100, Urban100, Manga109) and verify its superior performance by calculating Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) metrics. This ensures that our model can provide stable and high-quality image reconstruction results in different scenarios.

[0016] As a preferred embodiment of the image super-resolution construction method based on multi-scale feature refinement network described in this invention, in step two, the detail extraction engine module performs preliminary feature extraction on the input image using a 3×3 convolutional layer. This module designs a selective feature enhancer, which focuses on or emphasizes features at different locations in the input feature map by designing different convolutional kernel weights. The mathematical expression is as follows:

[0017]

[0018] Among them, v i Let W represent the output feature of the i-th channel, x represent the input feature map, and W represent the output feature of the i-th channel. i This represents the kernel weight of the i-th channel, * indicates the convolution operation, x i This represents the input feature of the i-th channel;

[0019] After SFE processing, the features are further refined using the ReLU activation function and a second SFE layer:

[0020]

[0021] Finally, the detail extraction engine module uses residual connections to combine the original input x with the enhanced features. Combined, ensure the retention of key information:

[0022]

[0023] As a preferred embodiment of the image super-resolution construction method based on multi-scale feature refinement network described in this invention, wherein: in step two, the global multi-scale context module inputs the feature map X∈R b×c×h×w First, it is expanded using a 1×1 convolutional layer, then divided into several groups, each processed using a different window size; the window size is determined by the set [s1, s2, ..., s...]. n The feature map is defined as follows: where each s corresponds to a different window size, such as [4, 8, 16]. For each window size s, the input feature map is shifted to capture different spatial contexts. The shifted feature map is represented as follows:

[0024]

[0025] Where: b is the batch size, c is the number of channels, h and w are the height and width of the feature map respectively, the scrolling operation moves the feature map cyclically in space, and X ~ represents the moved feature map;

[0026] Based on the moved feature maps, an attention mechanism is applied to capture spatial dependencies within each window. The attention mechanism first projects the features onto the query Q, key K, and value V representations, as shown in the following equation:

[0027]

[0028] Among them, W Q W K and W V It is a learnable weight matrix; the self-attention weights are calculated as follows:

[0029]

[0030] Where A represents the weight matrix Q, K and V represent the query, key, and value, respectively, and dk is the dimension of the key vector; the output features are calculated as follows:

[0031] Output = A·V(7)

[0032] Where · denotes matrix multiplication; the output features of all windows are concatenated through a 1×1 convolutional layer:

[0033] v = Conv 1×1 (Concat(Output))(8).

[0034] As a preferred embodiment of the image super-resolution construction method based on multi-scale feature thinning network described in this invention, the feature thinning module efficiently captures and reconstructs high-frequency details in the image through a series of downsampling and upsampling operations. The module first performs a 1×1 convolution projection on the input features to obtain feature v′, then performs two 2×2 convolutions on this feature for downsampling, then performs upsampling using bilinear interpolation, and finally thins it through 5×5 convolution and GELU activation.

[0035] As a preferred embodiment of the image super-resolution construction method based on multi-scale feature thinning networks described in this invention, the mathematical expressions for downsampling and upsampling are as follows:

[0036]

[0037] Then, FRM fuses the upsampled features with the previous downsampled output to ensure detail preservation and improved denoising performance. The mathematical expressions for feature fusion for upsampling and downsampling are as follows:

[0038]

[0039] The final output features are combined with the original input through residual connections:

[0040] v final =Proj out (Conv 3×3 (torch.cat(v,v out ,dim=1)))+v in (11)

[0041] `torch.cat` concatenates the input feature map `v` and the transformed feature map `v_out` along the channel dimension; Proj out It is a projection layer used to adjust the output of the convolutional layer to the same dimension as the original input; by adding the input feature map to the feature map after a series of transformations, information flow and feature extraction are achieved.

[0042] The beneficial effects of this invention: This invention provides a Multi-Scale Feature Refinement Network (MSFRN) for image super-resolution tasks. This network achieves a balance between reconstruction performance and computational efficiency by combining CNN and Transformer attention mechanisms. The design of MSFRN fully considers the characteristics of image super-resolution tasks, enhancing cross-region feature interaction and overcoming the limitations of restoring details in single-image super-resolution by introducing a Depth Extraction Engine (DEE), a Global Multi-Scale Context Module (GMCM), and a Feature Refinement Module (FRM). Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0044] Figure 1 : Overall design framework diagram of Multi-Scale Feature Refinement Network (MSFRN);

[0045] Figure 2 : Architecture diagram of the Detail Extraction Engine (DEE);

[0046] Figure 3 : Architecture diagram of the Global Multi-Scale Context Module (GMCM);

[0047] Figure 4 : Schematic diagram of the Feature Refinement Module (FRM);

[0048] Figure 5 Visual comparison of this invention with other super-resolution reconstruction methods on image Img_012 in the Urban100 dataset;

[0049] Figure 6 The present invention provides a visual comparison of image 78004 in dataset B100 with other super-resolution reconstruction methods. Detailed Implementation

[0050] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0051] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0052] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0053] Secondly, the present invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of the present invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not according to the usual scale. Furthermore, the schematic diagrams are merely examples and should not limit the scope of protection of the present invention. In addition, actual fabrication should include three-dimensional spatial dimensions of length, width, and depth.

[0054] Reference Figures 1-6 This paper presents an image super-resolution construction method based on a multi-scale feature thinning network, which includes the following steps:

[0055] Step 1: Construct high-resolution and low-resolution image pairs using the DIV2K dataset. Combine data augmentation methods (such as random cropping, rotation, flipping, etc.) and design diverse input features to cover different types of degradation modes and resolution requirements to initialize the network model and improve its generalization ability during training.

[0056] Step 2: Construct a multi-scale feature refinement network: The network includes a detail extraction engine, a global multi-scale context module, and a feature refinement module; wherein, the detail extraction engine focuses on extracting edge features and detailed textures of the input image; the global multi-scale context module captures local and global information through a multi-scale window grouping strategy; and the feature refinement module further optimizes and fuses features through upsampling, downsampling, and residual learning mechanisms to improve the image reconstruction quality;

[0057] Step 3: During training, we optimize the parameters of the multi-scale feature refinement network model by minimizing the L1 loss function and dynamically adjust the learning rate using the Adam optimization algorithm to reduce the difference between the reconstructed image and the real high-resolution image, thereby enhancing the model's ability to recover details.

[0058] Step 4: The trained multi-scale feature refinement network model is used to perform super-resolution processing on the low-resolution image to produce a high-resolution image as output. Subsequently, we evaluate the model on test datasets (Set5, Set11, B100, Urban100, Manga109) and verify its superior performance by calculating Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) metrics. This ensures that our model can provide stable and high-quality image reconstruction results in different scenarios.

[0059] Among them, as attached Figure 2 In step two, the detail extraction engine module performs preliminary feature extraction on the input image using a 3×3 convolutional layer. This module designs a selective feature enhancer, which focuses on or emphasizes features at different locations in the input feature map by designing different convolutional kernel weights. The mathematical expression is as follows:

[0060]

[0061] Among them, v i Let W represent the output feature of the i-th channel, x represent the input feature map, and W represent the output feature of the i-th channel. i This represents the kernel weight of the i-th channel, * indicates the convolution operation, x i This represents the input feature of the i-th channel;

[0062] After SFE processing, the features are further refined using the ReLU activation function and a second SFE layer:

[0063]

[0064] Finally, the detail extraction engine module uses residual connections to combine the original input x with the enhanced features. Combined, ensure the retention of key information:

[0065]

[0066] The Detail Extraction Engine (DEE) further includes a Selective Feature Enhancer (SFE), designed to selectively enhance features in specific regions or orientations of the input feature map to improve the representational power of local features. Specifically, by using specially designed convolutional kernel weights for different orientations or regions, SFE can weight different parts of the input feature map, thereby highlighting key information and enriching local features. Furthermore, by combining the results of all weighted channel groups, SFE not only improves the overall features of the feature map but also provides stronger support for cross-regional feature interactions, thus enhancing the performance of image super-resolution tasks.

[0067] Further details are attached. Figure 3 In step two, the global multi-scale context module inputs the feature map X∈R. b×c×h×w First, it is expanded using a 1×1 convolutional layer, then divided into several groups, each processed using a different window size; the window size is determined by the set [s1, s2, ..., s...]. n The feature map is defined as follows: where each s corresponds to a different window size, such as [4, 8, 16]. For each window size s, the input feature map is shifted to capture different spatial contexts. The shifted feature map is represented as follows:

[0068]

[0069] Where: b is the batch size, c is the number of channels, h and w are the height and width of the feature map, respectively. The scrolling operation moves the feature map cyclically in space. Represents the feature map after the movement;

[0070] Based on the moved feature maps, an attention mechanism is applied to capture spatial dependencies within each window. The attention mechanism first projects the features onto the query Q, key K, and value V representations, as shown in the following equation:

[0071]

[0072] Among them, W Q W K and W V It is a learnable weight matrix; the self-attention weights are calculated as follows:

[0073]

[0074] Where A represents the weight matrix Q, K and V represent the query, key, and value, respectively, and dk is the dimension of the key vector; the output features are calculated as follows:

[0075] Output = A·V(7)

[0076] Where · denotes matrix multiplication; the output features of all windows are concatenated through a 1×1 convolutional layer:

[0077] v = Conv 1×1 (Concat(Output))(8).

[0078] The Global Multi-Scale Context Module (GMCM) divides the input features into several groups of windows, each with a different size (e.g., 4×4, 8×8, and 16×16) to capture contextual information at different scales. The module utilizes a feature fusion method based on a self-attention mechanism to calculate attention weights within each window. Through window scrolling, each window considers information from neighboring windows when calculating attention weights, enhancing the interaction between features. Furthermore, a 1×1 convolution operation projects the multi-scale features back into the original space, ensuring feature diversity and consistency, ultimately achieving efficient modeling and fusion of local and global features. The window scrolling operation, by cyclically moving the feature map along the height and width dimensions by half the size of the window pixels, breaks the limitations of fixed window partitioning, allowing information exchange between features in different windows. This provides more comprehensive contextual information, enhances the model's ability to capture the global structure of the image, and improves the quality of image super-resolution.

[0079] Among them, as attached Figure 4 The feature refinement module efficiently captures and reconstructs high-frequency details in an image through a series of downsampling and upsampling operations. This module first projects the input features using a 1×1 convolution to obtain feature v′, then performs downsampling on this feature using two 2×2 convolutions, followed by upsampling using bilinear interpolation, and finally refines it using a 5×5 convolution and GELU activation. The mathematical expressions for downsampling and upsampling are as follows:

[0080]

[0081] Then, FRM fuses the upsampled features with the previous downsampled output to ensure detail preservation and improved denoising performance. The mathematical expressions for feature fusion for upsampling and downsampling are as follows:

[0082]

[0083] The final output features are combined with the original input through residual connections:

[0084] v final =Proj out (Conv 3×3 (torch.cat(v,v out ,dim=1)))+v in(11)

[0085] `torch.cat` concatenates the input feature map `v` and the transformed feature map `v_out` along the channel dimension; Proj out It is a projection layer used to adjust the output of the convolutional layer to the same dimension as the original input; by adding the input feature map to the feature map after a series of transformations, information flow and feature extraction are achieved.

[0086] The Feature Refinement Module (FRM) refines the input features through multi-layer convolutional operations combined with upsampling and downsampling mechanisms. The module extracts multi-scale high-frequency features using bilinear interpolation and high-dimensional feature convolution operations, and learns the difference between the original input and the refined features through residual connections, further enhancing the ability to recover detailed textures. The design of the Feature Refinement Module improves image reconstruction performance while ensuring the model's lightweight and efficient nature.

[0087] Experimental setup and results:

[0088] MSFRN is trained using DIV2K. To verify its performance, we conducted experiments on several public datasets, including Set5, Set14, B100, Urban100, and Manga109. In the experiments, we used the following configuration:

[0089] (1) Training parameters: The batch size was 64, and each batch contained 256×256 pixel patches. The optimizer was Adam, with an initial learning rate of 0.0002, and the learning rate was adjusted using a cosine annealing strategy. The training period was 500 epochs.

[0090] (2) Hardware and software environment: The experiment was conducted on an NVIDIA GTX1080 GPU using the PyTorch framework.

[0091] Table 1 shows a performance comparison of MSFRN with other lightweight SISR models on different datasets. It can be seen that MSFRN achieves the best performance on most datasets, demonstrating strong reconstruction quality and relatively efficient computational requirements. MSFRN achieves a practical balance between model complexity and image quality using a relatively small number of parameters and FLOPs.

[0092] Table 1: Comparison of average PSNR and SSIM between the present invention and existing lightweight super-resolution models (optimal is indicated in bold, suboptimal is indicated by underline):

[0093]

[0094]

[0095] This invention combines Convolutional Neural Networks (CNNs) and Transformer attention mechanisms to address the limitations of existing methods in enhancing cross-region feature interactions and restoring details in single-image super-resolution. MSFRN employs a modular design to synergistically enhance feature representation and visual quality. Specifically, the Detail Extraction Engine (DEE) strengthens local features through Selective Feature Enhancer (SFE), indirectly promoting cross-region feature interactions. The Multi-Scale Context Module (GMCM) uses a multi-scale window attention mechanism combined with cyclic scrolling to improve cross-region feature relevance, achieving effective fusion of global and local information. Furthermore, the Feature Refinement Module (FRM) progressively restores details and textures through multi-level upsampling and downsampling, effectively addressing the problem of insufficient detail and texture restoration. Experiments demonstrate that MSFRN achieves competitive performance and enhanced visual quality compared to state-of-the-art methods while maintaining computational efficiency. Future work will explore lightweight attention mechanisms to further reduce resource requirements and expand the model's application in a wider range of low-level vision tasks.

[0096] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for constructing image super-resolution based on a multi-scale feature thinning network, characterized in that, Includes the following steps: Step 1: Construct high-resolution and low-resolution image pairs using the DIV2K dataset. Combine data augmentation methods and design diverse input features to cover different types of degradation modes and resolution requirements, in order to initialize the network model and improve its generalization ability during training. Step 2: Construct a multi-scale feature refinement network: The network includes a detail extraction engine, a global multi-scale context module, and a feature refinement module; wherein, the detail extraction engine focuses on extracting edge features and detailed textures of the input image; the global multi-scale context module captures local and global information through a multi-scale window grouping strategy; and the feature refinement module further optimizes and fuses features through upsampling, downsampling, and residual learning mechanisms to improve the image reconstruction quality; In step two, the detail extraction engine module performs preliminary feature extraction on the input image using a 3×3 convolutional layer. This detail extraction engine module designs a selective feature enhancer, which focuses on or emphasizes features at different locations in the input feature map by designing different convolutional kernel weights. The mathematical expression is as follows: (1) in, Indicates the first Output characteristics of each channel Represents the input feature map. Indicates the first The convolution kernel weights for each channel, This represents the convolution operation. Indicates the first Input characteristics of each channel; After SFE processing, the features are further refined using the ReLU activation function and a second SFE layer: (2) Finally, the detail extraction engine module uses residual connections to process the original input. x With enhanced features Combined, ensure the retention of key information: (3)； In step two, the global multi-scale context module inputs the feature map. First, it is expanded using a 1×1 convolutional layer, then divided into several groups, each processed using a different window size; the window size is determined by the set [...]. s 1 ,s 2 ,...,s n ] Define, where each s Corresponding to different window sizes; for each window size s The input feature map is shifted to capture different spatial contexts; the shifted feature map is represented as follows: (4) in: b It refers to the batch size. c It is the number of channels. h and w These are the height and width of the feature map, respectively. The scrolling operation moves the feature map cyclically in space. The feature map after the movement; Based on the moved feature map, an attention mechanism is applied to capture the spatial dependencies within each window; the attention mechanism first projects the features into the query Q, key K, and value V representation, as shown in the following equation: (5) in, W Q , W K and W V It is a learnable weight matrix; the self-attention weights are calculated as follows: (6) in, A Represents the weight matrix. This refers to the dimension of the key vector; the output features are calculated as follows: (7) in This represents matrix multiplication; the output features of all windows are concatenated through a 1×1 convolutional layer. (8)； Step 3: During training, we optimize the parameters of the multi-scale feature refinement network model by minimizing the L1 loss function and dynamically adjust the learning rate using the Adam optimization algorithm to reduce the difference between the reconstructed image and the real high-resolution image, thereby enhancing the model's detail recovery ability. The specific optimization objective is: in Indicates the model parameters The generated reconstructed image, N It is the number of images in the training set; Step 4: Use the trained multi-scale feature refinement network model to perform super-resolution processing on the low-resolution image to produce a high-resolution image as output. Then, we evaluate the model on the test datasets Set5, Set11, B100, Urban100, and Manga109, and verify its superior performance by calculating the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM).

2. The image super-resolution construction method based on multi-scale feature thinning network according to claim 1, characterized in that: The feature refinement module efficiently captures and reconstructs high-frequency details in the image through a series of downsampling and upsampling operations; the feature refinement module first performs a 1×1 convolution projection on the input features to obtain the features. Then, this feature is downsampled using two 2×2 convolutions, followed by upsampling using bilinear interpolation, and finally upsampled using a 5×5 convolution. GELU Activate and refine it.

3. The image super-resolution construction method based on multi-scale feature thinning network according to claim 2, characterized in that: The mathematical expressions for downsampling and upsampling are as follows: (9) Then, the FRM fuses the upsampled features with the previous downsampled output to ensure detail preservation and improved denoising performance. The mathematical expressions for feature fusion for upsampling and downsampling are as follows: (10) The final output features are combined with the original input through residual connections: (11) This refers to input feature maps v and the transformed feature map The components are stitched together along the channel dimension. It is a projection layer used to adjust the output of the convolutional layer to the same dimension as the original input; by adding the input feature map to the feature map after a series of transformations, information flow and feature extraction are achieved.

Images