High-efficiency single image super-resolution system based on perception loss guidance

Abstract

The invention belongs to the technical field of computer vision and image processing, and particularly relates to an efficient single image super-resolution system based on perception loss guidance. The system comprises a feature extraction module, a denoising module and an up-sampling module. The feature extraction module adopts a multi-branch structure and supports structure re-parameterization so as to improve the reasoning efficiency; the up-sampling module realizes spatial reconstruction based on sub-pixel rearrangement. A small denoising module formed by depth separable convolution is introduced before up-sampling, and artifacts and noise are effectively suppressed at a feature level. Meanwhile, a feature extractor formed by a randomly initialized lightweight convolutional network is introduced and used for calculating the perception loss, so that the structure reduction capability is enhanced, and the training cost is also remarkably reduced; according to the invention, each module is activated and optimized in stages, so that the training stability and the final performance are improved; and excellent performance is realized under a lightweight design, and the method is suitable for various resource-limited scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and image processing, and in particular relates to an efficient single image super-resolution system guided by perceptual loss. Background Art

[0002] Image super-resolution is a crucial research area in computer vision. Its goal is to restore image details and other data from known image information—in short, to increase image resolution. It has enormous practical demand and applications in digital imaging, video coding and communication, deep space satellite remote sensing, target recognition and analysis, and medical image analysis.

[0003] Existing image super-resolution reconstruction methods typically use large convolutional neural networks or generative adversarial networks, but they often have various problems such as over-parameterization, high computational cost and high latency, making them unsuitable for practical deployment on resource-constrained devices. To overcome these limitations, techniques such as network pruning, knowledge distillation, and structural reparameterization have been invented and applied to image super-resolution. Recent research has focused more on fine-grained network design to maximize the use of limited computing resources. However, model size limitations often restrict the model's ability to fully extract and utilize high-frequency texture information, which is crucial for recovering fine details in super-resolution tasks. In addition, challenges such as artifacts and optimization difficulties still exist, especially under strict resource constraints.

[0004] In view of this, the inventors hope to design an efficient single image super-resolution system guided by perceptual loss. Summary of the Invention

[0005] The present invention provides an efficient single-image super-resolution system guided by perceptual loss, which is suitable for resource-constrained scenarios. It aims to effectively improve the image reconstruction quality while maintaining a lightweight structure, and solve the problems of insufficient detail recovery ability and poor artifact suppression effect in the existing technology under computationally constrained conditions.

[0006] In order to achieve the above technical objectives and the above technical effects, the present invention is implemented through the following technical solutions:

[0007] The present invention provides an efficient single image super-resolution system based on perceptual loss guidance, comprising:

[0008] A feature extraction module, which is used to extract features from the input image through a multi-branch convolutional structure, where each convolution branch includes an identity mapping branch, a 1×1 convolution branch, and a 3×3 convolution branch, and is merged into a single convolution path through structural reparameterization during the inference phase;

[0009] A denoising module, which is used to perform artifact and noise suppression on low-resolution feature maps to improve image quality and reduce overall computational complexity;

[0010] An upsampling module uses a sub-pixel rearrangement operation to restore the denoised feature map to a high-resolution image output.

[0011] Furthermore, in the above-mentioned efficient single image super-resolution system guided by perceptual loss, the output of the feature extraction module is connected to an enhanced spatial attention module for enhancing the local spatial perception capability of the image.

[0012] Furthermore, in the above-mentioned efficient single-image super-resolution system guided by perceptual loss, the denoising module, as a module for image super-resolution tasks, includes two depthwise separable convolutional units connected in sequence, each convolutional unit includes a 3×3 depthwise convolutional layer, a 1×1 pointwise convolutional layer and a LeakyReLU activation function, and there is no residual connection between the two convolutional units.

[0013] Furthermore, in the above-mentioned efficient single-image super-resolution system guided by perceptual loss, the upsampling module can receive the feature map output by the denoising module before performing the sub-pixel rearrangement operation, and adopt a spatial reconstruction ratio of 2x or 4x, and the upsampling operation of each pixel is based on feature fusion between multiple channels.

[0014] Furthermore, in the above-mentioned efficient single image super-resolution system guided by perceptual loss, the perceptual loss construction method adopted by the system during training includes the following steps:

[0015] Build a lightweight feature extractor consisting of two layers of randomly initialized convolutional layers;

[0016] Inputting the reconstructed image and the real image into the feature extractor to obtain corresponding feature maps;

[0017] Compute the difference between two feature maps at multiple spatial locations to construct a perceptual loss;

[0018] The perceptual loss is combined with the pixel-wise L1 loss for neural network training.

[0019] Furthermore, in the above-mentioned efficient single image super-resolution system guided by perceptual loss, the lightweight feature extractor consists of two 3×3 convolutional layers and a tanh activation function, and does not use any pre-trained parameters.

[0020] Furthermore, in the above-mentioned efficient single image super-resolution system guided by perceptual loss, the segmented training method of the system includes the following stages:

[0021] Phase 1: Training the feature extraction module and upsampling module to optimize basic reconstruction capabilities;

[0022] The second stage: freeze the feature extraction module and train the denoising module and upsampling module to improve the artifact removal capability;

[0023] Phase 3: Unfreeze all modules and perform joint training to optimize overall performance.

[0024] Furthermore, in the above-mentioned efficient single image super-resolution system guided by perceptual loss, different initial learning rates are used in the first stage, the second stage and the third stage respectively to improve training stability.

[0025] Furthermore, in the above-mentioned efficient single-image super-resolution system guided by perceptual loss, during the training process of the second stage, the denoising module separately receives the intermediate feature map output by the feature extraction module, and the training goal of this stage is to reduce artifacts and local texture noise.

[0026] The beneficial effects of the present invention are:

[0027] The present invention discloses an efficient image super-resolution neural network system, which aims to improve the image reconstruction quality while keeping the model lightweight. The system includes a feature extraction module, a denoising module and an upsampling module. The feature extraction module adopts a multi-branch structure and supports structural reparameterization to improve inference efficiency; the upsampling module realizes spatial reconstruction based on sub-pixel rearrangement. In particular, a small denoising module composed of depthwise separable convolution is introduced before upsampling to effectively suppress artifacts and noise at the feature level. At the same time, a feature extractor composed of a randomly initialized lightweight convolutional network is introduced to calculate the perceptual loss, which not only enhances the structural restoration ability, but also significantly reduces the training cost. The present invention also proposes a three-stage training strategy to activate and optimize each module in stages to improve training stability and final performance. The system achieves excellent performance under a lightweight design and is suitable for various resource-constrained scenarios.

[0028] Of course, any product implementing the present invention does not necessarily need to achieve all of the above advantages at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 A simplified diagram of the high-efficiency single-image super-resolution system architecture of the present invention;

[0031] Figure 2 This is a simplified diagram of the feature extraction module architecture during the training phase of the present invention;

[0032] Figure 3 This is a simplified diagram of the feature extraction module architecture in the reasoning phase of the present invention;

[0033] Figure 4 This is a simplified diagram of the denoising module architecture of the present invention;

[0034] Figure 5 This is a simplified diagram of the perceptual loss lightweight feature extractor of the present invention;

[0035] Figure 6 It is a flow chart of the segmented training method of the present invention;

[0036] Figure 7 A comparison chart of the results obtained by the present invention and other methods under 4x upsampling;

[0037] Figure 8 Schematic diagram of the performance indicators of the present invention and various indicators (PSNR / SSIM) on the test set. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0039] Example 1

[0040] This example uses the public DIV2K dataset as the training data source. Each high-resolution image is downsampled using bicubic interpolation to generate a corresponding low-resolution image, forming the input-supervision pair. During training, the images are cropped into 256×256 pixel patches and augmented with random rotations (90°, 180°, and 270°) and horizontal flips to improve the model's generalization capabilities.

[0041] like Figure 1 As shown in the figure, this embodiment provides an efficient image super-resolution neural network system whose input is a 3-channel RGB low-resolution image. First, the input image is passed through a 3×3 convolutional layer for initial feature extraction, generating a basic feature map with 52 channels. This significantly expands the channel dimension while maintaining the original spatial resolution, laying the foundation for subsequent deep feature modeling.

[0042] like Figure 2 As shown in Figure 1, the initial feature map is sequentially input into 6 feature extraction modules. The number of input and output channels of each module is 52 and the structure is the same.

[0043] In this embodiment, the feature extraction module adopts the following three multi-branch structures:

[0044] φ branch1 =BatchNorm(φ in )

[0045] φ branch2 =BatchNorm(Conv1(φ in ))

[0046] φ branch3 =BatchNorm(Conv3(φ in ))

[0047] φ out =φ branch1 +φ branch2 +φ branch3

[0048] Among them, φ in Represents the input feature map, Conv1 and Conv3 represent convolution operations with kernel sizes of 1 and 3 respectively, and BatchNorm represents batch normalization operation.

[0049] The outputs of the three branches are fused and processed through the LeakyReLU activation function to obtain the module output, which effectively improves the feature expression ability and nonlinear modeling ability, making it easier to capture multi-scale details.

[0050] like Figure 3 As shown in the figure, the multi-branch structure is only retained in the training phase. In the inference phase, it is equivalently merged into a single 3×3 convolutional layer through the structure reparameterization method, which significantly reduces the computational complexity of the network and improves the inference efficiency.

[0051] Specifically, the combined convolution kernel weight W rp and the bias term b rp is calculated as follows:

[0052] W rp =W 3×3 ·Γ2+pad(W 1×1 ·Γ3)+I·Γ1

[0053] b rp =β2-(μ2-b 3×3 )·Γ2+β3-(μ3-b 1×1 )·Γ3+β1-μ1·Γ1

[0054] Among them, W3×3 and b 3×3 is the weight and bias of the 3×3 convolution, W 1×1 and b 1×1 are the weights and biases of the 1×1 convolution, I represents the unit map (i.e., the identity convolution kernel), and pad(·) represents the operation of expanding the 1×1 convolution kernel to a 3×3 size by zero padding. Γ is the scaling factor of the batch normalization layer corresponding to each branch, as shown below:

[0055]

[0056] Furthermore, γ i is the scaling factor of the batch normalization layer, σ i and μ i Represents its standard deviation and mean, β i is the offset term, and ε is a numerical stability constant, usually 1×10 -5 .

[0057] The combined convolution kernel output is added to the original features, processed through a 1×1 convolution layer, and then connected to the enhanced spatial attention module. This module enhances the structural response of local areas through operations such as feature compression, maximum pooling, and interpolation upsampling, thereby helping to restore texture details and improve edge clarity.

[0058] After all six feature extraction modules are processed, the output features are integrated through a 3×3 convolutional layer with the number of channels maintained at 52, and are residually connected with the initial input feature map through element-by-element addition to enhance feature consistency and promote information transfer and convergence stability of the deep network.

[0059] like Figure 4 As shown in the figure, to reduce artifacts and detail noise that may be introduced during the upsampling process, the system introduces a 48-channel denoising module before the upsampling module. Unlike traditional image domain post-processing methods, this denoising module directly suppresses artifacts on intermediate feature maps in the feature domain, effectively reducing redundant computation and improving perceptual consistency.

[0060] In this embodiment, the denoising module structure is as follows:

[0061] φ d1 =LeakyReLU(DepthwiseConv(φ in ))

[0062] φ p1 =LeakyReLU(PointwiseConv(φ d1 ))

[0063] φ d2=LeakyReLU(DepthwiseConv(φ p1 ))

[0064] φ p2 =LeakyReLU(PointwiseConv(φ d2 ))

[0065] φ out =Conv1(φ p2 )

[0066] DepthwiseConv and PointwiseConv represent depthwise convolution and pointwise convolution, respectively. No residual connection is set between the two units, which helps simplify the information flow path, suppress the propagation of high-frequency noise, and improve noise reduction performance and structural recovery stability.

[0067] To further verify the computational efficiency advantage of the denoising module proposed in this invention, we now compare its complexity with that of the traditional convolutional structure. The computational complexity of the denoising module based on the three-layer traditional convolutional structure is:

[0068]

[0069] The computational complexity of the denoising module based on depthwise separable convolution proposed in this invention is:

[0070]

[0071] Among them, H and W represent the height and width of the input image respectively, n chan is the number of input channels, n embed is the number of intermediate feature channels, and FLOPs represents the number of floating-point operations. As can be seen from the comparison, while maintaining effective feature modeling capabilities, the denoising module of the present invention significantly reduces computational overhead, making it particularly suitable for image super-resolution deployment scenarios that are sensitive to computing power.

[0072] The upsampling module uses a PixelShuffle structure to scale the denoised feature map to the target size and output a 3-channel high-resolution RGB image. The upsampling ratio can be flexibly set to 2x, 4x, or other requirements based on the actual application scenario. The calculation format is as follows:

[0073] I SR =Pixelshuffle(φ(I LR ))

[0074] Among them, I LR is the input low-resolution image, φ(·) is the feature extracted by the network, I SR The reconstructed high-resolution image is output.

[0075] like Figure 5 As shown, this embodiment also proposes a lightweight perceptual loss construction method to replace the traditional complex perceptual path built based on a pre-trained VGG network, thereby improving training efficiency and significantly reducing model size. This method designs a feature extractor consisting of two randomly initialized 3×3 convolutional layers, with a Tanh activation function inserted between the first and second layers. This feature extractor has a simple structure and does not require pre-training. It can be directly embedded in the training process to extract intermediate feature representations between the reconstructed image and the real image.

[0076] During training, the reconstructed image and the real image are fed into the lightweight feature extractor, and feature maps are extracted at the output of each layer. The present invention calculates the difference between these two sets of feature maps in the spatial dimension and uses the resulting difference as a perceptual loss term. Specifically, the output feature maps of the two convolutional extractor layers are weighted to 1, meaning that each layer contributes the same weight to the perceptual loss.

[0077] The final loss function is a weighted combination of perceptual loss and pixel-level L1 loss, where both the perceptual and L1 loss terms are weighted equal to 1, meaning they contribute equally to the total loss. This design maintains the simplicity of the loss function while balancing perceptual consistency and pixel accuracy during training, significantly improving the model's convergence speed and reconstruction quality.

[0078] Compared with traditional perceptual loss methods, the lightweight perceptual loss method constructed in this invention has the advantages of simple structure, low computational overhead, no need for pre-training, and strong generalization in the training stage. It is suitable for efficient image super-resolution model training and deployment in resource-constrained scenarios.

[0079] like Figure 6 As shown, to improve training efficiency, stability, and final performance, this paper proposes a segmented training strategy that can effectively alleviate the optimization difficulties brought by direct end-to-end training. It gradually improves the functional expression capabilities of each module of the model in stages, thereby significantly improving the final super-resolution reconstruction effect while maintaining training stability. The complete process is refined into the following five steps, each step corresponding to a different stage in the overall training process:

[0080] Step 1: Initialize a model that only contains the feature extraction module and the upsampling module, and use low-resolution and high-resolution image pairs for training to achieve the learning of basic super-resolution reconstruction capabilities. This stage corresponds to the first stage. In this stage, the initial learning rate used is 5×10 -4 , and halved the learning rate after every 200,000 iterations to promote training convergence and prevent oscillation;

[0081] Step 2: Migrate the optimized feature extraction module parameters from the first-stage trained model to the complete model, and load the new structure including the denoising module. To maintain consistency in feature representation, the feature extraction module parameters are frozen at this stage and do not participate in gradient updates, serving as a transition to the second stage.

[0082] Step 3: After fixing the feature extraction module, only the denoising module and upsampling module are trained. This stage corresponds to the second stage and is mainly used to optimize the ability of artifact suppression and image detail recovery. In this stage, the learning rate is increased to 2×10 -3 , and halved after every 100,000 iterations to accelerate the effective training of the model on the new module;

[0083] Step 4: Unfreeze the feature extraction module so that it can participate in the joint optimization of the entire network in the final training phase, as a transition preparation for the third phase.

[0084] Step 5: Jointly train the complete model to further optimize the synergy between modules and improve the overall image reconstruction quality. This stage is the third stage. In this stage, the initial learning rate used is 1×10 -4 , and halved adjustment is performed after every 100,000 iterations to achieve a stable fine-tuning process.

[0085] The method in this example was used to perform super-resolution reconstruction experiments on low-resolution images from the public datasets Set5, Set14, BSD100, Urban100, and Manga109. The experiments were conducted on a computing platform equipped with a GeForce RTX 4070 GPU, ensuring the stability and efficiency of model training and inference.

[0086] Figure 7 The following figure compares the reconstruction effects of this embodiment and existing methods under 4x upsampling conditions. The figure shows the reconstructed images using bicubic interpolation (Bicubic), IMDN, RFDN, RLFN, and the method of the present invention. This comparison shows that the method proposed in the present invention performs better in restoring details such as textures and edges, effectively reconstructing clearer, more natural, and realistic image details. This observation fully demonstrates the significant advantages of the present method in maintaining image structural consistency and visual quality.

[0087] Figure 8 These are quantitative performance metrics for the model in this example on five standard test sets: Set5, Set14, BSD100, Urban100, and Manga109. The peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are used to evaluate the super-resolution images generated by different methods on the Y channel in the YCbCr color space.

[0088] The results show that under the premise of small number of parameters, low computational complexity and fast inference speed, the neural network proposed in this invention still maintains excellent image reconstruction capabilities, verifying its good balance between high efficiency and high performance.

[0089] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to specific embodiments. Obviously, many modifications and variations are possible based on the contents of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. An efficient single image super-resolution system guided by perceptual loss, characterized by: Includes the following connected in sequence: A feature extraction module, which is used to extract features from the input image through a multi-branch convolutional structure, where each convolution branch includes an identity mapping branch, a 1×1 convolution branch, and a 3×3 convolution branch, and is merged into a single convolution path through structural reparameterization during the inference phase; A denoising module, which is used to perform artifact and noise suppression on low-resolution feature maps to improve image quality and reduce overall computational complexity; An upsampling module uses a sub-pixel rearrangement operation to restore the denoised feature map to a high-resolution image output.

2. The efficient single image super-resolution system based on perceptual loss guidance according to claim 1, characterized in that The output of the feature extraction module is connected to the enhanced spatial attention module to enhance the local spatial perception ability of the image.

3. The high-efficiency single image super-resolution system based on perceptual loss guidance according to claim 1, characterized in that The denoising module, as a module for image super-resolution tasks, includes two depthwise separable convolutional units connected in sequence. Each convolutional unit includes a 3×3 depthwise convolutional layer, a 1×1 pointwise convolutional layer and a LeakyReLU activation function. There is no residual connection between the two convolutional units.

4. The high-efficiency single image super-resolution system based on perceptual loss guidance according to claim 1, characterized in that Before performing the sub-pixel rearrangement operation, the upsampling module can receive the feature map output by the denoising module and adopt a 2x or 4x spatial reconstruction magnification, and the upsampling operation of each pixel is based on feature fusion between multiple channels.

5. The high-efficiency single image super-resolution system based on perceptual loss guidance according to claim 1, characterized in that: The perceptual loss construction method used by the system during training includes the following steps: Build a lightweight feature extractor consisting of two layers of randomly initialized convolutional layers; Inputting the reconstructed image and the real image into the feature extractor to obtain corresponding feature maps; Compute the difference between two feature maps at multiple spatial locations to construct a perceptual loss; The perceptual loss is combined with the pixel-wise L1 loss for neural network training.

6. The high-efficiency single image super-resolution system based on perceptual loss guidance according to claim 5, characterized in that: The lightweight feature extractor consists of two 3×3 convolutional layers and a tanh activation function, and does not use any pre-trained parameters.

7. The high-efficiency single image super-resolution system based on perceptual loss guidance according to claim 1, characterized in that: The system's segmented training method includes the following stages: Phase 1: Training the feature extraction module and upsampling module to optimize basic reconstruction capabilities; The second stage: freeze the feature extraction module and train the denoising module and upsampling module to improve the artifact removal capability; Phase 3: Unfreeze all modules and perform joint training to optimize overall performance.

8. The high-efficiency single image super-resolution system based on perceptual loss guidance according to claim 7, characterized in that: The first, second, and third stages use different initial learning rates to improve training stability.

9. The high-efficiency single image super-resolution system based on perceptual loss guidance according to claim 8, characterized in that: During the training process of the second stage, the denoising module alone receives the intermediate feature map output by the feature extraction module, and the training goal of this stage is to reduce artifacts and local texture noise.

Images