Uncertainty perception single-image super-resolution network model based on self-convolution attention

Abstract

The invention discloses an uncertainty perception single-image super-resolution network model based on self-convolution attention. The model comprises a shallow feature extraction module SFE, a deep feature extraction module DFE and an image reconstruction module IR. The shallow feature extraction module SFE performs shallow feature extraction on an input low-resolution image through a convolutional layer to generate an initial feature F0: the deep feature extraction module DFE comprises a plurality of residual attention groups RAG, and each residual attention group RAG is composed of a plurality of cross-resolution mixed attention blocks CRHAB; the deep feature extraction module DFE is used for extracting a deep feature FDF from the initial feature F0, and the image reconstruction module IR is used for up-sampling the deep feature FDF and combining the deep feature FDF with the initial feature F0 to generate a final super-resolution output image ISRS.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and image processing, and in particular to a deep learning network structure for single image super-resolution (SISR), specifically an uncertainty-aware single image super-resolution network model based on a self-convolutional attention mechanism, which is used to improve the reconstruction performance of a single low-resolution image to a high-resolution image. Background Art

[0002] Single Image Super Resolution (SISR) is an image reconstruction technique that aims to reconstruct a high-resolution (HR) image from a low-resolution (LR) image. It is widely used in medical imaging, satellite image analysis, video surveillance and other fields. Traditional methods such as bicubic interpolation and iterative back-projection are simple and easy to use, but the reconstruction effect is limited. In recent years, Convolutional Neural Networks (CNNs) have been introduced into the field of SISR, and the reconstruction performance has been greatly improved through deep learning methods, such as SRCNN. [1] , EDSR [2] and ESRGAN [3] Models such as these have achieved improved image reconstruction quality through the introduction of end-to-end mapping, residual structure, and generative adversarial networks.

[0003] Although these methods have made significant progress in super-resolution tasks, they still have limitations in capturing long-range dependencies and multi-scale information. Recent studies have shown that Transformer networks perform well in image processing, capturing global context through multiple self-attention mechanisms and effectively improving feature extraction capabilities. For example, SwinIR [4] and Restormer [5] The context-awareness is enhanced through hierarchical design. However, these methods usually only focus on single-scale feature relationships and ignore the feature correlation across resolutions, which limits the performance of SISR. In addition, many SISR methods treat all pixels equally when processing images, ignoring the high uncertainty in different regions of the image, which leads to poor generalization performance in high-uncertainty areas. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a Self-Convolutional Attention-Based Uncertainty-Aware Network for Single Image Super-Resolution (SCAUAN). The network model introduces a heteroscedastic uncertainty-driven loss function, which can perform adaptive optimization for high uncertainty areas (such as textures and edges) to enhance the reconstruction quality. At the same time, the proposed cross-scale self-convolutional attention mechanism (CSSCA) can perform feature self-convolution at different resolutions, capture cross-resolution self-similarity, enhance the network's contextual perception and local feature extraction capabilities, and thus effectively improve the quality of details in super-resolution reconstruction.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] An uncertainty-aware single image super-resolution network model based on self-convolutional attention, comprising: a shallow feature extraction module SFE, a deep feature extraction module DFE and an image reconstruction module IR;

[0007] The shallow feature extraction module SFE extracts shallow features from the input low-resolution image through a convolutional layer to generate the initial feature F0:

[0008] F0=H SF (I LR ),

[0009] in, represents the input low-resolution image, H and W are the height and width of the low-resolution image, and C is the number of channels of the low-resolution image; H SF (·) represents the mapping function of the shallow feature extraction module, which is implemented through a convolutional layer, and F0 is the generated initial feature;

[0010] The deep feature extraction module DFE includes several residual attention groups RAG, each of which is composed of several cross-resolution hybrid attention blocks CRHAB; the deep feature extraction module DFE is used to extract the deep feature F from the initial feature F0 DF :

[0011] F DF =H DF (F0),

[0012] in, is the deep feature, H DF (·) represents the mapping function of the deep feature extraction module DFE, which contains M residual attention groups RAG and a 3×3 convolutional layer;

[0013] The image reconstruction module IR consists of a convolutional layer, a pixel reorganization layer and another convolutional layer. DF The pixel is added to the initial feature F0, and then upsampled by the image reconstruction module IR to generate the final super-resolution output image I SRS :

[0014] I SRS =H REC (F0+F DF ),

[0015] in, is the output super-resolution image, H ′ and W ′ is the high-resolution image size after upsampling, H REC (·) represents the mapping function of the IR module.

[0016] Furthermore, the calculation process of each residual attention group RAG in the deep feature extraction module DFE is as follows:

[0017]

[0018] in, is the output feature of the i-th residual attention group RAG, represents the mapping function of the i-th residual attention group. Each residual attention group consists of a cross-resolution hybrid attention block CRHAB and a convolutional layer. M represents the number of residual attention groups RAG, M = 6; for the input feature of the i-th residual attention group RAG Extracting intermediate features through N cross-resolution hybrid attention blocks CRHAB is the output feature of the j-th cross-resolution hybrid attention block CRHAB, j = 1, 2, ..., N, and the specific process is expressed as:

[0019]

[0020] in, is the mapping function of the jth cross-resolution hybrid attention block CRHAB in the i-th residual attention group RAG; N is the number of cross-resolution hybrid attention blocks CRHAB in each residual attention group RAG, N = 6; the output features of the last layer of cross-resolution hybrid attention blocks CRHAB are combined with the input features through a convolutional layer Perform residual connection and calculate the output of the i-th residual attention group RAG:

[0021]

[0022] in, is the convolutional layer in the i-th residual attention group RAG;

[0023] Finally, the output F of the last residual attention group RAG is M Input to the convolutional layer at the end of the deep feature extraction module DFE:

[0024] F DF =H CONV (F M ),

[0025] Among them, H CONV (·) represents the last convolutional layer of the deep feature extraction module DFE.

[0026] Furthermore, the cross-resolution hybrid attention block CRHAB includes shifted window multi-head self-attention SW-MSA and cross-scale self-convolutional attention CSSCA;

[0027] Output features of the j+1th cross-resolution hybrid attention block CRHAB in the i-th residual attention group RAG It includes the following steps:

[0028] (1) Output features of the jth cross-resolution hybrid attention block CRHAB in the i-th residual attention group RAG Normalized through the LayerNorm layer:

[0029]

[0030] Among them, H LN (·) indicates the operation of LayerNorm layer; is the output feature of the j-th cross-resolution hybrid attention block CRHAB, j = 1, 2, ..., N; F x is the normalization result;

[0031] (2) Extract self-similarity information through parallel shifted window multi-head self-attention SW-MSA and cross-scale self-convolutional attention CSSCA; shifted window multi-head self-attention SW-MSA uses pixels outside the window to optimize the reconstruction result; CSSCA extracts self-similarity information through cross-resolution self-convolution to enhance context modeling and local feature extraction:

[0032]

[0033] Among them, H SW-MSA (·) represents the shifted window multi-head self-attention, H CSSCA (·) represents cross-scale self-convolutional attention CSSCA; F y This is the result after enhanced context modeling and local feature extraction;

[0034] (3) After another LayerNorm layer and a multi-layer perceptron MLP, the output features are generated:

[0035]

[0036] Among them, H MLP (·) represents the mapping of multilayer perceptron;

[0037] The cross-scale self-convolutional attention mechanism module CSSCA is used to extract feature self-similarity at different resolutions to enhance context modeling and local feature extraction; the specific process is as follows:

[0038] Normalization results After upsampling, it is divided into several high-resolution pixel blocks p of size d×d i , arranged into a high-resolution pixel block sequence denoted as P = [p1,…,p K ], where d = 8 and K is the number of blocks;

[0039] High-resolution pixel blocks are used as convolution kernels and normalized processing results F x Perform convolution operation and extract the convolution feature p ci , get the self-convolution feature sequence: P CSSCA =[p c1 ,…,p cK ] Among them, p ci =F x *p i , * indicates convolution operation, with a step size of 1; P CSSCA Each term in the equation refers to the normalized result F x and high-resolution pixel blocks p i The self-convolution features obtained after the convolution operation;

[0040] Afterwards, the convolutional layer is fused to obtain the cross-scale self-convolution features:

[0041] F CSSCA =H CoNV (P CSSCA ),

[0042] Among them, H CONV (·) represents the mapping of the convolutional layer.

[0043] The present invention also provides a training method based on the super-resolution network model, comprising:

[0044] S1. Extract training images from the dataset;

[0045] Extract training images from a standard single image super-resolution dataset, including low-resolution LR and high-resolution HR image pairs; low-resolution LR images are used as input, and high-resolution HR images are used as reference targets;

[0046] S2. Perform data enhancement processing on the training images;

[0047] The extracted training images are subjected to data augmentation processing to increase the diversity of training data and the generalization ability of the super-resolution network model; data augmentation methods include random cropping, rotation, horizontal flipping, and color jittering;

[0048] S3. Input the enhanced image into the super-resolution network model for forward propagation and calculate the loss function;

[0049] The enhanced low-resolution image is input into the super-resolution network model, and a predicted super-resolution image is generated through the forward propagation process of the super-resolution network model; a heteroscedastic uncertainty-driven loss function between the predicted super-resolution image and the true high-resolution image is calculated;

[0050] S4. Update the super-resolution network model weights using the Adam optimizer;

[0051] Use the back-propagation algorithm to calculate the gradient of the loss function and use the Adam optimizer to update the weights of the super-resolution network model;

[0052] S5. Repeat S2-S4 until the training is completed;

[0053] Repeat data enhancement, forward propagation, loss calculation, and updating of super-resolution network model weights S, and continuously optimize super-resolution network model parameters until a predetermined number of training rounds or other termination conditions are reached;

[0054] S6. Evaluate the performance of super-resolution network models;

[0055] Evaluate the performance of the trained super-resolution network model on the test set.

[0056] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the function of the uncertainty-aware single image super-resolution network model based on self-convolutional attention when executing the program.

[0057] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the functions of the uncertainty-aware single image super-resolution network model based on self-convolutional attention.

[0058] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0059] 1. Improve feature extraction capability: By designing a shallow feature extraction module (SFE) and a deep feature extraction module (DFE), the present invention can fully exploit the multi-scale features in the image. The SFE module extracts initial features through a convolutional layer, ensuring the extraction of basic information of low-resolution images; the DFE module introduces several residual attention groups (RAGs) and cross-resolution hybrid attention blocks (CRHABs), which can effectively capture the self-similarity of images at different resolutions and enhance the expression capability of deep features.

[0060] 2. Enhanced reconstruction details: The image reconstruction module (IR) of the present invention uses a sub-pixel convolution layer to upsample deep features and fuse them with the initial features to generate a high-resolution output image. This design not only improves the resolution of the image, but also makes the reconstructed image details richer and more realistic through feature fusion, significantly improving the blur and distortion problems that are prone to occur in traditional upsampling methods.

[0061] 3. Efficient processing of complex image regions: By introducing the Cross-Resolution Hybrid Attention Block (CRHAB), the present invention can carefully process complex image regions (such as textures and edges). CRHAB can capture long-range dependencies between pixels at different scales, especially in high-uncertainty areas, and can effectively improve the accuracy and detail restoration capabilities of super-resolution reconstruction.

[0062] 4. Efficient and concise network model structure: The network model structure of the present invention is reasonably designed, using residual attention group (RAG) and sub-pixel convolution layer, which can achieve efficient image reconstruction while maintaining low computational complexity. At the same time, the introduction of cross-resolution feature fusion mechanism helps to improve the adaptability of the network to various scenarios, enabling it to quickly process images in practical applications.

[0063] 5. Wide application scenarios: This network model is suitable for various image super-resolution reconstruction tasks, such as medical imaging, satellite imaging, video surveillance, etc. It can provide more refined image reconstruction results and improve the image analysis and processing effects in applications.

[0064] 6. Image reconstruction performance that exceeds existing methods: Through experimental verification, the network model of the present invention outperforms existing mainstream super-resolution algorithms in multiple indicators such as peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and perceived image quality (LPIPS) on multiple standard datasets (such as RealSR and NTIRE2020). For example, compared with FBAnet, SCAUAN improved PSNR by 0.4dB, increased SSIM by 0.0237, and reduced LPIPS by 0.0214. This shows that the present invention can generate higher quality super-resolution images in various application scenarios.

[0065] 7. Improve the ability to restore details in complex areas: The network model of the present invention effectively captures the uncertainty of complex areas (such as textures and edges) through the cross-resolution hybrid attention block (CRHAB), and can adaptively optimize the reconstruction effect of these areas, significantly improving the ability to restore details of the image. Experiments show that the network model of the present invention is significantly better than existing methods in detail restoration in high-uncertainty areas such as complex textures and edges, and the generated images are more natural and consistent with human visual perception.

[0066] 8. Comprehensively improve visual effects: In visual comparison experiments, the network model of the present invention can better preserve the edge and texture details of the image. Compared with traditional methods (such as Bicubic interpolation, EDSR, etc.), SCAUAN can not only better restore image details, but also avoid visual distortion, making the generated high-resolution images clearer and more natural, and the visual effects are more in line with the subjective feelings of the human eye.

[0067] 9. Strong model generalization ability: Through training and testing on a variety of data sets, the network model of the present invention has demonstrated excellent generalization ability. Whether in synthetic data sets (such as DIV2K) or in real-world super-resolution scenarios (such as RealSR, NTIRE2020), the network model of the present invention has shown high robustness and adaptability, and can stably output high-quality super-resolution images in various scenarios.

[0068] 10. Efficient training and optimization process: The present invention ensures that the model can converge quickly and achieve optimal performance through a series of efficient training steps, including data enhancement, forward propagation, loss calculation driven by heteroscedastic uncertainty, and Adam optimizer weight update. Experiments show that SCAUAN can achieve better results than existing methods within a reasonable training time, improving training efficiency.

[0069] In summary, the network model proposed in the present invention effectively solves the shortcomings of existing super-resolution methods in detail restoration and high-uncertainty area processing through innovative network structure design and efficient training methods, greatly improves image reconstruction performance and visual effects, and has significant application value and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1a is the overall structure diagram of the single image super-resolution network model of the present invention, Figure 1b This is a schematic diagram of the cross-scale self-convolutional attention mechanism;

[0071] Figure 2 It is a training flow chart of the present invention;

[0072] Figure 3a and Figure 3b This is a comparison chart of the super-resolution reconstruction effects of various algorithms on the RealSR dataset. DETAILED DESCRIPTION

[0073] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0074] The single image super-resolution network model proposed in this embodiment can enhance context perception and local feature extraction in the cross-resolution feature domain by introducing the cross-scale self-convolution attention mechanism (CSSCA) and the heteroscedastic uncertainty driven loss function. Specifically, CSSCA uses image blocks of different resolutions as convolution kernels and performs convolution operations with the input feature map, thereby effectively capturing the self-similarity across resolutions. At the same time, the heteroscedastic uncertainty driven loss function dynamically adjusts the network's attention to different uncertainty areas, further improving the reconstruction effect of textures and edges.

[0075] like Figure 1a As shown in the figure, the single image super-resolution network model consists of three main modules: shallow feature extraction module (Shallow Feature Extraction, SFE), deep feature extraction module (Deep Feature Extraction, DFE) and image reconstruction module (Image Reconstruction, IR).

[0076] 1. Shallow Feature Extraction Module (SFE):

[0077] This module performs shallow feature extraction on the input low-resolution (LR) image through a convolutional layer to generate the initial feature F0:

[0078] F0=H SF (ILR ),

[0079] in, represents the input low-resolution image, H and W are the height and width of the low-resolution image, and C is the number of channels of the low-resolution image; H SF (·) represents the mapping function of the shallow feature extraction module, which is implemented through a convolutional layer, and F0 is the generated initial feature.

[0080] 2. Deep Feature Extraction Module (DFE):

[0081] The deep feature extraction module is the core part of the network model of the present invention, which contains multiple residual attention groups (RAGs), each of which is composed of multiple cross-resolution hybrid attention blocks (CRHABs). The function of the DFE module is to extract the deep feature F from the initial feature F0. DF :

[0082] F DF =H DF (F0),

[0083] in, is the deep feature, H DF (·) represents the mapping function of the DFE module. The deep feature extraction module contains M RAGs and a 3×3 convolutional layer.

[0084] The calculation process for each RAG is as follows:

[0085]

[0086] in, is the output feature of the i-th RAG, represents the mapping function of the i-th residual attention group, each residual attention group consists of CRHAB and convolutional layer, M represents the number of RAGs, which is set to 6 in this invention. For the input feature of the i-th RAG Extract intermediate features through N CRHABs is the output feature of the j-th cross-resolution hybrid attention block CRHAB, j = 1, 2, ..., N, and the specific process can be expressed as:

[0087]

[0088] in, is the mapping function of the jth CRHAB in the i-th RAG. N is the number of CRHABs in each RAG, which is set to 6 in the present invention. Then, the output features of the last layer of CRHABs are convolved with the input features through a convolution layer. Perform residual connection and calculate the output of the i-th RAG:

[0089]

[0090] in, is the convolutional layer in the i-th RAG.

[0091] Finally, the output F of the last RAG M Input to the convolutional layer at the end of the deep feature extraction module DFE:

[0092] F DF =H CONV (F M ),

[0093] Among them, H CONV (·) represents the last convolutional layer of the deep feature extraction module DFE.

[0094] 3. Image reconstruction module (IR):

[0095] The image reconstruction module consists of a convolutional layer, a pixel reorganization layer (PixelShuffle) and another convolutional layer. First, the deep feature F DF The pixel is added to the initial feature F0, and then upsampled by the image reconstruction module to generate the final super-resolution output image I SRS :

[0096] I SRS =H REC (F0+F DF ),

[0097] in, is the output super-resolution image, H ′ and W ′ is the high-resolution image size after upsampling, H REC (·) represents the mapping function of the IR module.

[0098] Specifically, Cross-resolution Hybrid Attention Block (CRHAB): CRHAB is based on Swin Transformer [6] The architecture design introduces shifted window-based multi-head self-attention (SW-MSA) and cross-scale self-convolutional attention CSSCA, see Figure 1b .

[0099] First, the output features of the j+1th cross-resolution hybrid attention block CRHAB in the i-th residual attention group RAG It includes the following steps:

[0100] (201) Output features of the jth CRHAB in the i-th RAG Normalized through the LayerNorm layer:

[0101]

[0102] Among them, H LN (·) indicates the operation of the LayerNorm layer. is the output feature of the j-th cross-resolution hybrid attention block CRHAB, j = 1, 2, ..., N; F x is the normalization result;

[0103] (202) Self-similarity information is extracted through parallel SW-MSA and CSSCA. Shifted Window-based Multi-Head Self-Attention (SW-MSA) can use pixels outside the window to optimize the reconstruction results. The CSSCA module extracts self-similarity information through cross-resolution self-convolution to enhance context modeling and local feature extraction:

[0104]

[0105] Among them, H SW-MSA (·) represents the shifted window multi-head self-attention, H CSSCA (·) represents cross-scale self-convolutional attention CSSCA. F y This is the result after enhanced context modeling and local feature extraction;

[0106] (203) After another LayerNorm layer and a multi-layer perceptron (MLP), the output features of the j+1th CRHAB in the i-th RAG are generated:

[0107]

[0108] Among them, H MLP (·) represents the mapping of a multilayer perceptron.

[0109] Secondly, the Cross-Scale Self-Convolutional Attention Mechanism Module (CSSCA): The CSSCA module is designed to extract feature self-similarity at different resolutions to enhance context modeling and local feature extraction. The specific process is as follows:

[0110] Normalization results After upsampling, it is divided into multiple high-resolution pixel blocks p of size d×d i , arranged into a high-resolution pixel block sequence denoted as P = [p1,…,p K ], where d=8 and K is the number of blocks.

[0111] High-resolution pixel blocks are used as convolution kernels and normalized processing results F x Perform convolution operation and extract the convolution feature p ci , get the self-convolution feature sequence: P CSSCA =[p c1 ,…,p cK ] Among them, p ci =F x *p i , * indicates convolution operation, with a step size of 1. CSSCA Each term in the equation refers to the normalized result F x and high-resolution pixel blocks p i The self-convolution features obtained after the convolution operation;

[0112] Afterwards, the convolutional layer is fused to obtain the cross-scale self-convolution features:

[0113] F CSSCA =H CONV (P CSSCA ),

[0114] Among them, H CoNV (·) represents the mapping of the convolutional layer.

[0115] Finally, heteroscedastic uncertainty-driven loss function: Traditional super-resolution models usually process all image regions uniformly, ignoring high uncertainty regions such as edges and textures. This embodiment introduces a heteroscedastic uncertainty-driven loss function (Heteroscedastic Uncertainty-Driven Loss Function, L HUL ), which is used to dynamically adjust the weights of high uncertainty areas and enhance the ability to restore texture and edge details. The specific loss function is defined as follows:

[0116] The loss function includes pixel uncertainty loss (Pixel Uncertainty Loss, L pix ) and feature uncertainty loss (Feature Uncertainty Loss, L per ):

[0117] L HUL =α pix L pix +αper L per ,

[0118] Among them, α pix =1 and α per = 0.01 is the optimal weight. The pixel uncertainty loss is based on the variance of the HR image, giving a larger weight to the high variance area:

[0119]

[0120] Where N is the total number of pixels in the image, and are high-resolution images and super-resolution output images respectively, is the pixel uncertainty weight, s i =ln(Var(I HR )+e -2 ). The feature domain uncertainty loss enhances attention to high uncertainty regions through perceptual loss:

[0121]

[0122] Among them, φ() is the use of VGG19 [7] The feature extractor implemented, t i =ln(Var(φ(I HR ))+e -2 ).

[0123] Preferably, this embodiment also provides a training method based on the above super-resolution network model. Figure 2 The training process is shown as follows:

[0124] Step 1. Extract training images from the dataset;

[0125] The training images are extracted from standard single image super-resolution datasets such as DIV2K, which contain low-resolution (LR) and high-resolution (HR) image pairs. The LR images are used as input and the HR images are used as reference targets.

[0126] Step 2. Perform data enhancement on the training images;

[0127] The extracted training images are subjected to data augmentation processing to increase the diversity of training data and the generalization ability of the super-resolution network model. Commonly used data augmentation methods include random cropping, rotation, horizontal flipping, color jittering, etc.

[0128] Step 3. Input the enhanced image into the super-resolution network model for forward propagation and calculate the loss function;

[0129] The enhanced low-resolution image is input into the super-resolution network model, and after the forward propagation process of the super-resolution network model, a predicted super-resolution image (SR) is generated. The heteroscedastic uncertainty-driven loss function between the predicted image and the true high-resolution image is calculated.

[0130] Step 4. Use the Adam optimizer to update the super-resolution network model weights;

[0131] The gradient of the loss function is calculated using the back-propagation algorithm, and the Adam optimizer is used to update the weights of the super-resolution network model.

[0132] Step 5. Repeat steps 2-4 until the training is completed;

[0133] Repeat the steps of data augmentation, forward propagation, loss calculation, and updating super-resolution network model weights to continuously optimize the super-resolution network model parameters until a predetermined number of training rounds (epochs) or other termination conditions are reached.

[0134] Step 6. Evaluate the super-resolution network model performance;

[0135] The performance of the trained super-resolution network model is evaluated on the test set, image quality indicators such as PSNR and SSIM are calculated, and compared with existing methods to analyze the effect and superiority of the super-resolution network model.

[0136] Specifically, about the data set and training details settings:

[0137] Dataset: This example uses RealSR [8] and NTIRE2020 [9] The dataset is used for training. RealSR contains 200 pairs of LR-HR image pairs for training and 50 pairs of test sets. NTIRE2020 contains two subtasks: one using DIV2K[ 10] Degraded images synthesized from the dataset (Track1), and another using DPED

[11] Real LR images in the dataset (Track2) and evaluated without reference HR images.

[0138] Training details: During the training of the SCAUAN network, the number of RAGs and CRHABs was set to 6. The number of attention heads in each CRHAB was set to 6, and the window size was 16. The number of output channels of the last convolutional layer of the network was 3, generating a three-channel RGB image, and the number of output channels of the remaining convolutional layers was fixed to 64. The kernel size of all convolutional layers was 3×3, the stride was 1, and “SAME” padding was used to maintain the feature map size. Training used the ADAM optimizer (β1=0.9,β2=0.999,∈=10-8 ), the batch size is 16, and the initial learning rate is 10 -4 , the learning rate was halved every 500 iterations, for a total of 2000 iterations.

[0139] In order to verify the effectiveness of the super-resolution network model machine method proposed in the present invention, this embodiment is compared with a variety of existing mainstream single image super-resolution algorithms. The comparison methods include Bicubic interpolation, EDSR [2] 、ZSSR

[12] ESRGAN [3] , FSSR

[13] DASR

[14] 、Impressionism

[15] 、DeFlow

[16] 、USR-DU

[17] and FBAnet

[18] The experiments were conducted on multiple standard real-scene datasets (such as RealSR and NTIRE2020), using multiple objective evaluation indicators and subjective visual effect comparisons such as peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and perceptual image quality assessment index (LPIPS).

[0140] Table 1 shows the quantitative analysis results of various methods on the RealSR test set. The super-resolution network model SCAUAN proposed in this invention achieves the best performance in terms of PSNR, SSIM, LPIPS and other indicators. Specifically, the method of this invention is 0.4dB higher than the FBAnet method in PSNR, 0.0237 higher in SSIM, and 0.0214 lower in LPIPS.

[0141] Table 2 shows the quantitative analysis results of various methods on the NTIRE2020 test set. Compared with the recent state-of-the-art real-world super-resolution networks, the super-resolution network model SCAUAN proposed in this paper shows strong performance in all indicators. It is worth noting that SCAUAN improves PSNR by 1.12dB, increases SSIM by 0.06, and reduces LPIPS by 0.0252 on the Track1 dataset compared to the NTIRE2020 challenge champion Impressionism. On the Track2 dataset, the NIQE value of SCAUAN is reduced by 0.7951.

[0142] Figure 3a and Figure 3bA comparison chart of the reconstruction effects of various algorithms on the RealSR dataset is shown. It can be seen that traditional methods (such as Bicubic, EDSR, etc.) perform poorly in detail recovery, while advanced methods (such as ESRGAN, FBAnet, etc.) can better recover some details, but there are still visual distortions. The SCAUAN method proposed in the present invention can better maintain the edge and texture details of the image, and the generated image is clearer and more natural, which conforms to human visual perception. It can be seen from the experimental results that the method proposed in the present invention has significant advantages in the single image super-resolution task, and can effectively recover more details in high uncertainty areas (such as complex textures and edges), improve image quality and visual effects, and verify the effectiveness and robustness of the method proposed in the present invention.

[0143] Table 1 Comparison results of objective evaluation indicators of various algorithms on the RealSR dataset

[0144]

[0145]

[0146] Table 2 Comparison results of objective evaluation indicators of each algorithm on the NTIRE2020 dataset

[0147]

[0148] Preferably, the embodiments of the present application also provide a specific implementation of an electronic device capable of implementing all steps in the uncertainty-aware single image super-resolution network model based on self-convolutional attention in the above embodiments, and the electronic device specifically includes the following contents:

[0149] Processor, memory, communications interface and bus;

[0150] Among them, the processor, memory, and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between related devices such as server-side devices, metering devices, and user-side devices.

[0151] The processor is used to call the computer program in the memory, and when the processor executes the computer program, all functions of the uncertainty-aware single image super-resolution network model based on self-convolution attention in the above embodiment are implemented.

[0152] An embodiment of the present application also provides a computer-readable storage medium capable of implementing all steps in the uncertainty-aware single image super-resolution network model based on self-convolution attention in the above-mentioned embodiment. A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, all functions of the uncertainty-aware single image super-resolution network model based on self-convolution attention in the above-mentioned embodiment are implemented.

[0153] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0154] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0155] Although the present application provides method operation steps such as embodiments or flow charts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0156] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0157] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0158] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0159] The present invention is not limited to the embodiments described above. The above description of the specific embodiments is intended to describe and illustrate the technical solution of the present invention. The above specific embodiments are merely illustrative and not restrictive. Without departing from the scope of the present invention and the scope of protection of the claims, a person of ordinary skill in the art can also make many forms of specific changes under the guidance of the present invention, which all fall within the scope of protection of the present invention.

[0160] References:

[0161] [1]Dong C, Loy CC, He K, et al.Image super-resolution using deepconvolutional networks[J]. IEEE Transactions on Pattern Analysis and MachineIntelligence, 2015, 38(2):295-307.

[0162] [2]Lim B,Son S,Kim H,et al.Enhanced deep residual networks for singleimage super-resolution[C] / / Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops.IEEE,2017:136-144.

[0163] [3]Wang

[0164] [4]Liang J,Cao J,Sun G,et al.SwinIR:Image restoration using swintransformer[C] / / Proceedings of the IEEE / CVF International Conference onComputer Vision.IEEE,2021:1833-1844.

[0165] [5]Zamir S W,Arora A,Khan S,et al.Restormer:Efficient transformer forhigh-resolution image restoration[C] / / Proceedings of the IEEE / CVF Conferenceon Computer Vision and Pattern Recognition.IEEE,2022:5728-5739.

[0166] [6]Liu Z,Lin Y,Cao Y,et al.Swin transformer:Hierarchical visiontransformer using shifted windows[C] / / Proceedings of the IEEE / CVFInternational Conference on Computer Vision.IEEE,2021:10012-10022.

[0167] [7]Simonyan K,Zisserman A.Very deep convolutional networks for large-scale image recognition[J / OL].arXiv preprint arXiv:1409.1556,2014.

[0168] [8]Cai J,Zeng H,Yong H,et al.Toward real-world single image super-resolution:A new benchmark and a new model[C] / / Proceedings of the IEEE / CVFInternational Conference on Computer Vision.IEEE,2019:3086-3095.

[0169] [9]Lugmayr A,Danelljan M,Timofte R.NTIRE 2020 challenge on real-worldimage super-resolution:Methods and results[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition Workshops.IEEE,2020:494-495.

[0170]

[10] Timofte R,Agustsson E,Van Gool L,et al.NTIRE 2017 challenge onsingle image super-resolution:Methods and results[C] / / Proceedings of the IEEEConference on Computer Vision and Pattern Recognition Workshops.IEEE,2017:114-125.

[0171]

[11] Ignatov A,Kobyshev N,Timofte R,et al.DSLR-quality photos onmobile devices with deep convolutional networks[C] / / Proceedings of the IEEEInternational Conference on Computer Vision.IEEE,2017:3277-3285.

[0172]

[12] Shocher A,Cohen N,Irani M."Zero-shot"super-resolution using deepinternal learning[C] / / Proceedings of the IEEE Conference on Computer Visionand Pattern Recognition.IEEE,2018:3118-3126.

[0173]

[13] Fritsche M,Gu S,Timofte R.Frequency separation for real-worldsuper-resolution[C] / / 2019 IEEE / CVF International Conference on ComputerVision Workshop(ICCVW).IEEE,2019:3599-3608.

[0174]

[14] Wei Y,Gu S,Li Y,et al.Unsupervised real-world image superresolution via domain-distance aware training[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition.IEEE,2021:13385-13394.

[0175]

[15] Cai J,Zeng H,Yong H,et al.Toward real-world single image super-resolution:A new benchmark and a new model[C] / / Proceedings of the IEEE / CVFInternational Conference on Computer Vision.IEEE,2019:3086-3095.

[0176]

[16] Wolf V,Lugmayr A,Danelljan M,et al.DeFlow:Learning complex imagedegradations from unpaired data with conditional flows[C] / / Proceedings of theIEEE / CVF Conference on Computer Vision and Pattern Recognition.IEEE,2021:94-103.

[0177]

[17] Ning Q,Tang J,Wu F,et al.Learning degradation uncertainty forunsupervised real-world image super-resolution[C] / / Proceedings of the 31stInternational Joint Conference on Artificial Intelligence.AAAI Press,2022:1261-1267.

[0178]

[18] Wei P,Cai J,Wang Z,et al.Towards real-world burst image super-resolution:Benchmark and method[C] / / Proceedings of the IEEE / CVF InternationalConference on Computer Vision.IEEE,2023.

Claims

1. An uncertainty-aware single image super-resolution network model based on self-convolutional attention, characterized in that: include: Shallow feature extraction module SFE, deep feature extraction module DFE and image reconstruction module IR; The shallow feature extraction module SFE extracts shallow features from the input low-resolution image through a convolutional layer to generate the initial feature F0: F0=H SF (I LR ), in, represents the input low-resolution image, H and W are the height and width of the low-resolution image, and C is the number of channels of the low-resolution image; H SF (·) represents the mapping function of the shallow feature extraction module, which is implemented through a convolutional layer, and F0 is the generated initial feature; The deep feature extraction module DFE includes several residual attention groups RAG, each of which is composed of several cross-resolution hybrid attention blocks CRHAB; the deep feature extraction module DFE is used to extract the deep feature F from the initial feature F0 DF : F DF =H DF (F0), in, is the deep feature, H DF (·) represents the mapping function of the deep feature extraction module DFE, which contains M residual attention groups RAG and a 3×3 convolutional layer; The image reconstruction module IR consists of a convolutional layer, a pixel reorganization layer and another convolutional layer. DF The pixel is added to the initial feature F0, and then upsampled by the image reconstruction module IR to generate the final super-resolution output image I SRS : I SRS =H REC (F0+F DF ), in, is the output super-resolution image, H ′ and W ′ is the high-resolution image size after upsampling, H REC (·) represents the mapping function of the IR module.

2. According to claim 1, the uncertainty-aware single image super-resolution network model based on self-convolutional attention is characterized in that: The calculation process of each residual attention group RAG in the deep feature extraction module DFE is as follows: in, is the output feature of the i-th residual attention group RAG, represents the mapping function of the i-th residual attention group. Each residual attention group consists of a cross-resolution hybrid attention block CRHAB and a convolutional layer. M represents the number of residual attention groups RAG, M = 6; for the input feature of the i-th residual attention group RAG Extracting intermediate features through N cross-resolution hybrid attention blocks CRHAB is the output feature of the j-th cross-resolution hybrid attention block CRHAB, j = 1, 2, ..., N, and the specific process is expressed as: in, is the mapping function of the jth cross-resolution hybrid attention block CRHAB in the i-th residual attention group RAG; N is the number of cross-resolution hybrid attention blocks CRHAB in each residual attention group RAG, N = 6; the output features of the last layer of cross-resolution hybrid attention blocks CRHAB are combined with the input features through a convolutional layer Perform residual connection and calculate the output of the i-th residual attention group RAG: in, is the convolutional layer in the i-th residual attention group RAG; Finally, the output F of the last residual attention group RAG is M Input to the convolutional layer at the end of the deep feature extraction module DFE: F DF =H CONV (F M ), Among them, H CONV (·) represents the last convolutional layer of the deep feature extraction module DFE.

3. The uncertainty-aware single image super-resolution network model based on self-convolutional attention according to claim 1 or 2, characterized in that: The cross-resolution hybrid attention block CRHAB includes shifted window multi-head self-attention SW-MSA and cross-scale self-convolutional attention CSSCA; Output features of the j+1th cross-resolution hybrid attention block CRHAB in the i-th residual attention group RAG It includes the following steps: (1) Output features of the jth cross-resolution hybrid attention block CRHAB in the i-th residual attention group RAG Normalized through the LayerNorm layer: Among them, H LN (·) indicates the operation of LayerNorm layer; is the output feature of the j-th cross-resolution hybrid attention block CRHAB, j = 1, 2, ..., N; F x is the normalization result; (2) Extract self-similarity information through parallel shifted window multi-head self-attention SW-MSA and cross-scale self-convolutional attention CSSCA; shifted window multi-head self-attention SW-MSA uses pixels outside the window to optimize the reconstruction result; CSSCA extracts self-similarity information through cross-resolution self-convolution to enhance context modeling and local feature extraction: Among them, H SW-MSA (·) represents the shifted window multi-head self-attention, H CSSCA (·) represents cross-scale self-convolutional attention CSSCA; F y This is the result after enhanced context modeling and local feature extraction; (3) After another LayerNorm layer and a multi-layer perceptron MLP, the output features are generated: Among them, H MLP (·) represents the mapping of multilayer perceptron; The cross-scale self-convolutional attention mechanism module CSSCA is used to extract feature self-similarity at different resolutions to enhance context modeling and local feature extraction; the specific process is as follows: Normalization results After upsampling, it is divided into several high-resolution pixel blocks p of size d×d i , arranged into a high-resolution pixel block sequence denoted as P = [p1,…,p K ], where d = 8 and K is the number of blocks; High-resolution pixel blocks are used as convolution kernels and normalized processing results F x Perform convolution operation and extract the convolution feature p ci , get the self-convolution feature sequence: P CSSCA =[p c1 ,…,p cK ] Among them, p ci =F x *p i , * indicates convolution operation, with a step size of 1; P CSSCA Each term in the equation refers to the normalized result F x and high-resolution pixel blocks p i The self-convolution features obtained after the convolution operation; Afterwards, the convolutional layer is fused to obtain the cross-scale self-convolution features: F CSSCA =H CONV (P CSSCA ), Among them, H CONV (·) represents the mapping of the convolutional layer.

4. A training method based on the super-resolution network model according to any one of claims 1 to 3, characterized in that: include: S1. Extract training images from the dataset; Extract training images from the standard single image super-resolution dataset, which contains low-resolution LR and high-resolution HR image pairs; The low-resolution LR image is used as input, and the high-resolution HR image is used as the reference target; S2. Perform data enhancement processing on the training images; Perform data augmentation on the extracted training images to increase the diversity of training data and the generalization ability of the super-resolution network model; Data augmentation methods include random cropping, rotation, horizontal flipping, and color jittering; S3. Input the enhanced image into the super-resolution network model for forward propagation and calculate the loss function; The enhanced low-resolution image is input into the super-resolution network model, and a predicted super-resolution image is generated through the forward propagation process of the super-resolution network model; a heteroscedastic uncertainty-driven loss function between the predicted super-resolution image and the true high-resolution image is calculated; S4. Update the super-resolution network model weights using the Adam optimizer; Use the back-propagation algorithm to calculate the gradient of the loss function and use the Adam optimizer to update the weights of the super-resolution network model; S5. Repeat S2-S4 until the training is completed; Repeat data enhancement, forward propagation, loss calculation, and updating of super-resolution network model weights S, and continuously optimize super-resolution network model parameters until a predetermined number of training rounds or other termination conditions are reached; S6. Evaluate the performance of super-resolution network models; Evaluate the performance of the trained super-resolution network model on the test set.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the function of the uncertainty-aware single image super-resolution network model based on self-convolutional attention as described in any one of claims 1 to 3 is realized.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it realizes the functions of the uncertainty-aware single image super-resolution network model based on self-convolutional attention as described in any one of claims 1 to 3.

Images