A SISR network training method and system based on adaptive weighted loss

Abstract

The present invention relates to a SISR network training method and system, and more specifically to a SISR network training method and system based on adaptive weighted loss. The method overcomes the problem that the existing single image super-resolution method based on deep learning has complex neural network model structure and too many parameters to improve network performance, which makes it difficult to apply to actual scenes. First, the overall observation model is constructed; second, the low-resolution LR image is input to the SISR network to train the SISR network, and the f(y i ) and σ i ; Then, the σ learned in step 2 is i As a guide, the adaptive weighted loss function is calculated; finally, the loss function #imgabs0# is used to guide SISR network training. By assigning higher weights to image texture and edge regions with greater uncertainty during training, this method not only improves objective performance but also restores higher visual quality without adding additional computational complexity during inference.

Description

Technical Field

[0001] The present invention relates to a SISR network training method and system, and in particular to a SISR network training method and system based on adaptive weighted loss. Background Art

[0002] In recent years, the rapid development of image sensors and the continuous updating of various image processing technologies have enabled the application of high-quality images and videos in fields closely related to people's lives, such as video surveillance (improving public safety and assisting in solving crimes), medical imaging (helping with disease analysis and disease location), high-definition video (HDTV, ultra-HD cinema, etc.), and remote sensing applications (meteorological monitoring, environmental analysis, military protection, etc.).

[0003] Today, as people's quality of life improves, their demands for image resolution are also increasing. The most direct way to obtain high-resolution images is to use high-resolution image sensors. However, due to the limitations of sensor and optical device manufacturing processes and costs, this is difficult to achieve in most scenarios. Furthermore, external influences such as noise, scene motion, and blur that may occur during the imaging process often result in images that fail to meet people's expectations in real life. Therefore, the task of achieving higher resolution in images has attracted widespread attention.

[0004] Single Image Super Resolution (SISR) is a fundamental computer vision task that aims to reconstruct a high-resolution (HR) image from its corresponding degraded low-resolution (LR) image and enhance its details to achieve a pleasing image. Generally speaking, this problem is inherently ill-posed because there are always multiple HR images corresponding to a single LR image. However, it has long been recognized that image textures and edges contain more important visual information than smooth regions.

[0005] To date, various classic single-image super-resolution methods have been proposed, which can be roughly divided into three types: interpolation-based methods, reconstruction-based methods, and learning-based methods. In recent years, with the rapid development of deep learning technology, single-image super-resolution methods based on deep learning have been actively explored. Compared with traditional single-image super-resolution methods that suffer from high computational complexity and suboptimal results, single-image super-resolution methods based on deep learning have shown superior performance due to their flexible learning strategies, powerful training tools, and rich datasets.

[0006] Single image super-resolution methods based on deep learning have been widely studied in super-resolution due to their powerful nonlinear representation capabilities. SRCNN was one of the first methods to introduce deep learning into SISR (see: Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning a deep convolutional network for image super-resolution. In European Conference on Computer Vision, pages 184–199, 2014). It consists of only three convolutional layers and achieves high-resolution image recovery through feature extraction, nonlinear mapping, and feature reconstruction.After that, in response to the shortcomings of existing research work, a series of works on SISR methods based on deep learning were proposed, such as VDSR and EDSR, which designed deeper network structures based on residual learning, see Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image Super-Resolution Using Deep Convolutional. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2): 295-307, 2016. Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1132–1140, 2017., and RCAN and SAN, which improve the performance of super-resolution by considering the feature correlation of spatial dimension or channel dimension, see Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 286–301, 2018. Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang. Second-order attention network for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11065–11074, 2019.

[0007] Based on the model-guided design of deep neural networks, DPDNN and MoG-DUN are respectively shown in Weisheng Dong, Peiyao Wang, Wotao Yin, Guangming Shi, Fangfang Wu, and Xiaotong Lu.Denoising prior driven deep neural network for image restoration.IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(10):2305–2318, 2019. Qian Ning, Weisheng Dong, Guangming Shi, Leida Li, and Xin Li.Accurate and lightweight image super-resolution with model-guided deep unfolding network.IEEE Journal of Selected Topics in Signal Processing, 15(2):240–252, 2021.doi:10.1109 / JSTSP.2020.3037516. etc. It can be seen that the unifying theme along this research direction seems to be that deeper, larger, and more complex networks can achieve the purpose of improving SISR performance by promoting the reconstruction of high-frequency details and restoring the texture and edges of the image.

[0008] Although research continues in the direction of designing deeper, larger, and more complex networks to further improve the performance of neural network models, this will also result in complex model structures with too many parameters, making them difficult to apply in practical scenarios. Summary of the Invention

[0009] The purpose of the present invention is to provide a SISR network training method and system based on adaptive weighted loss, so as to overcome the problems that the existing single image super-resolution methods based on deep learning blindly design the neural network model to improve network performance, have complex structure and too many parameters, and are difficult to apply to actual scenarios.

[0010] The concept of the present invention is:

[0011] Most existing methods use MSE or Loss function to optimize network parameters. MSE or The loss function assumes that all pixels are equally important, treating each pixel equally, regardless of whether it is in a textured / edge region or a smooth area. However, in photographic images, texture and edge regions carry more important visual information than smooth regions. Therefore, by assigning higher weights to image texture and edge regions during training, this method not only improves objective performance but also restores higher visual quality without adding additional computational complexity during inference. This is ideal for scenarios requiring the production of high-definition images.

[0012] The technical solution of the present invention is to provide a SISR network training method based on adaptive weighted loss, which is special in that it includes the following steps:

[0013] Step 1: Construct an overall observation model;

[0014] x i =f(y i )+∈σ i ,

[0015] Where y represents the low-resolution LR image, x represents the high-resolution HR image corresponding to the low-resolution LR image, i is a pixel, f(·) represents any SISR network, and f(y i ) indicates that the SISR network learns the super-resolution SR image, ∈ represents the Laplace distribution with zero mean and unit variance, σ i represents the arbitrary uncertainty learned by the SISR network;

[0016] Step 2: Input low-resolution LR images to the SISR network, train the SISR network, and learn f(y i ) and σ i ;

[0017] Step 3: The σ learned in step 2 i As a guide, calculate the loss function for the adaptive weighted loss:

[0018]

[0019] where w i =w i -min(w i ) is a non-negative linear scaling function; w i =lnσ i , N is the number of pixels;

[0020] Step 4: Based on the loss function Guide SISR network training.

[0021] Furthermore, in step 2, we simultaneously learn f(y i ) and σi :

[0022]

[0023] Where L is the Laplace distribution, is the loss function.

[0024] The present invention also provides a SISR network training system based on adaptive weighted loss, which is special in that it includes a generation unit, a calculation unit and a training unit;

[0025] Generation unit, used to build the overall observation model;

[0026] x i =f(y i )+∈σ i

[0027] Where y represents the low-resolution LR image, x represents the high-resolution HR image corresponding to the low-resolution LR image, i is a pixel, f(·) represents any SISR network, and f(y i ) indicates that the SISR network learns the super-resolution SR image, ∈ represents the Laplace distribution with zero mean and unit variance, σ i represents the arbitrary uncertainty learned by the SISR network;

[0028] Computational unit, used to train the SISR network and learn f(y i ) and σ i , and the learned σ i As a guide, calculate the uncertainty-driven loss function

[0029]

[0030] in is a non-negative linear scaling function; w i =lnσ i , N is the number of pixels;

[0031] The training unit is used to train the network model based on the uncertainty-driven loss function.

[0032] Furthermore, the calculation unit simultaneously learns f(y i ) and σ i :

[0033]

[0034] Where L is the Laplace distribution, is the loss function.

[0035] The beneficial effects of the present invention are:

[0036] By introducing differential distribution estimation into the SISR network, the present invention can simultaneously learn the mean of the high-resolution image (super-resolution image) and its corresponding uncertainty (variance). Ultimately, pixels with higher certainty (e.g., texture and edge pixels) will be prioritized by SISR according to their importance to visual quality. This uncertainty-driven loss proposed by the present invention is more efficient than MSE or The experimental results on the commonly used SISR network show that the uncertainty-driven loss function proposed in this paper achieves better PSNR results and better visual effects than the traditional loss function without increasing any computational complexity during the test process. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 HR images, SR images reconstructed by the EDSR network, and the difference between the two on the dataset Set14; (a) is the dataset Set14, (b) is the HR image, (c) is the SR image reconstructed by the EDSR network, and (d) is the difference between the HR image and the SR image reconstructed by the EDSR network;

[0038] Figure 2 Schematic diagram of the SISR network training process based on adaptive weighted loss;

[0039] Figure 3 Comparison of the visual quality of EDSR-S trained with different loss functions on the Urban100 dataset (bicubic downsampling 4 times), where (a) is the HR image, (b) is the SR image reconstructed by the EDSR-S network, (c) is the image trained with the GRAM loss function, (d) is the image trained with the UDL loss function, and (e) is The image trained by the loss function, (f) is Uncertainty obtained from loss function training;

[0040] Figure 4 Comparison of visual quality of DPDNN trained with different loss functions on the Urban100 dataset (bicubic downsampling 4 times), where (a) is the HR image, (b) is the SR image reconstructed by the DPDNN network, (c) is the image trained with the Gram loss function, (d) is the image trained with the UDL loss function, and (e) is The image trained by the loss function, (f) is Uncertainty obtained from loss function training; DETAILED DESCRIPTION

[0041] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in this field without creative work should fall within the scope of protection of the present invention.

[0042] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0043] Most existing methods use MSE or Loss function is used to optimize network parameters. As we all know, the traditional MSE or The loss function assumes that all pixels are equally important, regardless of whether they are in a texture / edge region or a smooth region. However, in photographic images, texture and edge regions carry more important visual information than smooth regions. For example, Figure 1 The HR image and the SR image reconstructed by the EDSR network are shown. It can be found that the restored texture areas (such as the baboon's hair) are not as good as the smooth areas (such as the baboon's nose). Figure 1 Figure (d) depicts the absolute difference between the HR image and the SR image reconstructed by the EDSR network. The spatial variation of this difference is evident from the figure. This observation indicates that the uncertainty in texture and edge regions, characterized by variance, is much greater than that in smooth regions. Therefore, addressing this uncertainty provides the foundation for the present invention.

[0044] Uncertainty in deep learning can be broadly categorized into two types. Epistemic / model uncertainty describes the degree of uncertainty a model has about its predictions. The other type, aleatoric / data uncertainty, refers to the noise inherent in the observed data. Alex Kendall and Yarin Gal, "What uncertainties do we need in bayesian deep learning for computer vision?" (I. Guyon, U.V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, Curran Associates, Inc., 2017), propose a Bayesian deep learning framework that combines aleatoric uncertainty with epistemic uncertainty for pixel-by-pixel semantic segmentation and deep regression tasks. Jie Chang, Zhonghao Lan, Changmao Cheng, and Yichen Wei. Data uncertainty learning in face recognition. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 5710-5719, 2020. Data uncertainty in face recognition is studied using the estimated mean and variance. The uncertainty-based loss functions proposed in these works can be summarized as:

[0045]

[0046] Where f(y i )and Denote the learned mean and variance, respectively. In these tasks, using the above loss function to treat pixels with high uncertainty as unreliable pixels and incur loss attenuation does improve their robustness to noisy data. However, in the SISR task, pixels with high uncertainty (such as complex textures or edge regions) should be prioritized because these regions are visually more important than pixels in smooth regions, so applying the above loss function to SISR will lead to performance degradation.

[0047] The latest progress related to the present invention is GRAM and UDL, see ChangWoo Lee and Ki-SeokChung.Gram: Gradient rescaling attention model for data uncertainty estimationin single image super resolution.In 2019 18th IEEE International ConferenceOn Machine Learning And ApplicatiohS(ICMLA), pages 8-13.IEEE, 2019.Qian Ning, Weisheng Dong, Xin Li, Jinjian Wu, and Guangming Shi.Uncertainty-driven loss for single image super-resolution.Advances in Neural Information ProcessingSystems, 34, 2021., among which GRAM analyzes the impact of arbitrary / data uncertainty on SISR reconstruction and reduces the loss attenuation of pixels with higher variance. This method achieves better results than directly applying the uncertainty loss function to SISR. However, when the pixel variance is high, the loss of GRAM is still attenuated, which still contradicts the intuition that pixels in texture and edge areas are more important. Therefore, since GRAM cannot prioritize pixels with high variance, it is still lower than the baseline method; UDL proposes a new uncertainty-driven loss function UDL by quantifying the uncertainty in SISR under a deep Bayesian framework, which makes the network focus more on pixels with higher variance to better reconstruct texture and edge areas, achieving better performance than the baseline method.

[0048] This paper proposes a new adaptive weighted loss function (KL-UDL). By modeling the difference between the real image x and the restored image f(y) as a Laplace distribution L(x i -f(y i ), σ i ), and explicitly constrain this distribution to be close to the Laplace distribution L(O, I), that is, minimizing the Kullback Lebler (KL) divergence of the two Laplace distributions to quantify the uncertainty in SISR. The method proposed in this paper achieves better performance than the baseline method and also achieves better results than similar methods.

[0049] Compared with traditional MSE or Loss averaging treats each pixel differently. The new adaptive weighted loss function proposed in this paper prioritizes pixels where the restored image differs significantly from the ground truth. These pixels are mostly texture and edge pixels, which are visually more important than pixels in smooth areas. To this end, this paper introduces a method for simultaneously estimating the intermediate result (mean) and uncertainty (variance) of the SR image in SISR.

[0050] Previous studies have shown that arbitrary uncertainty can improve performance and increase robustness to noisy data in high-level vision tasks such as image segmentation. This improvement can be explained by attenuating the weights of pixels with higher uncertainty. However, in low-level vision tasks such as SISR, the attenuation must be done in the opposite direction - that is, pixels with higher uncertainty should be assigned larger weights (for example, texture and edge pixels) because they are visually more important than pixels in smooth areas. It should be noted that existing work such as GRAM cannot recognize this difference and does not prioritize pixels with higher uncertainty, while UDL recognizes this difference. Unlike the UDL method, this paper proposes a new adaptive weighted loss - KL-based Uncertainty-Driven Loss (KL-UDL) based on KL divergence, which focuses on the difference distribution between the restored image and the real image in SISR.

[0051] There are two types of uncertainty in Bayesian models: arbitrary uncertainty that captures the inherent noise in the observation data and epistemic uncertainty that explains the uncertainty of the model's predictions. This paper chooses to study the former (arbitrary uncertainty) and explores its application in SISR by designing a new uncertainty-driven loss function. In order to better quantify the arbitrary uncertainty in SISR, we use y i , x i denote the LR image and the corresponding high-resolution HR image respectively. Let f(·) denote any SISR network, and any uncertainty can be expressed as an additive term σ i In this way, the overall observation model can be expressed as:

[0052] x i =f(y i )+∈σ i ,

[0053] where ∈ represents the Laplace distribution with zero mean and unit variance. The goal of existing deep learning-based SISR methods is to train the network to learn only the SR image (mean) f(y i ). In order to more accurately describe the arbitrary uncertainty of SISR, the present invention not only estimates the SR image (mean) f(y i ), and at the same time estimate the uncertainty (variance) σi .

[0054] First, let’s explain the KL divergence of the two Laplace distributions. The Laplace distribution p(z) and the Laplace distribution q(z) can be expressed as:

[0055]

[0056]

[0057] Where p(z) means that z obeys the Laplace distribution with mean μ1 and variance σ1, and q(z) means that z obeys the Laplace distribution with mean μ2 and variance σ2.

[0058] Then, the KL divergence of p(z) and q(z) can be expressed as:

[0059]

[0060] When p(z)=L(x i -f(y i ), σ i ), q(z)=L(0,I), that is, the present invention converts the real image x i and the restored image f(y i ) is modeled as a Laplace distribution, hoping to fit the distribution of this difference into a Laplace distribution with zero mean and standard deviation, so we can get:

[0061]

[0062] Among them, f(y i ) and σ i They represent the SR image (mean) and uncertainty (variance) learned by the deep neural network respectively.

[0063] use The loss can accurately estimate uncertainty (variance), but it only brings a weak performance improvement in SISR, as shown in Table 1. The reason for this observation is that The loss function does not include the variance term (σ i ) is separated, it is difficult to control how much the current pixel should be penalized. Figure 1 As shown in , pixels with larger uncertainty carry important visual information such as texture and edge, so pixels with larger variance should be given greater weight.

[0064] Table 1 Study on five datasets of BI degradation KL Average PSNR and SSIM results of loss function

[0065]

[0066] In order to better rank pixels with large uncertainty, a new adaptive weighted loss, KL-based uncertainty-driven loss (KL-UDL), is proposed:

[0067]

[0068] in is a non-negative linear scaling function. In order to prevent the uncertainty value from degenerating to zero, the result of the uncertainty estimation network in the first step is passed to the second step (w i =lnσ i ),like Figure 2 shown.

[0069] By using logarithmic variance to represent complex pixels with high uncertainty, this paper proposes a new weighted loss - uncertainty driven loss exist In the loss, regions with higher uncertainty, such as image textures and edges, tend to have greater weights than pixels in smooth areas. In summary, the estimated uncertainty σ i As a bridge connecting two steps: it is the output of the first step; but it is transferred to the second step as a calculation Loss guidance.

[0070] The present invention can also provide a SISR network training system based on adaptive weighted loss, comprising a generation unit, a calculation unit and a training unit; the generation unit is used to construct an overall observation model; the calculation unit is used to train the SISR network, and simultaneously learns f(y i ) and σ i :

[0071]

[0072] The computation unit is also used to convert the learned σ i As a guide, calculate the uncertainty-driven loss function;

[0073]

[0074] The training unit is used to train the network model based on the uncertainty-driven loss function.

[0075] To demonstrate the effectiveness of the proposed uncertainty-driven loss function, experiments were conducted on five benchmark datasets using a BI degradation model, namely bicubic downsampling. Furthermore, two different networks were selected to verify the effectiveness of the proposed uncertainty-driven loss function. The first is EDSR-S, which consists primarily of 16 ResBlocks with 64 channels and has 1.5M parameters. The second is DPDNN, whose denoising network is a U-net within a model-guided framework.

[0076] Table 2 Average PSNR and SSIM results of BI degradation on 5 benchmark datasets

[0077]

[0078] For bicubic downsampling (BI), we compare our proposed Loss function, UDL, GRAM and original loss function (MSE or The average PSNR (peak signal-to-noise ratio) and SSIM (structural similarity) results in Table 2 are obtained from retraining of the corresponding papers or officially released codes. The loss function outperforms GRAM and the original loss function (MSE or ), which is almost better than the UDL loss function. Moreover, the improvement achieved by the method proposed in the present invention does not bring any additional computational cost during the test. By comparing the cases of EDSR-S and DPDNN, the proposed method It can bring greater performance improvement than large networks. The visual image comparison results are as follows Figure 3 and Figure 4 As shown. Figure 3 As shown, the present invention proposes Better recovery than the original loss function, GRAM and UDL. Figure 3 (f) describes our The learned uncertainty reveals challenging pixels where reconstruction performance is poor. Figure 4 As can be seen in (e), the lines of the window are restored more clearly, and the uncertainty of the precise estimate is shown in (f). The visual quality improvement is mainly due to the fact that our proposed method can fully utilize the captured uncertainty to train the deep network, thereby focusing on challenging pixels with higher uncertainty.

[0079] In summary, the present invention proposes a new adaptive weighted loss The algorithm is used to train the SISR network, which focuses on challenging pixels with high uncertainty (such as texture pixels and edge pixels). By introducing difference distribution estimation in SISR, the mean of the high-resolution image and its corresponding uncertainty (variance) can be learned simultaneously. Ultimately, pixels with high certainty (such as texture and edge pixels) will be prioritized by SISR according to their importance to visual quality. This paper proves that this uncertainty-driven loss is more efficient than MSE or The loss has better effects. Experimental results on the commonly used SISR network show that without increasing any computational effort during the test process, our proposed uncertainty-driven loss function based on KL divergence achieves better PSNR results and better visual effects than the traditional loss function.

Claims

1. A SISR network training method based on adaptive weighted loss, characterized in that: The following steps are involved: Step 1: Construct an overall observation model; x i =f(y i )+∈σ i , Where y represents the low-resolution LR image, x represents the high-resolution HR image corresponding to the low-resolution LR image, i is a pixel, f(·) represents any SISR network, and f(y i ) indicates that the SISR network learns the super-resolution SR image, ∈ represents the Laplace distribution with zero mean and unit variance, σ i represents the arbitrary uncertainty learned by the SISR network; Step 2: Input low-resolution LR images to the SISR network, train the SISR network, and learn f(y i ) and σ i ; Step 3: The σ learned in step 2 i As a guide, calculate the loss function for the adaptive weighted loss The loss function of the adaptive weighted loss Used to assign greater weight to texture and edge pixels; in is a non-negative linear scaling function; w i =lnσ i , N is the number of pixels; Step 4: Based on the loss function Guide SISR network training; In step 2, f(y i ) and σ i : Where L is the Laplace distribution, is the loss function.

2. A SISR network training system based on adaptive weighted loss, characterized by: Includes generation unit, calculation unit and training unit; Generation unit, used to build the overall observation model; x i =f(y i )+∈σ i Where y represents the low-resolution LR image, x represents the high-resolution HR image corresponding to the low-resolution LR image, i is a pixel, f(·) represents any SISR network, and f(y i ) indicates that the SISR network learns the super-resolution SR image, ∈ represents the Laplace distribution with zero mean and unit variance, σ i represents the arbitrary uncertainty learned by the SISR network; Computational unit, used to train the SISR network and learn f(y i ) and σ i , and the learned σ i As a guide, calculate the loss function for the adaptive weighted loss The loss function of the adaptive weighted loss Used to assign greater weight to texture and edge pixels; in is a non-negative linear scaling function; w i =lnσ i , N is the number of pixels; The training unit trains the network model using the adaptive weighted loss function; Among them, the calculation unit simultaneously learns f(y based on the following formula i ) and σ i : Where L is the Laplace distribution, is the loss function.

Images