Self-supervised real image denoising method based on sampling difference as perturbation

Abstract

The application discloses a self-supervised real image denoising method based on sampling difference as disturbance. Mainly includes the following steps: a random subsample generation strategy is proposed and used to obtain more subsamples; a sampling difference loss function is proposed according to the random subsample generation strategy; a cycle strategy is introduced into the construction of the loss function, and a cycle sampling difference loss function is proposed; the denoising network is trained through the above steps to obtain a denoising model, and then a noisy image is input into the trained network model to obtain a high-quality denoised image. The method disclosed by the application can be trained without clean images and is superior to existing pseudo-supervised / self-supervised methods, and is an effective self-supervised real image denoising method.

Description

TECHNICAL FIELD

[0001] The present application relates to image denoising technology, in particular to a self-supervised real image denoising method based on sampling difference as perturbation, belonging to the field of digital image processing. BACKGROUND

[0002] Image is a digital form of recording and representing information such as shape, color and texture of objects. It has the characteristics of intuition and image, and is indispensable in human production activities. With the development of science and technology and the improvement of people's living quality, the requirement for image quality has become more and more strict. In the field of medical imaging, high-quality images can provide accurate information needed for diagnosis and treatment; in the field of security monitoring, high-quality images can help identify and track suspects; in the field of graphics and design, high-quality images can create high-precision graphics and images; in the field of media and entertainment, high-quality images can provide a better viewing experience. Therefore, the quality of images has become a key factor for achieving effective results in various application fields. However, due to the defects of imaging systems, transmission media and recording devices, images are inevitably contaminated by noise. Noise will have a negative impact on the quality of images, reducing the readability and information content of images, so it is necessary to weaken or remove image noise through technical means to improve image quality. Therefore, image denoising has become a research hotspot in image processing.

[0003] Image denoising aims to suppress and remove noise in images by using mathematical models and algorithms, and finally recover high-quality images from noisy images. Image denoising can be divided into filtering-based methods, model-based methods and learning-based methods after years of research and development. At present, the method based on deep learning has become the mainstream in the field of image denoising due to its excellent performance. However, the existing real image denoising method based on deep learning still has room for improvement in terms of denoising quality and network model structure. SUMMARY

[0004] The purpose of the present application is to solve the problem of poor network performance caused by insufficient training set in unsupervised algorithm, so as to construct an effective self-supervised real image denoising method.

[0005] The self-supervised real image denoising method based on sampling difference as perturbation (SDAP) proposed by the present application mainly includes the following operation steps:

[0006] (1) a random sub-sample generation strategy is proposed and used to obtain more sub-samples;

[0007] (2) a sampling difference loss function is proposed according to the random sub-sample generation strategy;

[0008] (3) introduce a cycle strategy into the construction of the loss function, and propose a cycle sampling difference loss function;

[0009] (4) training the denoising network through the above steps to obtain a denoising model, and then inputting the image with noise into the trained network model to obtain a high-quality denoised image. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is the principle diagram of the self-supervised real image denoising method based on sampling difference as perturbation of the application. Wherein, (a) is the training structure diagram of SDAP of the application, (b) is the test and effect increasing structure diagram of SDAP.

[0011] Figure 2 is the result of adding perturbation to the training data and training the blind spot network.

[0012] Figure 3 is an example diagram of the random sub-sample generation strategy proposed by the application.

[0013] Figure 4 is a comparison diagram of the denoising results of the "SIDD" data set by the application and another six methods. Wherein, (a) is a noisy image, (b) to (g) are the denoising results of method 1 to method 6, and (h) is the denoising result of the application.

[0014] Figure 5 is a comparison diagram of the denoising results of the "DND" data set by the application and another six methods. Wherein, (a) is a noisy image, (b) to (g) are the denoising results of method 1 to method 6, and (h) is the denoising result of the application. DETAILED DESCRIPTION

[0015] The application will be further described below in conjunction with the drawings:

[0016] Figure 1 Among them, the self-supervised real image denoising method based on sampling difference as perturbation can be divided into the following steps:

[0017] (1) propose and use a random sub-sample generation (RSG) strategy to obtain more sub-samples;

[0018] (2) propose a sampling difference loss function according to the random sub-sample generation strategy;

[0019] (3) introduce a cycle strategy into the construction of the loss function, and propose a cycle sampling difference loss function;

[0020] (4) Through the above steps, the denoising network is trained, and a high-quality denoised image can be obtained by inputting a noisy image into the trained network model.

[0021] Specifically, in step (1), in order to solve the problem of insufficient training data in self-supervision, the present application adds a disturbance (Gaussian distribution with mean 0 and variance σ 2 , where σ is the standard deviation) to the training data, generates different images, and obtains more training data to train the Blind-Spot Network (BSN). Figure 2 The results prove that adding disturbance can improve the performance of BSN. Therefore, the present application regards the sampling difference between the sub-images obtained by different sampling methods as a kind of disturbance; inspired by this idea, the present application further generates s 2 sub-samples by using the RSG strategy, and s is a step factor. The detailed steps of the RSG strategy used in the present application are shown in Figure 3 . Assuming that the length and width of the noisy image y are H and W respectively, the detailed implementation process of the RSG strategy is as follows:

[0022] 1) Segmentation: the image y is divided into H×W / s 2 non-overlapping blocks with a size of s×s. Then, these blocks are stretched into vectors with a length of s 2 . In order to better visualize, Figure 3 s=2 is shown, and s is set to 5 in actual training.

[0023] 2) Shuffle: randomly shuffle the elements in each vector. Since this process is strictly random, each time a different sub-sample is obtained.

[0024] 3) Reorganization: take the elements in the same position of each vector to form s 2 new vectors with a length of H×W / s 2 .

[0025] 4) Recovery: each vector is reformed into a sub-sample with a size of [W / s]×[H / s]. In this way, s 2 similar sub-samples

[0026] In step (2), the present application can obtain s 2 trainable sub-samples from one noisy image by using the RSG strategy, and the present application regards the difference between RSG i (y) and RSG i+1 (y) as disturbance, and regards them as the input and target of BSN respectively; a new sampling difference loss function LSDBSN :

[0027] L SDBSN =E y {||B(RSG i (y)-RSG i+1 (y)||1}

[0028] where RSG i (y) represents the i-th sub-sample generated by RSG.

[0029] In step (3), the present application introduces a cycle strategy in the training stage, unlike the method of training BSN, which only needs two sub-samples for each training of each image, the present application will use all s 2 sub-samples generated. Therefore, the present application proposes a new cycle difference loss function L CSDBSN :

[0030]

[0031]

[0032] Since the sub-samples obtained by the RSG strategy are randomly generated, the training data is also random. L CSDBSN In each input, the target is different, so as to avoid the formation of fixed mapping of two sub-samples during the cycle. This cycle loss has the following advantages: 1. It applies constraints to all pixels of the original noisy image; 2. It ensures that all sub-samples generated by RSG can be fully utilized; 3. It makes the training of BSN more robust. Figure 1 (a) also intuitively illustrates the training method of the present application, which includes the RSG strategy and the cycle difference loss function.

[0033] In step (4), the present application uses random sub-samples generated by the RSG strategy to avoid the formation of fixed mapping of the denoising network, but the randomness inevitably leads to the decline of the performance of the denoising network in the test stage. Since each point of the pixel-shuffle downsampling (PD) sub-sample has a fixed position in the original image, it has a fixed offset direction, and its randomness is smaller than the RSG strategy proposed by the present application. Therefore, in step (4), the present application uses the PD sampling mode in the test stage to stabilize the test results. The specific way is shown in Figure 1 (b), the present application introduces the PD strategy in the test stage, and the present application denoises the denoised image again by BSN to remove the non-smooth artifacts and make the image more natural.

[0034] To better illustrate the effectiveness of the present application, comparisons were made on commonly used real noise image datasets "SIDD" and "DND". The comparison denoising algorithms are:

[0035] Method 1: BM3D method proposed by Dabov et al., reference "Dabov K, Foi A, Katkovnik V, et al. Color image denoising via sparse 3D collaborative filtering with grouping constraint in luminance-chrominance space [C]. IEEE International Conference on Image Processing, 2007, 1: I-313-I-6."

[0036] Method 2: TNRD method proposed by Chen et al., reference "Chen Y, Pock T. Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 39(6): 1256-72."

[0037] Method 3: CBDNet method proposed by Guo et al., reference "Guo S, Yan Z, Zhang K, et al. Toward convolutional blind denoising of real photographs [C]. IEEE Conference on Computer Vision and Pattern Recognition, 2019: 1712-22."

[0038] Method 4: C2N method proposed by Jang et al., reference "Jang G, Lee W, Son S, et al. C2N: Practical generative noise modeling for real-world denoising [C]. IEEE International Conference on Computer Vision, 2021: 2350-9."

[0039] Method 5: CVF-SID method proposed by Neshatavar et al., reference literature "Neshatavar R, Yavartanoo M, Son S, et al. CVF-SID: Cyclic multi-variate function for self-supervised image denoising by disentangling noise from image [C]. IEEE Conference on Computer Vision and Pattern Recognition, 2022: 17583-91."

[0040] Method 6: AP-BSN method proposed by Lee et al., reference literature "Lee W, Son S, Lee K M. AP-BSN: Self-supervised denoising for real-world images via asymmetric pd and blind-spot network [C]. IEEE Conference on Computer Vision and Pattern Recognition, 2022: 17725-34."

[0041] The content of the comparative experiment is as follows:

[0042] Experiment 1, the SIDD validation set and SIDD Benchmark in the SIDD dataset are used as the comparison dataset for the validation dataset, and methods 1 to 6 and the present application are used for denoising processing. Among them, the subjective evaluation standard is visual effect, which can be seen from Figure 4 . The objective evaluation parameters are peak signal to noise ratio (PSNR, unit dB) and structural similarity (SSIM), and the higher the values of the two parameters, the better the denoising effect of the image.

[0043] Table 1 Comparison of objective indicators of different methods on SIDD dataset

[0044]

[0045] Experiment 2, the DND Benchmark in the DND dataset is used as the comparison dataset for the validation dataset, and methods 1 to 6 and the present application are used for denoising processing. Among them, the subjective evaluation standard is visual effect, which can be seen fromFigure 5 The objective evaluation parameters are PSNR and SSIM, and the higher the values of the two parameters, the better the denoising effect of the image.

[0046] Table 2 Comparison of objective indicators of different methods on DND dataset

[0047]

[0048] Figure 4 and Figure 5 The denoising results of different methods in the SIDD and DND datasets are respectively shown. It can be obviously observed that the present application can more accurately remove noise and maintain good image texture structure in these pictures.

[0049] In summary, compared with the comparative methods, the denoising results of the present application have great advantages in subjective and objective evaluation. Therefore, the present application is an effective self-supervised real-world image denoising method.

Claims

1. A self-supervised real image denoising method based on sampling difference as perturbation, characterized in that The method comprises the following steps: Step 1: a random subsample generation strategy is proposed and used to obtain more subsamples; Step 2: a sampling difference loss function is proposed according to the random subsample generation strategy; Step 3: a cycle strategy is introduced into the construction of the loss function, and a cycle sampling difference loss function is proposed; Step four: training the denoising network through the above steps to obtain a denoising model, and then inputting the image with noise into the trained network model to obtain a high-quality denoised image; the random sub-sample generation strategy proposed in step one is denoted as Random Sub-samples Generation, RSG for short, and the RSG strategy generates s random and different sub-samples each time through the processes of segmentation, shuffling, recombination and restoration, wherein the segmentation is to divide the image into s×s non-overlapping blocks with a size of H×W / s 2 , the shuffling is to randomly shuffle the elements in each vector, the recombination is to take the elements at the same position of each vector to form a new vector with a size of H×W / s 2 , and the restoration is to change the new vector into s 2 sub-samples, wherein s is a down-sampling step factor, H is the length of the image, and W is the width of the image. 2 2 ​​ The sampling difference loss function proposed in step two regards the difference between RSG i (y) and RSG i+1 (y) as a disturbance, and regards them as the input and target of the blind spot network respectively; a new sampling difference loss function L SDBSN = E y {||B(RSG i (y))-RSG i+1 (y)||1} is constructed, where RSG i (y) represents the i-th sub-sample generated by the RSG policy; the cycle policy proposed in step three uses all s 2 sub-samples generated by the RSG policy for cycle training, and a new cycle difference loss function

Images