Self-supervised real image denoising method based on sampling difference as disturbance

Abstract

The invention discloses a self-supervised real image denoising method based on sampling difference as disturbance. The method mainly comprises the following steps: proposing and using a random sub-sample generation strategy to obtain more sub-samples; providing a sampling difference loss function according to a random subsample generation strategy; introducing a loop strategy into construction of a loss function, and proposing a loop sampling difference loss function; the denoising network is trained through the above steps to obtain the denoising model, then the image with noise is used as input, and a high-quality denoised image is obtained through the trained network model. The method does not need clean images for training, is superior to an existing pseudo-supervision / self-supervision method, and is an effective self-supervision real image denoising method.

Description

Technical Field

[0001] The invention relates to an image denoising technology, and in particular to a self-supervised real image denoising method based on sampling difference as disturbance, and belongs to the field of digital image processing. Background Art

[0002] An image is a digital form that records and represents information such as the shape, color, and texture of an object. It is intuitive and vivid, and is indispensable in human production activities. With the development of science and technology and the improvement of people's quality of life, the requirements for image quality have become more and more stringent. In the field of medical imaging, high-quality images can provide accurate information required for diagnosis and treatment; in the field of security monitoring, high-quality images help identify and track suspects; in the field of graphics and design, high-quality images can create high-precision graphics and images; in the field of media and entertainment, high-quality images can provide a better viewing experience. Therefore, the quality of the image has become a key factor in achieving effective results in various application fields. However, due to defects in imaging systems, transmission media, and recording devices, images are inevitably contaminated by noise. Noise will have a negative impact on the quality of the image, reduce the readability and information content of the image, so it is necessary to reduce or remove image noise through technical means to improve image quality. Therefore, image denoising has become a research hotspot in image processing.

[0003] Image denoising aims to suppress and remove noise in images by using mathematical models and algorithms, and finally restore high-quality images from noisy images. After years of research and development, image denoising can be divided into filtering-based methods, model-based methods, and learning-based methods. At present, deep learning-based methods have become the mainstream in the field of image denoising due to their excellent performance. However, the existing real image denoising methods based on deep learning still have room for further improvement in terms of denoising quality and the construction of network model structures. Summary of the invention

[0004] The purpose of the present invention is to solve the problem of poor network performance caused by insufficient training sets in unsupervised algorithms, thereby constructing an effective self-supervised real image denoising method.

[0005] The self-supervised real image denoising method based on sampling difference as perturbation (SDAP) proposed in the present invention mainly includes the following steps:

[0006] (1) Propose and use random subsample generation strategies to obtain more subsamples;

[0007] (2) A sampling difference loss function is proposed based on the random sub-sample generation strategy;

[0008] (3) The cyclic strategy is introduced into the construction of the loss function, and the cyclic sampling difference loss function is proposed;

[0009] (4) The denoising network is trained through the above steps to obtain a denoising model, and then the noisy image is taken as input to obtain a high-quality denoised image through the trained network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a principle block diagram of the self-supervised real image denoising method based on sampling difference as disturbance of the present invention. Among them, (a) is the training structure block diagram of SDAP of the present invention, and (b) is the test and effect increase structure block diagram of SDAP.

[0011] Figure 2 This is the result of adding disturbances to the training data and training the blind spot network.

[0012] Figure 3 It is an example diagram of the random sub-sample generation strategy proposed in the present invention.

[0013] Figure 4 The figure is a comparison of the denoising results of the "SIDD" dataset by the present invention and six other methods. Among them, (a) is a noisy image, (b) to (g) are the denoising results of methods 1 to 6, and (h) is the denoising result of the present invention.

[0014] Figure 5 This is a comparison chart of the denoising results of the "DND" dataset by the present invention and six other methods. Among them, (a) is a noisy image, (b) to (g) are the denoising results of methods 1 to 6, and (h) is the denoising result of the present invention. DETAILED DESCRIPTION

[0015] The present invention will be further described below in conjunction with the accompanying drawings:

[0016] Figure 1 In the paper, the self-supervised real image denoising method based on sampling difference as perturbation can be divided into the following steps:

[0017] (1) Propose and use the Random Sub-samples Generation (RSG) strategy to obtain more sub-samples;

[0018] (2) A sampling difference loss function is proposed based on the random sub-sample generation strategy;

[0019] (3) The cyclic strategy is introduced into the construction of the loss function, and the cyclic sampling difference loss function is proposed;

[0020] (4) The denoising network is trained through the above steps, and the noisy image is used as input. A high-quality denoised image can be obtained through the trained network model;

[0021] Specifically, in step (1), in order to solve the problem of lack of training data in self-supervision, the present invention adds disturbances (mean 0, variance σ 2 Gaussian distribution, where σ is the standard deviation), generate different images, and obtain more training data to train the blind-spot network (BSN) Figure 2 The result proves that adding disturbance can improve the performance of BSN. Therefore, the present invention regards the sampling difference between sub-images obtained by different sampling methods as a disturbance; inspired by this idea, the present invention further adopts the RSG strategy to generate s 2 subsamples, s is the step size factor. The detailed steps of the RSG strategy adopted by the present invention are as follows: Figure 3 Assume that the length and width of the noise image y are H and W respectively, the detailed implementation process of the RSG strategy is as follows:

[0022] 1) Segmentation: The image y is divided into H×W / s 2 Then, these blocks are stretched into s×s non-overlapping blocks. 2 For better visualization, Figure 3 The case of s=2 is shown in FIG. 1 , and s is set to 5 during actual training in the present invention.

[0023] 2) Shuffle: Randomly shuffle the elements in each vector. Since this process is strictly random, the subsamples obtained each time are different.

[0024] 3) Reorganization: Take the elements in the same position of each vector to form s 2 The length is H×W / s 2 The new vector.

[0025] 4) Restoration: Reshape each vector into a subsample of size [W / s]×[H / s]. In this way, the present invention obtains s 2 Similar subsamples

[0026] In step (2), the present invention can obtain s using a noisy image through the RSG strategy. 2 sub-samples available for training, the present invention uses RSG i (y) and RSG i+1 The difference between (y) is regarded as perturbation, and they are regarded as input and target of BSN respectively; a new sampling difference loss function L is constructedSDBSN :

[0027] L SDBSN =E y {||B(RSG i (y)-RSG i+1 (y)||1}

[0028] RSG i (y) represents the i-th subsample generated by RSG.

[0029] In step (3), the present invention introduces a loop strategy in the training phase. Unlike the conventional method of training BSN, which only requires two subsamples for each training of each image, the present invention uses all the generated s 2 Therefore, the present invention proposes a new cycle difference loss function L CSDBSN :

[0030]

[0031]

[0032] Since the subsamples obtained by the RSG strategy are randomly generated, the training data is also random. CSDBSN The target corresponding to each input in is different, thus avoiding the formation of a fixed mapping between two sub-samples during the cycle. This cycle loss has the following advantages: 1. It imposes constraints on all pixels of the original noisy image; 2. It ensures that all sub-samples generated by RSG can be fully utilized; 3. It makes the training of BSN more robust. Figure 1 (a) also intuitively illustrates the training method of the present invention, which includes the RSG strategy and the cyclic difference loss function.

[0033] In step (4), the present invention uses the random subsamples generated by the RSG strategy to avoid the denoising network from forming a fixed mapping, but its randomness will inevitably lead to a decrease in the performance of the denoising network during the test phase. Since each point of the pixel-shuffle downsampling (PD) subsample has a fixed position in the original image, it has a fixed offset direction, and its randomness is smaller than that of the RSG strategy proposed by the present invention. Therefore, in step (4), the present invention uses the PD sampling mode to stabilize the test results during the test phase. The specific method is as follows Figure 1 As shown in (b), the present invention introduces the PD strategy in the testing phase, and the present invention denoises the denoised image again through BSN to remove the non-smooth artifacts and make the image more natural.

[0034] In order to better illustrate the effectiveness of the present invention, a comparison is made on the commonly used real noise image datasets "SIDD" and "DND". The denoising algorithms compared are:

[0035] Method 1: BM3D method proposed by Dabov et al., reference “Dabov K, Foi A, Katkovnik V, et al. Color image denoising via sparse 3D collaborative filtering with grouping constraint in luminance-chrominance space[C]. IEEE International Conference on Image Processing, 2007, 1: I-313-I-6.”

[0036] Method 2: TNRD method proposed by Chen et al., reference “Chen Y, Pock T. Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 39 (6): 1256-72.”.

[0037] Method 3: CBDNet method proposed by Guo et al., reference "Guo S, Yan Z, Zhang K, et al. Toward convolutional blind denoising of real photographs [C]. IEEE Conference on Computer Vision and Pattern Recognition, 2019: 1712-22.".

[0038] Method 4: C2N method proposed by Jang et al., reference “Jang G, Lee W, Son S, et al. C2N: Practical generative noise modeling for real-world denoising [C]. IEEE International Conference on Computer Vision, 2021: 2350-9.”

[0039] Method 5: CVF-SID method proposed by Neshatavar et al., reference "Neshatavar R, Yavartanoo M, Son S, et al. CVF-SID: Cyclic multi-variate function for self-supervised image denoising by disentangling noise from image[C]. IEEE Conference on Computer Vision and Pattern Recognition, 2022: 17583-91.".

[0040] Method 6: AP-BSN method proposed by Lee et al., reference "Lee W, Son S, Lee K M. AP-BSN: Self-supervised denoising for real-world images via asymmetric pd and blind-spot network[C]. IEEE Conference on Computer Vision and Pattern Recognition, 2022: 17725-34.".

[0041] The contents of the comparative experiment are as follows:

[0042] In Experiment 1, the validation data set uses the SIDD validation set and SIDD Benchmark in the SIDD data set as the comparison data set, and uses methods 1 to 6 and the present invention to perform denoising respectively. The subjective evaluation criterion is the visual effect, which can be obtained from Figure 4 The objective evaluation parameters are Peak Signal to Noise Ratio (PSNR, unit: dB) and Structure Similarity Index (SSIM). The higher the values ​​of the two parameters, the better the denoising effect of the image.

[0043] Table 1 Comparison of objective indicators of different methods on SIDD dataset

[0044]

[0045] In Experiment 2, the validation dataset uses the DND Benchmark in the DND dataset as a comparison dataset, and uses methods 1 to 6 and the present invention to perform denoising. The subjective evaluation criterion is the visual effect, which can be obtained from Figure 5 The objective evaluation parameters are PSNR and SSIM. The higher the values ​​of the two parameters, the better the denoising effect of the image.

[0046] Table 2 Comparison of objective indicators of different methods on the DND dataset

[0047]

[0048] Figure 4 and Figure 5 The denoising results of different methods in SIDD and DND datasets are shown respectively. It can be clearly observed that the present invention can more accurately remove noise and maintain good image texture structure in these images.

[0049] In summary, compared with the comparative method, the denoising results of the present invention have great advantages in both subjective and objective evaluation. Therefore, the present invention is an effective self-supervised real-world image denoising method.

Claims

1. A self-supervised real image denoising method based on sampling difference as perturbation, characterized by The following steps are involved: Step 1: Propose and use a random subsample generation strategy to obtain more subsamples; Step 2: Propose a sampling difference loss function based on the random sub-sample generation strategy; Step 3: Introduce the loop strategy into the construction of the loss function and propose a loop sampling difference loss function; Step 4: Train the denoising network through the above steps to obtain a denoising model, and then use the noisy image as input to obtain a high-quality denoised image through the trained network model.

2. The self-supervised real image denoising method based on sampling difference as perturbation according to claim 1, characterized in that The random sub-sample generation strategy proposed in step 1, where the English expression of random sub-sample generation is Random Sub-samples Generation, referred to as RSG, can generate more random and different s each time through the process of segmentation, scrambling, reorganization, and recovery. 2 sub-samples, s is the step size factor of downsampling; since adding disturbance to the data set can improve the denoising performance of the self-supervised network, through this method, the present invention can achieve the goal of augmenting the training data set, thereby alleviating the problem of scarce training data and improving the denoising ability of the self-supervised network.

3. The self-supervised real image denoising method based on sampling difference as perturbation according to claim 1, characterized in that The sampling difference loss function proposed in step 2 is obtained by using RSG i (y) and RSG i+1 The difference between (y) is regarded as a perturbation, and they are regarded as the input and target of the blind network respectively; a new sampling difference loss function L is constructed SDBSN =E y {||B(RSG i (y)-RSG i+1 (y)||1}, where RSG i (y) represents the i-th subsample generated by the RSG strategy.

4. The self-supervised real image denoising method based on sampling difference as perturbation according to claim 1, characterized in that The loop strategy proposed in step 3, this network is different from the conventional method of training blind spot networks in the model. Each training of each image only requires two subsamples. The present invention will use all s generated by the RSG strategy. 2 Sub-samples are trained cyclically, and a new cyclic difference loss function is proposed in The network trained using this cycle strategy has better robustness and denoising performance than other self-supervised networks.

Images