A deep-enhanced image denoising method based on non-local attention

Abstract

The present invention discloses a deep enhanced image denoising method based on non-local attention, comprising: inputting images in a data set into a pre-built network framework, dividing the clean image into blocks to obtain a preprocessed image; adding additive noise to the preprocessed image to obtain a noisy image; inputting the noisy image into a non-local attention module to obtain a feature map with non-local information of the image; performing a convolution operation on the feature map with non-local information of the image to obtain an abstract feature map; performing a convolution operation on the abstract feature map to perform image reconstruction to obtain a reconstructed image; inputting the reconstructed image into the non-local attention module to obtain a denoised image; calculating the loss between the denoised image and the clean image, and reducing the loss through forward and backward propagation of the network until the number of iterations reaches a threshold or the learning rate decays to a lower limit, thereby obtaining a denoising model.

Description

Technical Field

[0001] The present invention relates to a depth enhancement image denoising method based on non-local attention, belonging to the technical field of image denoising. Background Art

[0002] Image denoising is a classic computer vision task that aims to restore high-quality images from various degraded images. It has been widely used in many practical applications, such as medical image processing, surveillance, synthetic aperture radar (SAR) image processing, and image compression. Traditional methods solve the image denoising problem by building handcrafted models based on specific degradation prior knowledge. However, such methods, including block matching and 3D filtering (BM3D) [1], non-local means (NLM) [2], and sparse coding [3], usually have limited robustness to real data.

[0003] In recent years, deep convolutional neural networks have achieved great success in the field of image denoising. Compared with traditional denoising methods, deeper network structures can better capture image features and have powerful representation capabilities. In addition, with the continuous improvement of hardware performance, support has been provided for the calculation of deep network models, enabling deep convolutional networks to be calculated in parallel on powerful GPUs, improving the performance of the entire network during runtime. Therefore, it is feasible and promising to apply deep convolutional network models to image denoising tasks. Typical deep convolutional network models include SRCNN[4], DnCNN[5], and RDN[6].

[0004] The main defects of current image denoising are: (1) Since the image acquisition process is inevitably interfered by various noises, the denoising effect of general neural network denoising methods is poor; (2) Most neural network denoising methods fail to utilize the non-local information of the image during the denoising process, which limits the effect of image denoising.

[0005] [1] K.Dabov, A.Foi, V.Katkovnik, and K.Egiazarian, “Image denoising by sparse 3D transform-domain collaborative filtering,” IEEE Trans. Image Process., vol. 16, no. 8, pp. 2080–2095, Aug. 2007.

[0006] [2] A. Buades, B. Coll, and J. M. Morel, “A non-local algorithm for image denoising,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Dec. 2005, pp. 60–65.

[0007] [3] M.Elad and M.Aharon, “Image denoising via sparse and redundant representations over learned dictionaries,” IEEE Trans. Image Process., vol. 15, no. 12, pp. 3736–3745, Dec. 2006.

[0008] [4] Y.Gao, H.Li, J.Dong, and G.Feng, “A deep convolutional network formedical image super-resolution,” in Proc. Chin. Autom. Congr. (CAC), Oct. 2017, pp. 184–199.

[0009] [5] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a Gaussiandenoiser: Residual learning of deep CNN for image denoising,” IEEE Trans. Image Process., vol. 26, no. 7, pp. 3142–3155, Jul. 2017.

[0010] [6] Y. Zhang, Y. Tian, ​​Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image restoration,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 7, pp. 2480–2495, Jul. 2021. Summary of the Invention

[0011] The purpose of the present invention is to provide a depth enhancement image denoising method based on non-local attention to solve the defects of the prior art.

[0012] A deep enhancement image denoising method based on non-local attention, comprising:

[0013] Input the images in the dataset into the pre-built network framework, divide the clean images into blocks, and obtain pre-processed images;

[0014] Adding additive noise to the preprocessed image to obtain a noisy image;

[0015] Input the noisy image into the non-local attention module to obtain a feature map with non-local information of the image;

[0016] Perform convolution operation on the feature map with non-local information of the image to obtain an abstract feature map;

[0017] Perform convolution operation on the abstract feature map to reconstruct the image and obtain the reconstructed image;

[0018] The reconstructed image is input into the non-local attention module to obtain the denoised image;

[0019] The loss between the denoised image and the clean image is calculated, and the loss is reduced through forward and backward propagation of the network until the number of iterations reaches a threshold or the learning rate decays to a lower limit, and the denoising model is obtained.

[0020] Furthermore, the parameters of the network framework are set as follows: the number of forward and backward propagation of training data num_epoch = 800, batch size batch_size = 64, learning rate base_lr at the beginning of training = 0.001, learning rate decay weight power = 1.5, and learning rate end_lr at the end of training = 0.0001.

[0021] Furthermore, the loss function used by the denoising model is as follows:

[0022]

[0023] At this point, the noisy image used for testing is input into the denoising model to obtain a denoised image.

[0024] Furthermore, the method for obtaining a feature map with non-local information of an image includes:

[0025] First, perform two convolutions on the noisy image with a kernel size of 1*1 to obtain the feature map parameter features. Then, dot product the two feature parameters of the two convolution kernels. The similarity feature matrix is ​​processed by softmax to obtain the similarity weight. The autocorrelation in the feature map is calculated to obtain the similarity weight matrix between each position in the feature map and all other positions.

[0026] Perform another convolution on the input with a convolution kernel size of 1*1 to obtain parameter features, perform dot product on the convolution result and the similarity weight matrix, and finally perform a convolution on the result obtained by the dot product with a convolution kernel size of 1*1 and use the result of this convolution as the final output feature map.

[0027] Furthermore, the network framework includes a non-local attention module, a DDF-S module and a DDF module;

[0028] The non-local attention module is used to add a non-local attention module to the denoising network by finding similarities between different positions in the feature map;

[0029] The DDF-S module has the same structure as the DDF module and is used for feature extraction.

[0030] Furthermore, the image preprocessing includes: taking a group of clean images from the data set as a group of data, dividing the clean images y in the group of data into blocks, and the size of the images after the blocks is 50*50 to obtain preprocessed images.

[0031] Compared with the existing technology, the beneficial effects achieved by the present invention are as follows: the present invention mainly solves the problem of image denoising. By adding a non-local attention mechanism to the neural network, a new DDFN-N image denoising method is established to achieve the purpose of image denoising. Compared with the current image denoising algorithm, the DDFN-N image denoising algorithm can effectively utilize the non-local information of the image by adding a non-local attention mechanism; in the real world, almost all images have non-local information, so the algorithm is suitable for denoising of various images, and has better denoising effect on images rich in non-local information. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is the image denoising flow chart of the present invention;

[0033] Figure 2 It is a framework diagram of the entire network of the present invention;

[0034] Figure 3 It is a structural diagram of the DDF module of the present invention;

[0035] Figure 4 This is a structural diagram of the DDF-S module of the present invention;

[0036] Figure 5 This is the structural diagram of the non-local attention module of the present invention. DETAILED DESCRIPTION

[0037] In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the present invention is further described below in conjunction with specific implementation methods.

[0038] This paper aims to address the poor denoising performance of current image denoising models. Furthermore, when learning denoising models using neural networks, the non-local information of the image is not utilized, resulting in the denoising model being unable to learn the underlying information of the image itself. This paper proposes a deep enhancement denoising model based on non-local attention. This model not only effectively restores damaged images and suppresses noise, but also introduces a non-local attention mechanism that captures long-term dependencies between pixels in the input feature map, expanding the receptive field and thus improving the image denoising effect, making the image easier to discern. The specific steps are as follows:

[0039] Step 1: Build the network framework of the neural network denoising model and set the parameters in the network reasonably. The parameters set are: the number of forward propagation and backward propagation of all training data num_epoch = 800, batch size batch_size = 64, the learning rate base_lr at the beginning of training = 0.001, the weight of learning rate decay power = 1.5, and the learning rate end_lr at the end of training = 0.0001;

[0040] Step 2: Take 64 clean images from the dataset as a batch of data, divide the clean images y in the batch into blocks, and the image size after block division is 50*50 to obtain the preprocessed image;

[0041] Step 3: Add additive Gaussian noise that conforms to the normal distribution to the preprocessed image to obtain the noisy image z j (j=1,2…m);

[0042] Step 4: Convert the noisy image z j (j=1,2…m) is input to the non-local attention module to obtain the feature map x with non-local information of the image i(i=1,2…n); Specifically, first perform two convolutions with a kernel size of 1*1 on the input to obtain the feature map parameter features, then dot product the first two feature parameters, and softmax the similarity feature matrix to obtain the similarity weight. Through this process, the autocorrelation in the feature map is calculated, and the similarity weight matrix between each position in the feature map and all other positions is obtained. Then, perform another convolution with a kernel size of 1*1 on the input to obtain the parameter features, dot product the convolution result with the similarity weight matrix, and finally perform a convolution with a kernel size of 1*1 on the result of the dot product and use the result of this convolution as the final output feature map;

[0043] Step 5: Perform a series of convolution operations on the feature map with non-local information of the image to further obtain a more abstract feature map Specifically, the feature maps are trained on two paths. One path first performs a general convolution with a kernel size of 3*3, and then performs an expanded convolution with a kernel size of 3*3. The other path does the opposite. This allows the network to learn richer features. Finally, we concatenate the feature maps obtained from the two paths as the output. In order to prevent gradient explosion caused by the deepening of the network, we concatenate the input feature map and the output as the output of a module. This can also reduce the loss of image texture details.

[0044] Step 6: Reconstruct the image by performing a convolution operation with a convolution kernel size of 1*1 on the abstract feature map to obtain the reconstructed image

[0045] Step 7: Reconstructed image The image is input into the non-local attention module to enhance the non-local information of the image. After the non-local attention module, the denoised image y' is obtained.

[0046] Step 8: Calculate the loss between the denoised image y' and the clean image, and reduce the loss through forward and backward propagation of the network until the number of training times for all training data reaches 800 or the learning rate decays to 0.0001, and the denoising model is obtained. The loss function used by the model is as follows:

[0047]

[0048] At this point, the noisy image used for testing is input into the denoising model to obtain a denoised image.

[0049] In this embodiment, the network framework includes a non-local attention module, a DDF-S module and a DDF module;

[0050] The non-local attention module is used to add a non-local attention module to the denoising network by finding similarities between different positions in the feature map;

[0051] The DDF-S module has the same structure as the DDF module and is used for feature extraction;

[0052] DDF module: adopts the design idea of ​​extended path fusion. In order to realize this idea, we use convolution operation with a convolution kernel size of 1*1 to aggregate the features of the leading layer. The feature maps are trained on two paths respectively. One path is to first perform general convolution with a convolution kernel size of 3*3, and then perform dilated convolution with a convolution kernel size of 3*3. Dilated convolution increases the receptive field by inserting zero values ​​on the original convolution kernel. The other path is to perform dilated convolution first and then general convolution. This allows the network to learn richer features. Finally, we splice the feature maps obtained on the two paths as output, and in order to prevent the gradient explosion caused by the deepening of the network, we splice the input feature map and the output together as the output of a module. This can also reduce the loss of image texture details.

[0053] DDF-S: Consistent with the DDF structure, since the DDF-S module is before the DDF module, the DDF-S module does not need to use a convolution operation with a convolution kernel size of 1*1 to aggregate the leading layer features, and serves as feature extraction.

[0054] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A deep enhancement image denoising method based on non-local attention, characterized in that include: Input the images in the dataset into the network framework of the pre-built neural network denoising model, divide the clean images into blocks, and obtain preprocessed images; Adding additive noise to the preprocessed image to obtain a noisy image; Input the noisy image into a non-local attention module to obtain a feature map with non-local information of the image. The non-local attention module specifically includes: performing convolution with a convolution kernel size of 1*1 on the input to obtain two feature map parameter features, performing dot product on the two feature map parameter features to obtain a similarity feature matrix, and performing softmax processing on the similarity feature matrix to obtain a similarity weight matrix; Perform another convolution with a kernel size of 1*1 on the input to obtain a third feature map parameter feature, perform a dot product operation on the third feature map parameter feature and the similarity weight matrix, perform another convolution with a kernel size of 1*1 on the result of the dot product operation, and use the convolution result as the final output feature map; Performing a convolution operation on the feature map with non-local information of the image to obtain an abstract feature map, specifically: training the feature map with non-local information of the image on two paths respectively, one path is first performing a general convolution with a convolution kernel size of 3*3, and then performing a dilated convolution with a convolution kernel size of 3*3, and the other path is first performing a dilated convolution with a convolution kernel size of 3*3, and then performing a general convolution with a convolution kernel size of 3*3, splicing the feature maps obtained on the two paths as output, and finally splicing the feature map with non-local information of the image and the output as the abstract feature map; Perform a convolution operation with a convolution kernel size of 1*1 on the abstract feature map to reconstruct the image and obtain the reconstructed image; Input the reconstructed image into the non-local attention module to obtain the image after denoising in this round of iteration; Calculate the loss between the denoised image and the clean image, and reduce the loss through forward and backward propagation of the neural network until the number of iterations reaches a threshold or the learning rate decays to a lower limit, and then obtain a trained neural network denoising model; The noisy image used for testing is input into the trained neural network denoising model to obtain a denoised image.

2. The method for depth-enhanced image denoising based on non-local attention according to claim 1, wherein: The parameters of the network framework of the neural network denoising model are set as follows: the number of forward and backward propagation of training data num_epoch=800, batch size batch_size=64, learning rate base_lr=0.001 at the beginning of training, learning rate decay weight power=1.5, and learning rate end_lr=0.0001 at the end of training.

3. The method for depth-enhanced image denoising based on non-local attention according to claim 1, wherein: The loss function used by the neural network denoising model is as follows: 。 4. The method for depth-enhanced image denoising based on non-local attention according to claim 1, wherein: Image preprocessing includes: taking a set of clean images from the data set as a set of data, dividing the set of clean images into blocks, and the image size after block division is 50*50 to obtain preprocessed images.

Images