A double residual denoising method based on attention allocation mechanism

Abstract

The present invention discloses a dual residual denoising method based on an attention allocation mechanism. The steps of the method include constructing a training data set, performing a preprocessing operation on the training data set; constructing a network denoising model using an attention allocation mechanism and a convolutional neural network with a dual residual network structure; setting hyperparameters and a loss function of the network denoising model, and optimizing the loss function; adding noise of different levels to the training data set and training to obtain a trained network model; performing image denoising using the trained network model, and evaluating the noisy image using structural similarity and peak signal-to-noise ratio indicators, which has the effect of improving noise reduction performance and imaging quality.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a double residual denoising method based on an attention allocation mechanism. Background Art

[0002] Images are stored in computers as digital data. However, due to the influence of environment, equipment, human factors, etc. during the acquisition and transmission process, noise is inevitably introduced, which affects the readability of the image and causes the image quality to deteriorate. This phenomenon of image is called degradation. In order to solve this degradation phenomenon, image denoising technology is proposed. Denoising is one of the very important methods to improve image quality. At the same time, denoising facilitates subsequent processing of image classification, segmentation and recognition tasks.

[0003] Image denoising is a classic problem in the field of image processing and an important step in computer vision preprocessing. Although traditional image denoising algorithms can achieve good denoising effects, they still have two disadvantages: 1. These methods usually involve complex optimization problems and have slow calculation times. 2. Most models are non-convex, and the optimization process requires manual parameter setting, so it is difficult to efficiently obtain good image restoration effects.

[0004] In recent years, image denoising methods based on deep learning have been well developed in academic research. At the same time, the attention mechanism in deep learning has been continuously improved. The image denoising network combined with the attention mechanism still has room for improvement, including: the existing denoising network is mainly implemented through local convolution, with a small receptive field; there is no excessive attention to the detailed texture part, and the denoised image is prone to edge smoothing.

[0005] The publication number in the prior art is: CN113610719A, and the name is: A method for image denoising using an attention and densely connected residual block convolution kernel neural network. The disclosed network denoising model includes an input layer, a hidden layer, a convolution layer and an output layer. However, firstly, the image augmentation method is relatively simple during the training process, and secondly, the attention mechanism is globally consistent, and the problem of attention allocation is not considered. Summary of the invention

[0006] In order to solve the above problems, the present invention provides a double residual denoising method based on attention allocation mechanism with good denoising effect and high image quality.

[0007] In order to achieve the above object, the present invention is implemented by the following technical solutions:

[0008] The present invention is a double residual denoising method based on an attention allocation mechanism, comprising the following steps:

[0009] Step 1, preprocessing the input training image to obtain a noise image containing noise after preprocessing;

[0010] Step 2: Use D-Block with attention allocation module to build a denoising network model;

[0011] Step 3, using the preprocessed noisy image to train the denoising network model, and using the loss function to constrain it, to obtain the image denoising network model;

[0012] Step 4, adding different levels of noise to the test set, and using the obtained image denoising network model to denoise the test set to obtain a denoised result image;

[0013] Step 5: Use structural similarity and peak signal-to-noise ratio indicators to evaluate the denoised result image.

[0014] A further improvement of the present invention is that the preprocessing operation of the training data set in step 1 is specifically as follows:

[0015] Step 1.1, selecting training samples from the training data set as the original training set, wherein the images in the training samples are all noise-free images of the same size;

[0016] Step 1.2, scale the images in the training data set by 1, 0.9, 0.8, and 0.7 times, respectively, and use a sliding window to extract image blocks from each scaled image;

[0017] Step 1.3, augment the extracted image block;

[0018] A further improvement of the present invention is that the denoising network model constructed in step 2 includes a convolution operation, a ReLU activation function and 6 D-Blocks.

[0019] The further improvement of the present invention is that: the D-Block components include the residual units at the front end and the back end, the trunk branch, the mask branch, and the attention allocation module;

[0020] The residual units at the front end and the back end are both residual modules without the batch normalization layer. The module consists of two convolutional layers and two layers of ReLU activation functions.

[0021] The main branch includes a pair of convolution kernels Conv-A and Conv-B of different sizes, and a set of residual connections are added between the convolution kernels Conv-A and Conv-B;

[0022] The mask branch includes 2 Squeeze-Excitation-Blocks, 1 Squeeze-Excitation-Block residual unit, 1 convolution layer and 1 Sigmoid activation function. The Squeeze-Excitation-Block residual unit includes a downsampling layer, a Squeeze-Excitation-Block, an upsampling layer and a set of residual connections. The downsampling layer and the upsampling layer are implemented using large-step convolution kernels and deconvolution operations respectively. Finally, the Sigmoid activation function generates weights and performs feature weighting on the trunk branch.

[0023] The attention allocation module includes a spatial attention module and a channel attention module. The spatial attention module includes two pooling layers and a spatial main line. The two pooling layers are the maximum pooling layer and the average pooling layer. The feature map is spliced ​​into a large feature map after the maximum pooling operation and the average pooling operation. It is first convolved by the convolution layer and then the Sigmoid activation function generates a spatial weight for it. The spatial main line is weighted with spatial features and input into the channel attention module. The channel attention module includes an average pooling layer, a multi-layer perceptron and a Softmax activation function.

[0024] A further improvement of the present invention is that the operation formula of the spatial attention module in the denoising network model constructed in step 2 is as follows:

[0025] F sa (x) = σ(f i *Cat(x avg ,x max ))

[0026] Where σ is the sigmoid activation function, f i is a convolution operation with a convolution kernel size of i. In this embodiment, i is 7. Cat is a concatenation operation. x avg and x max They are the feature maps after average pooling and the feature maps after maximum pooling respectively;

[0027] The calculation formula of the inter-channel attention module is as follows:

[0028] F ca (x) = δ[W2(W1x avg +b1)+b2]

[0029] Where W1 and W2 are the weights of the multilayer perceptron, b1 and b2 are the biases of the multilayer perceptron;

[0030] The weights generated by the attention allocation module are multiplied with the attention branch and the non-attention branch in the network, expressed as:

[0031]

[0032] Where, X i The attention allocation process, F i att and F i non Represent the attention branch and the non-attention branch respectively. and They represent the dynamic weights generated by the attention allocation module.

[0033] A further improvement of the present invention is that the hyperparameters of the denoising network model in step 3 include batch size, initial learning rate, number of iterations, and learning rate decay strategy, and the loss function is mean square error, which is expressed as follows:

[0034]

[0035] Where: Θ network model parameters, R(y i ; Θ) is the residual image learned by the network, y i is the noise image, x i represents a clean image, and N is the number of training samples.

[0036] A further improvement of the present invention is that the specific operation steps of step 4 are as follows:

[0037] Step 4.1: Add Gaussian white noise with noise levels of 15, 25, and 50 to the images in the test set.

[0038] Get a noisy training image;

[0039] In step 4.2, the noisy training image is input into the denoising network model for training, thereby obtaining a trained denoising network model, and saving it every 20 epochs.

[0040] A further improvement of the present invention is that the test set in step 4 includes Set12 and BSD68 grayscale data sets.

[0041] The beneficial effects of the present invention are as follows: the present invention designs an attention allocation module, an attention branch, and a non-attention branch by considering the spatial attention weight and the inter-channel attention weight, introduces attention modules such as SE-Block in the attention branch, and allocates the attention weight and non-attention weight generated by the attention allocation module to the attention branch and the non-attention branch, respectively, so as to better guide the network to denoise and ensure the denoising quality of the image. By designing a dual residual structure for the non-attention branch, both global information and local information are taken into account, and this method significantly improves the denoising performance and imaging quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of the present invention.

[0043] Figure 2 It is a network denoising model.

[0044] Figure 3 This is a schematic diagram of the D-Block module in the network.

[0045] Figure 4 This is the Squeeze-Excitation-Block diagram.

[0046] Figure 5 It is a schematic diagram of denoising results.

[0047] Figure 6 It is a noise diagram. DETAILED DESCRIPTION

[0048] The following will be combined with the accompanying drawings in the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the implementation of the present invention, not all of the implementation. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] like Figure 1 As shown, the present invention is a denoising network model with a double residual structure based on an attention allocation mechanism, comprising the following steps:

[0050] Step 1: preprocess the input training image to obtain a noisy image containing noise after preprocessing; the specific steps of the preprocessing operation are as follows:

[0051] Step 1.1, selecting training samples from the training data set as the original training set, wherein the images in the training samples are all noise-free images of the same size; in this embodiment, the training samples are the training set Train400, which has a capacity of 400 images with a height of 180 pixels and a width of 180 pixels;

[0052] Step 1.2, scale the images in the training data set by 1, 0.9, 0.8, and 0.7 times, respectively, and use a sliding window to extract image blocks from each scaled image;

[0053] In step 1.3, the extracted image blocks are augmented, that is, the image is flipped upside down, rotated 90°, rotated 180°, rotated 270°, flipped upside down and then rotated 90°, flipped upside down and then rotated 180°, and flipped upside down and then rotated 270°.

[0054] Step 2: Use D-Block with attention allocation module to build a denoising network model;

[0055] Step 3: Set the batch size, initial learning rate, number of iterations, learning rate decay strategy and loss function of the denoising network model, and optimize the loss function; the loss function is the mean square error, and its mathematical expression is:

[0056]

[0057] Where: Θ is the network model parameter, R(y i ; Θ) is the residual image learned by the network, y i is the noise image, x i Represents a clean image, and N is the number of training samples. The present invention uses the Train400 dataset as the training dataset. In each epoch, 40 image blocks of size 128×128 are randomly extracted as samples, and the Adam optimizer is used for training. The initial learning rate is set to 1×10 -4 , the learning rate decreases by 0.2 times every 10 epochs, and the network is trained for a total of 100 epochs;

[0058] Step 4, add Gaussian white noise with noise levels of 15, 25, and 50 to the test data set, and use the obtained image denoising network model to denoise the test data set to obtain the denoised result image, and save it every 20 epochs; among them, the test set includes Set12 and BSD68 grayscale data sets.

[0059] In step 5, the noisy image is evaluated using the structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) indicators.

[0060] like Figure 2As shown in the figure, the denoising network model constructed in step 2 includes convolution operations, ReLU activation functions and 6 D-Blocks, where D-Block includes ResNet-Block, Squeeze-Excitation-Block, upsampling, downsampling, Sigmoid activation function, etc., and a double residual structure is used inside the D-Block, and it includes a pair of convolution layers with different convolution kernel sizes. Except for D-Block, the convolution kernel size used in the network model is 3*3.

[0061] like Figure 3 As shown, D-Block includes residual units at the front and back ends, trunk branches, mask branches, and attention allocation modules;

[0062] The residual units at the front end and back end are residual modules without the batch normalization layer, which include two convolutional layers with a convolution kernel size of 3*3 and two layers of ReLU activation functions.

[0063] The main branch includes a pair of convolution kernels Conv-A and Conv-B of different sizes. A set of residual connections is added between the convolution kernels Conv-A and Conv-B. The two residual connections in the main branches of the two middle D-Blocks are used to connect the previous and next D-Blocks respectively. The residual of the first D-Block comes from its own network branch. The double residual output of the last D-Block is not used, but the output of the main line of the network is used.

[0064] The mask branch includes 2 Squeeze-Excitation-Blocks, 1 Squeeze-Excitation-Block residual unit, 1 convolution layer and 1 Sigmoid activation function. The Squeeze-Excitation-Block residual unit includes a downsampling layer, a Squeeze-Excitation-Block, an upsampling layer and a set of residual connections. The downsampling layer and the upsampling layer are implemented using large-step convolution kernels and deconvolution operations respectively. Finally, the Sigmoid activation function generates weights and performs feature weighting on the trunk branch.

[0065] The attention allocation module includes a spatial attention module and a channel attention module. The spatial attention module includes two pooling layers and a spatial main line. The two pooling layers are the maximum pooling layer and the average pooling layer. The feature map is spliced ​​into a large feature map after the maximum pooling operation and the average pooling operation. After the convolution operation, the Sigmoid activation function generates a spatial weight for it, and the spatial main line is weighted with spatial features and input into the channel attention module. Spatial Attention Module F sa The calculation formula is as follows:

[0066] F sa (x) = σ(f i *Cat(x avg ,x max ))

[0067] Where σ is the sigmoid activation function, f i is a convolution operation with a convolution kernel size of i, Cat is a concatenation operation, x avg and x max They are the feature maps after average pooling and the feature maps after maximum pooling, respectively.

[0068] The channel attention module includes an average pooling layer, a multi-layer perceptron, and a Softmax activation function. The multi-layer perceptron includes two linear layers. Finally, it will output the attention weights for the trunk branch and the non-attention weights for a pair of residual connections in the trunk branch. ca The calculation formula is as follows

[0069] F ca (x) = δ[W2(W1x avg +b1)+b2]

[0070] Where δ is the softmax activation function, W1 and W2 are the weights of the multilayer perceptron, b1 and b2 are the biases of the multilayer perceptron, and x avg It is the feature map after average pooling.

[0071] The weights generated by the attention allocation module are multiplied with the attention branch and the non-attention branch in the network. The operation process can be expressed as:

[0072]

[0073] In the formula, F i att and F i non Represent the attention branch and the non-attention branch respectively. and They represent the dynamic weights generated by the attention allocation module, respectively. This step assigns different weights to the attention main line and non-attention branches.

[0074] The evaluation results of step 5 are shown in Tables 1 and 2;

[0075] Table 1 PSNR values ​​of test results

[0076]

[0077] Table 2 SSIM values ​​of test results

[0078]

[0079] like Figure 5 and Figure 6 As shown in the figure, this algorithm can restore texture details and preserve edge information. For example, in the picture "airplane", the text slogans on the fuselage can be restored well, and in the picture "starfish", the algorithm can restore the texture information of the starfish itself well, completing the image denoising while preserving the information intact.

[0080] The present invention takes into account the combination of the dual residual mechanism and the attention mechanism, and assigns different weights to the attention main line and the non-attention branch lines, which effectively improves the reuse efficiency of attention and indirectly improves the receptive field of the network, so that the network can better obtain contextual information.

[0081] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been discussed above with a preferred embodiment, it is not used to limit the present invention. Any technician familiar with the profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A double residual denoising method based on attention allocation mechanism, characterized by: The steps include: Step 1, preprocessing the input training image to obtain a noise image containing noise after preprocessing; Step 2: Use D-Block with attention allocation module to build a denoising network model; The denoising network model includes a convolution operation, a ReLU activation function and 6 D-Blocks; the D-Block components include residual units at the front end and the back end, a trunk branch, a mask branch, and an attention allocation module; The residual units at the front end and the back end are both residual modules without the batch normalization layer. The module consists of two convolutional layers and two layers of ReLU activation functions. The trunk branch includes a pair of convolution kernels Conv-A and Conv-B of different sizes, and a set of residual connections are added between the convolution kernel Conv-A and the convolution kernel Conv-B; The mask branch includes 2 Squeeze-Excitation-Blocks, 1 Squeeze-Excitation-Block residual unit, 1 convolution layer and 1 Sigmoid activation function, wherein the Squeeze-Excitation-Block residual unit includes a downsampling layer, a Squeeze-Excitation-Block, an upsampling layer and a group of residual connections, the downsampling layer and the upsampling layer are respectively implemented using a large-step convolution kernel and a deconvolution operation, and finally a weight is generated by the Sigmoid activation function, and feature weighting is performed on the trunk branch; The attention allocation module includes a spatial attention module and a channel attention module. The spatial attention module includes two pooling layers and a spatial main line. The two pooling layers are respectively a maximum pooling layer and an average pooling layer. The feature map is spliced ​​into a large feature map after the maximum pooling operation and the average pooling operation. The convolution layer first undergoes a convolution operation and then generates a spatial weight for it by a Sigmoid activation function. The spatial main line is weighted by spatial features and input into the channel attention module. The channel attention module includes an average pooling layer, a multi-layer perceptron and a Softmax activation function. The calculation formula of the spatial attention module is as follows: ； In the formula, is the sigmoid activation function, is a convolution operation with a convolution kernel size of i, Cat is a concatenation operation, and They are the feature maps after average pooling and the feature maps after maximum pooling respectively; The calculation formula of the inter-channel attention module is as follows: ； In the formula, and are the weights of the multi-layer perceptron, and are the biases of the multilayer perceptron respectively; The weights generated by the attention allocation module are multiplied with the attention branch and the non-attention branch in the network, expressed as: ； In the formula, The process of allocating attention, and Represent the attention branch and the non-attention branch respectively. and They represent the dynamic weights generated by the attention allocation module; Step 3, using the preprocessed noisy image to train the denoising network model, and using the loss function to constrain it, to obtain the image denoising network model; Step 4, adding different levels of noise to the images in the test set, and using the obtained image denoising network model to denoise the test set to obtain the denoised result image; Step 5: Use structural similarity and peak signal-to-noise ratio indicators to evaluate the denoised result image.

2. The double residual denoising method based on attention allocation mechanism according to claim 1, characterized in that: The preprocessing operations for the training data set in step 1 are as follows: Step 1.1, selecting training samples from the training data set as original training sets, wherein the images in the training samples are all noise-free images of the same size; Step 1.2, scale the images in the training data set by 1, 0.9, 0.8, and 0.7 times, respectively, and use a sliding window to extract image blocks from each scaled image; Step 1.3, augment the extracted image blocks.

3. The double residual denoising method based on attention allocation mechanism according to claim 1, characterized in that: The hyperparameters of the denoising network model in step 3 include batch size, initial learning rate, number of iterations, and learning rate decay strategy. The loss function is the mean square error, and the expression is as follows: ； Where: Network model parameters, is the residual image learned by the network, is a noisy image, represents a clean image, is the number of training samples.

4. The double residual denoising method based on attention allocation mechanism according to claim 1, characterized in that: The specific steps for step 4 are as follows: Step 4.1, add Gaussian white noise with noise levels of 15, 25, and 50 to the images in the test set to obtain noise training images; In step 4.2, the noisy training image is input into the denoising network model for training, thereby obtaining a trained denoising network model, and saving it every 20 epochs.

5. The double residual denoising method based on attention allocation mechanism according to claim 1, characterized in that: The test set in step 4 includes Set12 and BSD68 grayscale datasets.

Images