Image Denoising Method Based on Deep Learning Relu and Channel Correlation Attention

Abstract

The present invention discloses an image denoising method based on deep learning Relu and channel correlation attention, which solves the problems of low efficiency of the denoising model in the prior art, and the problem that the edge contour information of the image after denoising is over-smoothed and the texture information is lost. The implementation steps of the present invention are as follows: designing a Relu activation layer and a channel correlation attention layer, which are respectively used to construct a multi-bias Relu double-branch module and a channel correlation attention module, and constructing a multi-path Unet denoising network through the multi-bias Relu double-branch module and the channel correlation attention module; training the multi-path Unet denoising network by using the generated training set; denoising the image. The present invention can better integrate useful features, improve the denoising efficiency of the model, and better preserve the edge contour information and texture information in the image after denoising.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and further relates to an image denoising method based on deep learning Relu and channel correlation attention in the field of image restoration technology. The present invention can be used for image denoising, and while removing the image noise information, more original image information can be restored. Background Art

[0002] Image denoising is one of the basic tasks of image processing and has important applications in real life, including the restoration of old photos and videos, algorithms for mobile phone shooting at night, and night-time autonomous driving. Images will inevitably generate noise during the acquisition and transmission process, especially in the case of insufficient light at night, where insufficient exposure time will cause the image to contain a large amount of noise. The current image denoising methods are mainly divided into two categories: one is the traditional image denoising method, which mainly performs denoising processing on the image in the spatial domain and the transform domain; the other is the deep learning-based image denoising method, which uses a convolutional neural network to denoise the image.

[0003] Syed Waqas Zamir et al. proposed a three-stage deep learning image denoising method in their published paper "Multi-Stage Progressive Image Restoration" (Published as a conference paper at CVPR 2021, hereinafter referred to as MPRnet). In the first two stages of this method, an encoder-decoder Unet sub-network is used to learn extensive context information; then in the third stage, a sub-network that operates on the resolution of the original input image is adopted, so as to retain the required fine texture in the final output image; finally, a supervised attention module is added between every two stages to readjust the feature map of the previous stage before passing them to the next stage. The disadvantage of this method is that the Unet network is used in both the first two stages, increasing the number of model parameters and the amount of calculation, resulting in low model efficiency.

[0004] Chongqing University of Posts and Telecommunications discloses an end-to-end image denoising method in its patent literature application "A Spatial Attention Image Denoising Method Combining Local Information and Global Information" (Application No.: 202011305283.2, Application Date: November 19, 2020, Publication No.: CN 112419184A). This method first proposes an image shallow feature extraction module, an image feature enhancement group, and an image reconstruction module; then combines a non-local module, a spatial attention residual block, and a dilated convolution in series and parallel to simultaneously capture local and global features of the image; finally, restores the clean image by taking the residual with the noisy image. However, the deficiency of this method is that the end-to-end residual learning optimization target is for the single-frame overall image, without region segmentation of the target image's edge and flat regions, and the proposed spatial attention mechanism cannot focus on the image edge, resulting in over-smoothing of the edge contour information of the denoised image. Summary of the Invention

[0005] The purpose of the present invention is to address the above deficiencies of the prior art and propose an image denoising method based on deep learning Relu and channel correlation attention, which is used to solve the problems that using the Unet network increases the number of model parameters and computational complexity, resulting in low model efficiency, and the spatial attention mechanism cannot focus on the image edge, resulting in over-smoothing of the edge contour information of the denoised image.

[0006] The idea of achieving the purpose of the present invention is that since the present invention calculates the similarity of features between channels, uses the similarity between channels to integrate similar features and reduce redundant features, and improves the utilization rate of the model, thus solving the problem of low denoising model efficiency caused by not considering the similarity between channels in the prior art. Since the present invention uses the difference between the denoised image and the real image to obtain masks with different levels of difficulty, divides the single-frame image to be optimized into two regions with different denoising difficulties, and denoises them independently. This solves the problem that the prior art does not perform region segmentation on different denoising difficulties of the target image, resulting in over-smoothing of the edge contour information of the denoised image and loss of texture information.

[0007] To achieve the above purpose, the specific implementation steps of the present invention are as follows:

[0008] Step 1, construct a multi-bias Relu dual-branch module:

[0009] Step 1.1, build an upper-branch circuit, whose structure is sequentially connected in series as: a layer normalization layer, a first convolutional layer with a kernel size of 1*1, a depthwise separable convolutional layer with a kernel size of 3*3, a multi-bias activation group composed of Relu activation layers in parallel, and a second convolutional layer with a kernel size of 1*1;

[0010] Step 1.2: Build a lower branch circuit, whose structure is connected in series in turn as follows: a layer normalization layer, a first convolutional layer with a convolutional kernel size of 1*1, a depthwise separable convolutional layer with a convolutional kernel size of 5*5, a multi-bias activation group composed of a Relu activation layer in parallel, and a second convolutional layer with a convolutional kernel size of 1*1;

[0011] Step 1.3: After paralleling the input ends of the upper and lower branch circuits and then connecting them to an adder, connect the output ends of the upper and lower branch circuits to a matrix multiplier, and then connect them in series with a convolutional layer with a convolutional kernel size of 1*1 and an adder in turn to form a multi-bias Relu double-branch module;

[0012] Step 2: Construct a channel correlation attention module:

[0013] Build a channel correlation attention module including three branches. The first branch and the second branch are respectively connected to a multiplier and then connected to the third branch. Among them, the first branch is composed of a layer normalization layer, a convolutional layer with a convolutional kernel size of 1*1, a depthwise separable convolutional layer with a convolutional kernel size of 3*3, a Relu activation layer, and a channel correlation attention layer connected in series in turn; the second branch is composed of connecting the output end of the Relu activation layer to the channel correlation attention layer; the third branch is composed of connecting the input end of the normalization layer to an adder;

[0014] The implementation process of the channel correlation attention layer is as follows: calculate the cosine similarity matrix of the channel dimension for the output result of the Relu activation layer, perform softmax on the pixel values of the cosine similarity matrix row by row, then replace the maximum pixel value with the sub-maximum value of the cosine similarity matrix pixel value row by row, and then set the pixel values less than the mean value of the row in the cosine similarity matrix pixel value to 0;

[0015] Step 3: Construct a multi-path Unet denoising network:

[0016] Step 3.1: Build three subnets, namely a hard denoising subnet HardNet, an easy denoising subnet EasyNet, and a backbone denoising subnet BackboneNet. Each subnet is composed of five branches. Among them, the first branch is composed of a first convolutional layer with a convolutional kernel size of 3*3, a first-level downsampling module, a second-level downsampling module, a third-level downsampling module, a first-level upsampling module, a second-level upsampling module, a third-level upsampling module, and a second convolutional layer with a convolutional kernel size of 3*3 connected in series in turn; the second branch is composed of connecting the output end of the first-level downsampling module to the input end of the third-level upsampling module; the third branch is composed of connecting the output end of the second-level downsampling module to the input end of the second-level upsampling module; the fourth branch is composed of connecting the first convolutional layer and the second convolutional layer in series; the fifth branch is composed of connecting the input end of the first convolutional layer and the output end of the second convolutional layer to an adder respectively;

[0017] The structures of the first to third upsampling modules are the same, and each is composed of a PixelUnshuffle upsampling layer, a multi-bias Relu double-branch module, and a channel correlation attention module connected in series in sequence; the structures of the first to third downsampling modules are the same, and each is composed of a PixelUnshuffle downsampling layer, a multi-bias Relu double-branch module, and a channel correlation attention module connected in series in sequence;

[0018] Step 3.2, connect the input ends of the HardNet (difficult denoising subnet) and the EasyNet (easy denoising subnet) in parallel, and then connect the output ends of the two subnets to the input end of the BackboneNet (main denoising subnet) to form a multi-path Unet denoising network;

[0019] Step 4, generate a training set:

[0020] Step 4.1, select at least 300 pairs of images, and each pair of images consists of a real noisy image and a manually annotated clean image;

[0021] Step 4.2, crop each image into image patches of size 256*256, and perform data augmentation on each image patch;

[0022] Step 4.3, form all the augmented image patches into a training set;

[0023] Step 5, train the multi-path Unet denoising network:

[0024] Randomly extract noisy image patches and the corresponding manually annotated clean image patches from the training set, input the extracted noisy image patches into the multi-path Unet denoising network. The BackboneNet (main denoising subnet) uses the mean absolute error loss function L1loss, the HardNet (difficult denoising subnet) uses the hard denoising loss function hardloss, and the EasyNet (easy denoising subnet) uses the easy denoising loss function easyloss. Calculate the loss values between the image patches output by the three subnets and the clean image patches respectively. Adopt the gradient descent method to iteratively update the parameters of each layer of the multi-path Unet denoising network until the mean absolute error loss function converges, and obtain the trained multi-path Unet denoising network;

[0025] Step 6, denoise the image:

[0026] Adopt the same method as in Step 4.2 to process the image to be denoised, input the processed image into the trained multi-path Unet denoising network, and output the denoised image.

[0027] The present invention has the following advantages compared with the existing technologies:

[0028] First, since the present invention calculates the similarity of features between channels, it overcomes the problem of low efficiency of the denoising model in the prior art due to the lack of consideration of the similarity between channels, enabling the present invention to better integrate useful features and improve the denoising efficiency of the model.

[0029] Second, since the present invention utilizes the difference between the denoised image and the real image to obtain masks with different levels of difficulty, it overcomes the problem of excessive smoothing of the edge contour information and loss of texture information in the denoised image in the prior art due to the lack of regional segmentation of different denoising difficulties of the target image, enabling the present invention to better preserve the edge contour information and texture information in the denoised image. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flowchart of the present invention;

[0031] Figure 2 is a schematic diagram of the upper branch circuit of the present invention;

[0032] Figure 3 is a schematic diagram of the lower branch circuit of the present invention;

[0033] Figure 4 is a schematic diagram of the multi-bias Relu dual-branch module of the present invention;

[0034] Figure 5 is a schematic diagram of the channel correlation attention module of the present invention;

[0035] Figure 6 is a schematic diagram of the multi-path Unet denoising network of the present invention;

[0036] Figure 7 is a schematic diagram of the hard denoising subnet HardNet, easy denoising subnet EasyNet, and backbone denoising subnet BackboneNet of the present invention;

[0037] Figure 8 is a schematic diagram of the upsampling module of the present invention;

[0038] Figure 9 is a schematic diagram of the downsampling module of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0039] The present invention will be further described in detail below with reference to the drawings and embodiments.

[0040] Refer to Figure 1 , and the implementation steps of the embodiments of the present invention will be further described in detail.

[0041] Step 1, construct a multi-bias Relu dual-branch module.

[0042] Refer toFigure 2 , a further description of the structure of the built upper branch circuit is given.

[0043] Build an upper branch circuit, the structure of which is successively connected in series as: a layer normalization layer, a first convolutional layer with a convolutional kernel size of 1*1, a depthwise separable convolutional layer with a convolutional kernel size of 3*3, a multi-bias activation group composed of parallel Relu activation layers, and a second convolutional layer with a convolutional kernel size of 1*1; The Relu activation layer is implemented by the following formula:

[0044] Relu(i) = relu(i - bias) + bias

[0045] where i represents the output result of the depthwise separable convolutional layer, relu represents the rectified linear unit function, and bias is a learnable bias of the same size as i, initialized as an orthonormal matrix.

[0046] Refer to Figure 3 , a further description of the structure of the built lower branch circuit is given.

[0047] Build a lower branch circuit, the structure of which is successively connected in series as: a layer normalization layer, a first convolutional layer with a convolutional kernel size of 1*1, a depthwise separable convolutional layer with a convolutional kernel size of 5*5, a multi-bias activation group composed of parallel Relu activation layers, and a second convolutional layer with a convolutional kernel size of 1*1.

[0048] Refer to Figure 4 , a further description of the structure of the built multi-bias Relu double-branch module is given.

[0049] After paralleling the input ends of the upper and lower branch circuits and then connecting them to an adder, and connecting the output ends of the upper and lower branch circuits to a matrix multiplier, and then successively connecting them in series with a convolutional layer with a convolutional kernel size of 1*1 and an adder, a multi-bias Relu double-branch module is formed.

[0050] Step 2, construct a channel correlation attention module.

[0051] Refer to Figure 5 , a further description of the structure of the built channel correlation attention module is given.

[0052] Build a channel correlation attention module including three branches. The first branch and the second branch are respectively connected to a multiplier and then connected to the third branch. Among them, the first branch is successively connected in series by a layer normalization layer, a convolutional layer with a convolutional kernel size of 1*1, a depthwise separable convolutional layer with a convolutional kernel size of 3*3, a Relu activation layer, and a channel correlation attention layer; The second branch is composed of connecting the output end of the Relu activation layer and the channel correlation attention layer; The third branch is composed of connecting the input end of the normalization layer and an adder.

[0053] The channel correlation attention layer is implemented by Channel Similarity:

[0054] Channel Similarity(h) = Z(R(softmax(Cosine(h))))

[0055] Where h represents the output result of the Relu activation layer, Cosine(·) represents the operation of calculating the cosine similarity matrix in the channel dimension, softmax represents the normalized exponential function, R represents replacing the maximum pixel value of each row with the second maximum pixel value of the cosine similarity matrix pixel values row by row, and Z(·) represents setting the pixel values less than the mean value of each row of the cosine similarity matrix pixel values to 0 row by row.

[0056] The implementation process of the channel correlation attention layer is as follows: calculate the cosine similarity matrix in the channel dimension for the output result of the Relu activation layer, perform softmax on the pixel values of the cosine similarity matrix row by row, then replace the maximum pixel value with the second maximum pixel value of the cosine similarity matrix pixel values row by row, and then set the pixel values less than the mean value of each row of the cosine similarity matrix pixel values to 0 row by row.

[0057] Step 3, construct a multi-path Unet denoising network.

[0058] Refer to Figure 6 for a further description of the structure of the multi-path Unet denoising network.

[0059] Refer to Figure 7 for a further description of the three subnets: the hard denoising subnet HardNet, the easy denoising subnet EasyNet, and the backbone denoising subnet BackboneNet.

[0060] Construct three subnets: the hard denoising subnet HardNet, the easy denoising subnet EasyNet, and the backbone denoising subnet BackboneNet. Each subnet consists of five branches. Among them, the first branch is composed of a first convolutional layer with a convolution kernel size of 3*3, a first-level downsampling module, a second-level downsampling module, a third-level downsampling module, a first-level upsampling module, a second-level upsampling module, a third-level upsampling module, and a second convolutional layer with a convolution kernel size of 3*3 connected in series in sequence; the second branch is composed of connecting the output end of the first-level downsampling module to the input end of the third-level upsampling module; the third branch is composed of connecting the output end of the second-level downsampling module to the input end of the second-level upsampling module; the fourth branch is composed of connecting the first convolutional layer and the second convolutional layer in series; the fifth branch is composed of connecting the input end of the first convolutional layer and the output end of the second convolutional layer to an adder respectively;

[0061] Refer to Figure 8, a further description of the structure of the upsampling module built is given.

[0062] The structures of the first to third level upsampling modules are the same, and each is composed of a pixel recombination upsampling layer PixelUnshuffle, a multi-bias Relu double-branch module, and a channel correlation attention module connected in series in sequence.

[0063] Refer to Figure 9 , a further description of the structure of the downsampling module built is given.

[0064] The structures of the first to third level downsampling modules are the same, and each is composed of a pixel recombination downsampling layer PixelUnshuffle, a multi-bias Relu double-branch module, and a channel correlation attention module connected in series in sequence;

[0065] The input ends of the hard denoising subnet HardNet and the easy denoising subnet EasyNet are connected in parallel, and then the output ends of the two subnets are connected to the input end of the backbone denoising subnet BackboneNet to form a multi-path Unet denoising network.

[0066] Step 4, generate a training set.

[0067] Select 320 pairs of images from the publicly available dataset SIDD. Each pair of images consists of a real noisy image and a manually annotated clean image; each image is cropped into image patches of size 256*256, and data augmentation is performed on each image patch; all the augmented image patches are combined to form a training set.

[0068] Step 5, train the multi-path Unet denoising network.

[0069] Randomly extract noisy image patches and the corresponding manually annotated clean image patches from the training set. Input the extracted noisy image patches into the multi-path Unet denoising network. The backbone denoising subnet BackboneNet uses the mean absolute error loss function L1loss, the hard denoising subnet HardNet uses the hard denoising loss function hardloss, and the easy denoising subnet EasyNet uses the easy denoising loss function easyloss. Calculate the loss values between the image patches output by the three subnets and the clean image patches respectively. Using the gradient descent method, iteratively update the parameters of each layer of the multi-path Unet denoising network until the mean absolute error loss function converges, and obtain the trained multi-path Unet denoising network.

[0070] The mean absolute error loss function L1loss, the hard denoising loss function hardloss, and the easy denoising loss function easyloss are as follows:

[0071] L1loss(y, x) = M(|y - x|)

[0072] If |x - y| > M(|x - y|), then hardmask = hardmask + 1

[0073] Otherwise hardmask = hardmask

[0074] hardloss(z, x) = L1loss(z * hardmask, x * hardmask)

[0075] easyloss(β, x) = L1loss(β * (1 - hardmask), x * (1 - hardmask))

[0076] Where x represents the clean image patch, y represents the image patch output by the backbone denoising subnet, z represents the image patch output by the hard denoising subnet, β represents the image patch output by the easy denoising subnet, hardmask represents an intermediate variable initialized to 0 with the same size as x, |·| represents the absolute value operation, and M(·) represents the average value operation.

[0077] Step 6, denoise the image.

[0078] Crop the image to be denoised into image patches of size 256 * 256, input the processed image patches into the trained multi-path Unet denoising network, and output the denoised image.

Claims

1. An image denoising method based on deep learning Relu and channel correlation attention, characterized in that, Design a Relu activation layer and a channel correlation attention layer to construct a multi-bias Relu double-branch module and a channel correlation attention module respectively; the specific steps of this method are as follows: Step 1, construct a multi-bias Relu double-branch module: Step 1.1, build an upper-branch circuit, and its structure is connected in series in turn as follows: a layer normalization layer, a first convolutional layer with a kernel size of 1*1, a depthwise separable convolutional layer with a kernel size of 3*3, a multi-bias activation group composed of parallel Relu activation layers, and a second convolutional layer with a kernel size of 1*1; Step 1.2, build a lower-branch circuit, and its structure is connected in series in turn as follows: a layer normalization layer, a first convolutional layer with a kernel size of 1*1, a depthwise separable convolutional layer with a kernel size of 5*5, a multi-bias activation group composed of parallel Relu activation layers, and a second convolutional layer with a kernel size of 1*1; Step 1.3, connect the input ends of the upper and lower branch circuits in parallel and then connect them to an adder, connect the output ends of the upper and lower branch circuits to a matrix multiplier, and then connect them in series with a convolutional layer with a kernel size of 1*1 and an adder in turn to form a multi-bias Relu double-branch module; Step 2, construct a channel correlation attention module: Build a channel correlation attention module including three branches. The first branch and the second branch are respectively connected to a multiplier and then connected to the third branch. Among them, the first branch is composed of a layer normalization layer, a convolutional layer with a kernel size of 1*1, a depthwise separable convolutional layer with a kernel size of 3*3, a Relu activation layer, and a channel correlation attention layer connected in series in turn; the second branch is composed of connecting the output end of the Relu activation layer and the channel correlation attention layer; the third branch is composed of connecting the input end of the normalization layer and an adder; The implementation process of the channel correlation attention layer is as follows: calculate the cosine similarity matrix of the channel dimension for the output result of the Relu activation layer, perform softmax on the pixel values of the cosine similarity matrix row by row, then replace the pixel value maximum with the sub-maximum value of the pixel values of the cosine similarity matrix row by row, and then set the pixel values less than the row mean of the cosine similarity matrix pixel values to 0; Step 3, construct a multi-path Unet denoising network: Step 3.1, construct three subnets: the Hard-to-Denoise Subnet HardNet, the Easy-to-Denoise Subnet EasyNet, and the Backbone Denoising Subnet BackboneNet. Each subnet consists of five branches. Among them, the first branch is composed of a first convolutional layer with a convolutional kernel size of 3*3, a first-level downsampling module, a second-level downsampling module, a third-level downsampling module, a first-level upsampling module, a second-level upsampling module, a third-level upsampling module, and a second convolutional layer with a convolutional kernel size of 3*3 connected in series in sequence; the second branch is composed of connecting the output end of the first-level downsampling module to the input end of the third-level upsampling module; the third branch is composed of connecting the output end of the second-level downsampling module to the input end of the second-level upsampling module; the fourth branch is composed of connecting the first convolutional layer and the second convolutional layer in series; the fifth branch is composed of connecting the input end of the first convolutional layer and the output end of the second convolutional layer to an adder respectively; The structures of the first to third-level upsampling modules are the same, and each is composed of a Pixel Unshuffle upsampling layer, a multi-bias Relu double-branch module, and a channel correlation attention module connected in series in sequence; the structures of the first to third-level downsampling modules are the same, and each is composed of a Pixel Unshuffle downsampling layer, a multi-bias Relu double-branch module, and a channel correlation attention module connected in series in sequence; Step 3.2, connect the input ends of the Hard-to-Denoise Subnet HardNet and the Easy-to-Denoise Subnet EasyNet in parallel, and then connect the output ends of the two subnets to the input end of the Backbone Denoising Subnet BackboneNet to form a multi-path Unet denoising network; Step 4, generate a training set: Step 4.1, select at least 300 pairs of images, and each pair of images consists of a real noisy image and a manually annotated clean image; Step 4.2, crop each image into image patches of size 256*256, and perform data augmentation on each image patch; Step 4.3, form all the augmented image patches into a training set; Step 5, train the multi-path Unet denoising network: Randomly extract noisy image patches and the corresponding manually annotated clean image patches from the training set, input the extracted noisy image patches into the multi-path Unet denoising network. The Backbone Denoising Subnet BackboneNet uses the mean absolute error loss function L1loss, the Hard-to-Denoise Subnet HardNet uses the hard-to-denoise loss function hardloss, and the Easy-to-Denoise Subnet EasyNet uses the easy-to-denoise loss function easyloss. Calculate the loss values between the image patches output by the three subnets and the clean image patches respectively. Adopt the gradient descent method to iteratively update the parameters of each layer of the multi-path Unet denoising network until the mean absolute error loss function converges, and obtain the trained multi-path Unet denoising network; Step 6, denoise the image: Adopt the same method as in Step 4.2 to process the image to be denoised, input the processed image into the trained multi-path Unet denoising network, and output the denoised image.

2. The image denoising method based on deep learning Relu and channel correlation attention according to claim 1, characterized in that The Relu activation layer described in Step 1.1 is implemented by the following formula: Relu(i) = relu(i - bias) + bias Where, i represents the output result of the depthwise separable convolutional layer, relu represents the rectified linear unit function, and bias is a learnable bias of the same size as i, initialized as an orthogonalization matrix.

3. The image denoising method based on deep learning Relu and channel correlation attention according to claim 1, characterized in that The channel correlation attention layer described in step 2 is implemented by Channel Similarity: Channel Similarity(h) = Z(R(softmax(Cosine(h)))) Where, h represents the output result of the Relu activation layer, Cosine(·) represents the operation of calculating the cosine similarity matrix in the channel dimension, softmax represents the normalized exponential function, R represents replacing the pixel value of the maximum value in each row with the second maximum value of the pixel values in the cosine similarity matrix, and Z(·) represents setting the pixel values less than the mean value of each row in the cosine similarity matrix to 0 for each row.

4. The image denoising method based on deep learning Relu and channel correlation attention according to claim 1, wherein The mean absolute error loss function L1loss, the hard denoising loss function hardloss, and the easy denoising loss function easyloss described in step 5 are as follows: L1loss(y, x) = M(|y - x|) If |x - y| > M(|x - y|), then hardmask = hardmask + 1 Otherwise hardmask = hardmask hardloss(z, x) = L1loss(z * hardmask, x * hardmask) easyloss(β, x) = L1loss(β * (1 - hardmask), x * (1 - hardmask)) Where, x represents the clean image patch, y represents the image patch output by the backbone denoising subnet, z represents the image patch output by the hard denoising subnet, β represents the image patch output by the easy denoising subnet, hardmask represents an intermediate variable of the same size as x initialized to 0, |·| represents the absolute value operation, and M(·) represents the averaging operation.

Images