A dense LSTM residual network denoising method based on dynamic attention

Abstract

A denoising method based on a dynamic attention-based dense LSTM residual network includes: constructing a training dataset and preprocessing it; building a denoising model using a dynamic attention mechanism and a dense LSTM residual network; setting the hyperparameters and loss function of the denoising model; adding different levels of noise to the training dataset and training it to obtain the trained network model; and testing the trained network model on images, evaluating the denoising effect using structural similarity and peak signal-to-noise ratio. This method effectively improves denoising performance and image quality, enhancing image quality and facilitating subsequent high-level image understanding and analysis tasks such as image classification, segmentation, recognition, and tracking.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing, and specifically relates to a denoising method based on a dense LSTM residual network with dynamic attention. Background Technology

[0002] In recent years, with the continuous development of the information society, image information has become a major way for people to receive information. Digital images are inevitably affected by environmental, equipment, and human factors during acquisition and transmission, leading to noise that reduces image quality and affects readability. Image denoising is one of the most important methods to improve image quality. For many years, image denoising has been applied in various fields. For example, in the medical field, denoising medical images containing noise improves image clarity, helping medical staff better assess patient conditions. In aerospace and other fields, images are often interfered with by electronic devices during imaging, resulting in Gaussian white noise, salt-and-pepper noise, etc., which need to be improved through image denoising. Handheld mobile devices or vehicle dashcams often generate noise at night due to factors such as light and environment, affecting image quality and causing many inconveniences for subsequent image analysis and processing. At the same time, denoising facilitates subsequent image classification, segmentation, and recognition tasks.

[0003] Image denoising is a classic problem in image processing and a crucial preprocessing step in computer vision. Traditional image denoising algorithms rely heavily on prior model assumptions and manual parameter tuning, involving complex optimization problems and requiring significant time and resources. In recent years, with the continuous improvement of deep learning theory, experts have successfully applied it to image denoising, achieving excellent results. Deep neural networks effectively extract feature information from images through operations such as convolution and activation, contributing greatly to image denoising research. Currently, attention mechanisms are frequently used in building convolutional neural networks, widely applied in natural language understanding, image denoising, and recognition. While attention mechanisms can improve the denoising performance of convolutional neural networks, they require manual setting of the attention module's position within the network, and the intensity of the effect varies at different positions. This necessitates numerous experiments with different network structures to effectively improve network performance through attention mechanisms, significantly increasing labor costs. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a denoising method based on a dense LSTM residual network with dynamic attention. By considering spatial attention weights and inter-channel attention weights, it better guides the network in denoising while ensuring image denoising quality. Through the design of the LSTM structure, the dependency between global and local information is balanced, resulting in a significant improvement in both denoising performance and imaging quality.

[0005] A denoising method based on a dense LSTM residual network with dynamic attention, characterized by the following steps:

[0006] Step S1: Construct a training dataset and perform preprocessing operations on the dataset;

[0007] Step S2: Establish a denoising network model consisting of a dynamic attention mechanism and a dense LSTM residual network;

[0008] Step S3: Set the hyperparameters and loss function of the denoising network model;

[0009] Step S4: Add different levels of noise to the training dataset and train the network to obtain the trained denoising network model.

[0010] Step S5: Input the test set into the trained denoising network model to obtain the denoised image, and evaluate the noisy image using structural similarity and peak signal-to-noise ratio (substitute the denoised image output by the network model and the original image into the calculation formula for evaluation); put the denoising network model that has passed the evaluation into use, input the image to be processed, and output the denoised image.

[0011] Furthermore, in step S1, the preprocessing operation on the training dataset includes the following steps:

[0012] S1.1) Select training samples from the training dataset as the original training set, wherein the images in the training samples are all noise-free images of the same size;

[0013] S1.2) Scale the images in the training dataset by 1x, 0.9x, 0.8x, and 0.7x respectively, and segment each scaled image using a sliding segmentation window of a fixed size;

[0014] S1.3) The segmented image is augmented. The augmentation operation includes: flipping the image vertically, rotating it 90°, rotating it 180°, rotating it 270°, flipping it vertically and then rotating it 90°, flipping it vertically and then rotating it 180°, and flipping it vertically and then rotating it 270°.

[0015] Furthermore, in step S2, the denoising network model consists of convolution, LReLU, and 8 dynamic attention modules. The dynamic attention module includes front-end and back-end residual units, non-attention branches, attention branches, and weight allocation branches. The dynamic attention module adopts an LSTM structure, and the convolution kernel size used in the network model is 3×3 and 5×5.

[0016] Furthermore, the residual units at the front and back ends are residual modules, which consist of two convolutional layers with a kernel size of 3×3 and an LReLU activation function. Let Y be the input, and the mathematical expression is as follows:

[0017] X0 = ψ(W2*ψ(W1*Y+b1)+b2)

[0018] In the above formula, W1 and W2 represent 3×3 convolutional layers with 64 output channels, b1 and b2 are biases, ψ represents the LReLU activation function, and X0 represents the extracted shallow features.

[0019] Define X i-1 This represents the input to the dynamic attention module; the front-end feature extraction part consists of one residual module and an LReLU activation function, and the corresponding mathematical expression is:

[0020]

[0021] In the formula, E represents the residual module; then, Input the attention branch and the non-attention branch respectively.

[0022] Furthermore, the non-attention branch consists of a backbone network composed of two residual units and one LSTM module. Each residual unit contains one 3×3 convolutional layer and one 5×5 convolutional layer. The LSTM module is a recurrent network structure composed of residual modules, LSTM units, convolutional layers, and an attention mask layer. The mathematical expression for the LSTM unit is as follows:

[0023] i t =σ(W xi *X t +W hi *H t-1 +W ci ⊙C t-1 +b i )

[0024] f t =σ(W xf *X t +W hf *H t-1 +W cf ⊙C t-1 +b f )

[0025] C t =f t ⊙C t-1 +i t ⊙tanh(W xc *X t +W hc *H t-1 +bc )

[0026] o t =σ(W xo *X t +W ho *H t-1 +W co ⊙C t +b o )

[0027] H t =o t ⊙tanh(C t )

[0028] The LSTM consists of input gate i t Forgotten Gate t Output gate o t Long-term memory state C t and short-term memory state H t Composition; σ and tanh represent the Sigmoid activation function and the Tanh activation function, respectively, W jk This represents a convolution operation from j to k, ⊙ represents matrix dot product, and b j X represents the activation value of the j-th unit; t C represents the feature map obtained by the residual module. t The features sent to the next LSTM are encoded, H t As the current LSTM cell output, and input to the following LSTM cell.

[0029] Furthermore, the attention branch is composed of the backbone network B i and mask part M i Composition, represented as:

[0030]

[0031] In the above formula, F att Let represent the attention branch, i represent the i-th dynamic attention module, and B be the backbone network. i It contains 2 residual modules, and the mask part M i It includes dilated convolution, channel attention module, downsampling operation, upsampling operation, and sigmoid function.

[0032] Furthermore, the weight allocation branch consists of a spatial attention module and a channel attention module. The spatial attention module includes a pair of max pooling operations and average pooling operations. The feature maps are concatenated into a large feature map after being processed by the two pooling operations. Then, the spatial weights are obtained by the Sigmoid activation function and input into the channel attention module. The channel attention module consists of an average pooling layer, two linear layers, and a Softmax activation function. Finally, the dynamic attention weight distribution is used for both the non-attention branch and the attention branch.

[0033] Furthermore, in step S3, the hyperparameters of the denoising network model include batch size, initial learning rate, number of iterations, and learning rate decay strategy; the loss function is the mean squared error, and its mathematical expression is:

[0034]

[0035] In the formula, Θ represents the parameters of the dense LSTM residual network with dynamic attention mechanism, and R(y) i ;Θ) represents the residual image learned by the network, y i For a noisy image, x i N represents a clean image, and N is the number of training samples. The Train400 dataset is used as the training dataset. In each round, 40 image patches of size 128×128 are randomly extracted as samples, and the Adam optimizer is used for training with an initial learning rate of 1×10⁻⁶. -4 The learning rate decreases by 0.2 times every 10 rounds, and the network is trained for a total of 100 rounds.

[0036] Furthermore, in step S4, the training method for the denoising network model is as follows:

[0037] S4.1) Add Gaussian white noise with noise levels of 15, 25, and 50 to the images in the original training data set respectively;

[0038] S4.2) The training image with added noise is input into the network model for training, thereby obtaining a trained denoised network model, which is then saved. The specific training steps are implemented by running PYCharm.

[0039] Furthermore, in step S5, the formula for calculating structural similarity is:

[0040]

[0041] Where x and y are two images, The mean; σ is the variance; xy Let be the covariance of x and y; d1 = (k1L) 2 d2=(k2L)2 L is the dynamic range of the pixel; k1 = 0.01, k2 = 0.03;

[0042] The formula for calculating peak signal-to-noise ratio is:

[0043]

[0044] In the formula: x(i,j) and y(i,j) represent the pixel values ​​at corresponding positions in the initial image x(i,j) and the denoised image y(i,j), respectively, and Q represents the maximum gray value in the image.

[0045] The beneficial effects achieved by this invention are as follows:

[0046] (1) By considering spatial attention weights and inter-channel attention weights, the network is better guided to perform denoising and the denoising quality of the image is guaranteed.

[0047] (2) By designing an LSTM structure, the dependence of global and local information is taken into account. This method has a significant improvement in noise reduction performance and imaging quality.

[0048] (3) It is beneficial for subsequent high-level image understanding and analysis tasks such as image classification, segmentation, recognition and tracking. Attached Figure Description

[0049] Figure 1 This is a flowchart of the method in an embodiment of the present invention.

[0050] Figure 2 This is a schematic diagram of the denoising network model in an embodiment of the present invention.

[0051] Figure 3 This is a schematic diagram of the dynamic attention module in the network in an embodiment of the present invention.

[0052] Figure 4 This is a table of PSNR values ​​for the denoising results of various algorithms in the embodiments of the present invention.

[0053] Figure 5 This is a table of SSIM values ​​for the denoising results of various algorithms in the embodiments of the present invention. Detailed Implementation

[0054] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0055] like Figure 1 As shown, a dense LSTM residual network denoising algorithm based on dynamic attention includes the following steps:

[0056] Step S1: Construct a training dataset and perform preprocessing operations on the dataset.

[0057] In step S1, the preprocessing operation includes the following steps:

[0058] S1.1) Select training samples from the training dataset as the original training set, wherein the images in the training samples are all noise-free images of the same size; in this embodiment, the size of the training samples is 400 images with a height of 180 pixels and a width of 180 pixels.

[0059] S1.2) Scale the images in the training dataset by 1x, 0.9x, 0.8x and 0.7x respectively, and use a sliding segmentation window of fixed size to segment each scaled image.

[0060] S1.3) The segmented image is augmented. The augmentation operation includes: flipping the image vertically, rotating it 90°, rotating it 180°, rotating it 270°, flipping it vertically and then rotating it 90°, flipping it vertically and then rotating it 180°, and flipping it vertically and then rotating it 270°.

[0061] S1.4) Add Gaussian white noise with noise levels of 15, 25, and 50 to the images in the dataset.

[0062] Step S2: A denoising algorithm using a dynamic attention mechanism and a dense LSTM residual network.

[0063] like Figure 2 As shown, the network denoising model consists of convolutional operations, LReLU, and eight dynamic attention modules. The dynamic attention modules include spatial attention, channel attention, residual modules, and splicing layers. An LSTM structure is used inside the dynamic attention modules. The convolutional kernel sizes used in the network model are 3×3 and 5×5.

[0064] like Figure 3 As shown, the dynamic attention module comprises: front-end and back-end residual units, non-attention branches, attention branches, and weight allocation branches. The front-end and back-end residual units are residual modules, which consist of two 3×3 convolutional layers and an LReLU activation function. Let Y be the input, and the mathematical expression is as follows:

[0065] X0 = ψ(W2*ψ(W1*Y+b1)+b2)

[0066] In the above formula, W1 and W2 represent 3×3 convolutional layers with 64 output channels, b1 and b2 are biases, ψ represents the LReLU activation function, and X0 represents the extracted shallow features.

[0067] Suppose X i-1This represents the input to the dynamic attention module. The front-end feature extraction part consists of a residual module and an LReLU activation function, with the corresponding mathematical expression as follows:

[0068]

[0069] In the formula, E represents the residual module. Next, we will... Input the attention branch and the non-attention branch respectively.

[0070] The non-attention branch consists of a backbone network comprising two residual units and one LSTM module. The residual units contain two 3×3 and 5×5 convolutional layers, while the LSTM module is a recurrent network structure consisting of residual modules, LSTM units, convolutional layers, and an attention mask layer. The mathematical expression for the LSTM unit is as follows:

[0071] i t =σ(W xi *X t +W hi *H t-1 +W ci ⊙C t-1 +b i )

[0072] f t =σ(W xf *X t +W hf *H t-1 +W cf ⊙C t-1 +b f )

[0073] C t =f t ⊙C t-1 +i t ⊙tanh(W xc *X t +W hc *H t-1 +b c )

[0074] o t =σ(W xo *X t +W ho *H t-1 +W co ⊙C t +b o )

[0075] H t =o t ⊙tanh(C t )

[0076] The LSTM consists of input gate i t Forgotten Gate t Output gate o t Long-term memory state C t and short-term memory state H t Composition. σ and tanh represent the Sigmoid activation function and the Tanh activation function, respectively, W jk This represents a convolution operation from j to k, ⊙ represents a dot product between matrices, and b j X represents the activation value of the j-th unit. t C represents the feature map obtained by the residual module. t The features sent to the next LSTM are encoded, H t As the current LSTM cell output, and input to the following LSTM cell.

[0077] Attention branch is formed by backbone network B i and mask part M i Composition can be represented as:

[0078]

[0079] In the above formula, the backbone network B i It contains 2 residual modules, and the mask part M i It includes dilated convolution, channel attention module, downsampling operation, upsampling operation, and sigmoid function.

[0080] The weight allocation branch consists of a spatial attention module and a channel attention module. The spatial attention module includes a pair of max pooling and average pooling operations. The feature maps are concatenated into a large feature map after undergoing both pooling operations. Then, the spatial weights are obtained by the sigmoid activation function and input into the channel attention module. The channel attention module consists of an average pooling layer, two linear layers, and a softmax activation function. Finally, the dynamic attention weight distribution is used for both the non-attention and attention branches.

[0081] Step S3: Set the hyperparameters and loss function of the denoising network model.

[0082] The hyperparameters of the network model include batch size, initial learning rate, number of iterations, and learning rate decay strategy.

[0083] The loss function is the mean squared error, and its mathematical expression is:

[0084]

[0085] In the formula, Θ represents the parameters of the dense LSTM residual network with dynamic attention mechanism, and R(y) i;Θ) represents the residual image learned by the network, y i For a noisy image, x i This represents a clean image, and N is the number of training samples. The Train400 dataset is used as the training dataset. In each round, 40 image patches of size 128×128 are randomly extracted as samples, and the Adam optimizer is used for training with an initial learning rate of 1×10⁻⁶. -4 The learning rate decreases by 0.2 times every 10 rounds, and the network is trained for a total of 100 rounds.

[0086] Step S4: Add different levels of noise to the training dataset and train the network to obtain the trained network model.

[0087] In step S4, the training method for the denoising network model is as follows:

[0088] S4.1): Add Gaussian white noise with noise levels of 15, 25, and 50 to the images in the original training set, respectively;

[0089] S4.2): Input the training image with added noise into the network model for training, thereby obtaining the trained denoised network model, and save it.

[0090] Step S5: Input the test set into the network to obtain the denoised image, and evaluate the noisy image using structural similarity and peak signal-to-noise ratio.

[0091] The test set includes Set12 and BSD68 grayscale datasets, which are input into the trained network model for image denoising.

[0092] The formula for calculating structural similarity is:

[0093]

[0094] Where x and y are two images, The mean; σ is the variance; xy Let be the covariance of x and y; d1 = (k1L) 2 d2=(k2L) 2 L is the dynamic range of the pixel; k1 = 0.01, k2 = 0.03.

[0095] The formula for calculating peak signal-to-noise ratio is:

[0096]

[0097] In the formula, x(i,j) and y(i,j) represent the pixel values ​​at corresponding positions in the initial image x(i,j) and the denoised image y(i,j), respectively, and Q represents the maximum gray value in the image.

[0098] Figure 4 and Figure 5 The PSNR and SSIM metrics of various algorithms on different datasets are presented. Our proposed method achieves good PSNR results under three different noise levels. On the Set12 test dataset, our method achieves average PSNR metrics of 33.05 dB and 27.59 dB for noise levels of 15 and 50, respectively, which are 0.20 dB higher than the FFDNet algorithm. At a noise level of 25, the PSNR of our algorithm on Set12 is slightly lower than that of FDnCNN. Under high noise levels of 15 and 25, our method outperforms other algorithms in terms of SSIM metrics.

[0099] The above description is only a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. Any equivalent modifications or changes made by those skilled in the art based on the content disclosed in the present invention should be included within the scope of protection set forth in the claims.

Claims

1. A denoising method based on a dense LSTM residual network with dynamic attention, characterized in that: Includes the following steps: Step S1: Construct a training dataset and perform preprocessing operations on the dataset; Step S2: Establish a denoising network model consisting of a dynamic attention mechanism and a dense LSTM residual network. In step S2, the denoising network model consists of convolution, LReLU, and 8 dynamic attention modules. The dynamic attention module includes front-end and back-end residual units, non-attention branches, attention branches, and weight allocation branches. An LSTM structure is used inside the dynamic attention module. The convolution kernel size used inside the network model is 3×3 and 5×5. Step S3: Set the hyperparameters and loss function of the denoising network model; Step S4: Add different levels of noise to the training dataset and train the network to obtain the trained denoising network model. Step S5: Input the test set into the trained denoising network model to obtain the denoised image, and evaluate the noisy image using structural similarity and peak signal-to-noise ratio; put the denoising network model that has passed the evaluation into use, input the image to be processed, and output the denoised image.

2. The denoising method based on a dense LSTM residual network with dynamic attention according to claim 1, characterized in that: Step S1 involves preprocessing the training dataset, including the following steps: S1.1) Select training samples from the training dataset as the original training set, wherein the images in the training samples are all noise-free images of the same size; S1.2) Scale the images in the training dataset by 1x, 0.9x, 0.8x, and 0.7x respectively, and slide a fixed-size segmentation window to extract image patches from each scaled image; S1.3) The segmented image is augmented. The augmentation operation includes: flipping the image vertically, rotating it 90°, rotating it 180°, rotating it 270°, flipping it vertically and then rotating it 90°, flipping it vertically and then rotating it 180°, and flipping it vertically and then rotating it 270°.

3. The denoising method based on a dense LSTM residual network with dynamic attention according to claim 2, characterized in that: The residual units at the front and back ends are residual modules, which consist of two convolutional layers with a kernel size of 3×3 and an LReLU activation function. Let Y be the input, and the mathematical expression is as follows: X0 = ψ(W2*ψ(W1*Y+b1)+b2) In the above formula, W1 and W2 represent 3×3 convolutional layers with 64 output channels, b1 and b2 are biases, ψ represents the LReLU activation function, and X0 represents the extracted shallow features. definition X i-1 This represents the input to the dynamic attention module; the front-end feature extraction part consists of one residual module and an LReLU activation function, and the corresponding mathematical expression is: In the formula, E represents the residual module; then, Input the attention branch and the non-attention branch respectively.

4. The denoising method based on a dense LSTM residual network with dynamic attention according to claim 2, characterized in that: The non-attention branch consists of a backbone network composed of two residual units and one LSTM module. Each residual unit contains one 3×3 convolutional layer and one 5×5 convolutional layer. The LSTM module is a recurrent network structure composed of residual modules, LSTM units, convolutional layers, and an attention mask layer. The mathematical expression for the LSTM unit is as follows: i t =σ(W xi *X t +W hi *H t-1 +W ci ⊙C t-1 +b i ) f t =σ(W xf *X t +W hf *H t-1 +W cf ⊙C t-1 +b f ) C t =f t ⊙C t-1 +i t ⊙tanh(W xc *X t +W hc *H t-1 +b c ) o t =σ(W xo *X t +W ho *H t-1 +W co ⊙C t +b o ) H t =o t ⊙tanh(C t ) The LSTM consists of input gate i t Forgotten Gate t Output gate o t Long-term memory state C t and short-term memory state H t Composition; σ and tanh represent the Sigmoid activation function and the Tanh activation function, respectively, W jk This represents a convolution operation from j to k, ⊙ represents a matrix dot product, and b j X represents the activation value of the j-th unit; t C represents the feature map obtained by the residual module. t The features sent to the next LSTM are encoded, H t As the current LSTM cell output, and input to the following LSTM cell.

5. The denoising method based on a dense LSTM residual network with dynamic attention according to claim 2, characterized in that: The attention branch is formed by the backbone network B. i and mask part M i Composition, represented as: In the above formula, F att Let represent the attention branch, i represent the i-th dynamic attention module, and B be the backbone network. i It contains 2 residual modules, and the mask part M i It includes dilated convolution, channel attention module, downsampling operation, upsampling operation, and sigmoid function.

6. The denoising method based on a dense LSTM residual network with dynamic attention according to claim 2, characterized in that: The weight allocation branch consists of a spatial attention module and a channel attention module. The spatial attention module includes a pair of max pooling and average pooling operations. The feature maps are concatenated into a large feature map after being processed by the two pooling operations. Then, the spatial weights are obtained by the Sigmoid activation function and input into the channel attention module. The channel attention module consists of an average pooling layer, two linear layers, and a Softmax activation function. Finally, the dynamic attention weight distribution is used for both the non-attention branch and the attention branch.

7. The denoising method based on a dense LSTM residual network with dynamic attention according to claim 2, characterized in that: In step S3, the hyperparameters of the denoising network model include batch size, initial learning rate, number of iterations, and learning rate decay strategy; the loss function is the mean squared error, and its mathematical expression is: In the formula, Θ represents the parameters of the dense LSTM residual network with dynamic attention mechanism, and R(y) i ;Θ) represents the residual image learned by the network, y i For a noisy image, x i N represents a clean image, and N is the number of training samples. The Train400 dataset is used as the training dataset. The network hyperparameters are set such that 40 image patches of size 128×128 are randomly extracted as samples in each round, and the Adam optimizer is used for training with an initial learning rate of 1×10⁻⁶. -4 The learning rate decreases by 0.2 times every 10 rounds, and the network is trained for a total of 100 rounds.

8. The denoising method based on a dense LSTM residual network with dynamic attention according to claim 7, characterized in that: In step S4, the training method for the denoising network model is as follows: S4.1) Add Gaussian white noise with noise levels of 15, 25, and 50 to the images in the original training data set respectively; S4.2) The training image with added noise is input into the network model for training, thereby obtaining the trained denoised network model and saving it.

9. A method for denoising dense LSTM residual networks based on dynamic attention according to claim 8, characterized in that: In step S5, the formula for calculating structural similarity is: Where x and y are two images, The mean; σ is the variance; xy Let be the covariance of x and y; d1 = (k1L) 2 d2=(k2L) 2 L is the dynamic range of the pixel; k1 = 0.01, k2 = 0.03; The formula for calculating peak signal-to-noise ratio is: In the formula: x(i,j) and y(i,j) represent the pixel values ​​at corresponding positions in the initial image x(i,j) and the denoised image y(i,j), respectively, and Q represents the maximum gray value in the image.

Images