Image denoising method and device based on Jacobian dynamic approximation

Abstract

The present invention discloses an image denoising method and device based on Jacobian dynamic approximation, and relates to the field of computer vision technology. The method comprises: obtaining an image to be denoised; inputting the image to be denoised into a shallow feature extraction unit of an image denoising network to obtain a shallow feature image; inputting the shallow feature image into a denoising unit of an image denoising network to obtain a denoised image; inputting the denoised image into a dual attention unit of an image denoising network to obtain a final denoised image. The present invention constructs an end-to-end image denoising network based on Jacobian dynamic approximation, and designs an ordinary differential equation solving module based on Jacobian matrix to realize the forward derivation of a neural network. The present invention can be successfully applied to image denoising of synthetic noise and real noise, and compared with the classical image denoising method, it can achieve significant denoising performance improvement in both the subjective and objective quality of the reconstructed image.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an image denoising method and device based on Jacobian dynamic approximation. Background Art

[0002] During the image acquisition or processing process, due to equipment limitations and environmental influences, the acquired images often inevitably introduce varying degrees of noise, which greatly reduces the visual quality of the image and further limits the performance improvement of computer vision tasks. Therefore, image denoising is one of the important research topics in the field of computer vision and has received extensive attention from industry and academia.

[0003] According to the characteristics of noise, traditional image denoising methods mainly include image noise filtering algorithms based on spatial domain or frequency domain. Spatial domain filtering mainly designs filters based on the differences between noise pixels and image pixels, such as Gaussian filtering, Wiener filtering, median filtering, etc.; frequency domain filtering mainly uses Fourier transform, wavelet transform and other technologies to transform the image into the frequency domain, and uses the difference between noise and image information in the transform domain to separate information from noise in the image. The noise signal is analyzed and removed based on the correlation of local information, without using the neighboring relationship in pixel space. Most traditional image denoising methods remove noise under the premise of known noise distribution characteristics (such as additive Gaussian white noise), but the complexity of noise in real environments is high, resulting in the noise distribution not necessarily obeying a specific distribution characteristic, which restricts the practical application of these algorithms.

[0004] In recent years, thanks to the powerful learning ability of CNN (Deep Convolutional Neural Networks), deep neural networks are used to construct noise models to estimate the distribution parameters of noise and remove complex noise in real images. The representative technology is the image denoising deep convolutional neural network based on residual learning strategy. Although the residual learning strategy has achieved good denoising effect on real images, these strategies use feature correlation under a single scale, which limits the improvement of denoising performance. In order to improve the stability and approximation ability of the residual network, the residual network is linked to the dynamic system, and ODENet (Ordinary Differential Equation Networks, deep neural network based on ordinary differential equations) is constructed. The adaptive ODE (Ordinary Differential Equation) solver is integrated into the forward propagation process, and the appropriate step size is selected according to the given error tolerance to approximate the solution of the ordinary differential equation. The existing technology has proved that the ODE network can achieve the same performance as the residual network with fewer parameters.

[0005] Therefore, how to better remove synthetic noise and real noise in noisy images and improve the user's visual experience is a problem existing in the existing technology. Summary of the invention

[0006] The present invention aims at solving the problem of how to better remove synthetic noise and real noise in noisy images in the prior art and improve the visual experience of users, and proposes the present invention.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] On the one hand, the present invention provides an image denoising method based on Jacobian dynamic approximation, the method is implemented by an electronic device, and the method includes:

[0009] S1. Obtain an image to be denoised.

[0010] S2. Input the image to be denoised into the shallow feature extraction unit of the image denoising network to obtain a shallow feature image.

[0011] S3. Input the shallow feature image into the denoising unit of the image denoising network to obtain the denoised image.

[0012] S4. Input the denoised image into the dual attention unit of the image denoising network to obtain the final denoised image.

[0013] Optionally, the shallow feature extraction unit in S2 is a 3×3 convolutional layer.

[0014] Optionally, the denoising unit in S3 includes a plurality of Jacobian blocks.

[0015] The shallow feature image is input into the denoising unit of the image denoising network, and the denoised image obtained includes:

[0016] S31. Let k=1.

[0017] S32, input the shallow feature image y0 into the kth Jacobian block of the denoising unit to obtain the denoised image y k .

[0018] S33, the denoised image y k Input into the k+1th Jacobian block of the denoising unit to obtain the denoised image y k+1 .

[0019] S34, let k=k+1; repeat the above step S33 until the preset number of iterations is reached, and output the denoised image y k+1 .

[0020] Optionally, multiple Jacobian blocks are cascaded using short connections.

[0021] Optionally, in S33, the denoised image y k Input into the k+1th Jacobian block of the denoising unit to obtain the denoised image y k+1 include:

[0022] S331, the denoised image y k The multi-scale feature extraction module is input into the Jacobian block to obtain the multi-scale global features of the image.

[0023] S332, input the multi-scale global features into the convolution layer of the Jacobian block for feature fusion to obtain the predicted image after noise removal

[0024] S333, according to image y k ,image And the Jacobian solution module of the Jacobian block to obtain the denoised image y k+1 .

[0025] Optionally, the multi-scale feature extraction module in S331 includes three residual blocks with convolution kernel sizes of 3×3, 5×5 and 7×7 respectively.

[0026] Optionally, the denoised image y in S333 k+1 , as shown in the following formula (1):

[0027]

[0028] Among them, y k+1 is the image after denoising in the k+1th step; y k is the image after denoising in step k; is the mapping function.

[0029] Optionally, the step of inputting the denoised image into a dual attention unit of an image denoising network in S4 to obtain a final denoised image includes:

[0030] The denoised image is input into the dual attention unit of the image denoising network to obtain the enhanced image reconstruction features, and the enhanced image reconstruction features are input into the single-layer convolution unit to obtain the final denoised image.

[0031] Optionally, the dual attention unit includes a spatial attention extraction module and a channel attention extraction module.

[0032] The denoised image is input into the dual attention unit of the image denoising network, and the enhanced image reconstruction features obtained include:

[0033] The denoised image is input into the spatial attention extraction module to obtain the spatial attention weight.

[0034] The denoised image is input into the channel attention extraction module to obtain the channel attention weight.

[0035] The spatial attention weights and channel attention weights are fused, and the fused weights are used to weight the features of the denoised image to obtain enhanced image reconstruction features.

[0036] On the other hand, the present invention provides an image denoising device based on Jacobian dynamic approximation, which is used to implement an image denoising method based on Jacobian dynamic approximation, and the device includes:

[0037] The acquisition module is used to acquire the image to be denoised.

[0038] The extraction module is used to input the image to be denoised into the shallow feature extraction unit of the image denoising network to obtain a shallow feature image.

[0039] The denoising module is used to input the shallow feature image into the denoising unit of the image denoising network to obtain a denoised image.

[0040] The output module is used to input the denoised image into the dual attention unit of the image denoising network to obtain the final denoised image.

[0041] Optionally, the shallow feature extraction unit is a 3×3 convolutional layer.

[0042] Optionally, the denoising unit comprises a plurality of Jacobian blocks.

[0043] Optionally, the extraction module is further configured to:

[0044] S31. Let k=1.

[0045] S32, input the shallow feature image y0 into the kth Jacobian block of the denoising unit to obtain the denoised image y k .

[0046] S33, the denoised image y k Input into the k+1th Jacobian block of the denoising unit to obtain the denoised image y k+1 .

[0047] S34, let k=k+1; repeat the above step S33 until the preset number of iterations is reached, and output the denoised image y k+1 .

[0048] Optionally, multiple Jacobian blocks are cascaded using short connections.

[0049] Optionally, the denoising module is further configured to:

[0050] S331, the denoised image y k The multi-scale feature extraction module is input into the Jacobian block to obtain the multi-scale global features of the image.

[0051] S332, input the multi-scale global features into the convolution layer of the Jacobian block for feature fusion to obtain the predicted image after noise removal

[0052] S333, according to image y k ,image And the Jacobian solution module of the Jacobian block to obtain the denoised image y k+1 .

[0053] Optionally, the multi-scale feature extraction module includes three residual blocks with convolution kernel sizes of 3×3, 5×5 and 7×7 respectively.

[0054] Optionally, the denoised image y k+1 , as shown in the following formula (1):

[0055]

[0056] Among them, y k+1 is the image after denoising in the k+1th step; y k is the image after denoising in step k; is the mapping function.

[0057] Optionally, the output module is further configured to:

[0058] The denoised image is input into the dual attention unit of the image denoising network to obtain the enhanced image reconstruction features, and the enhanced image reconstruction features are input into the single-layer convolution unit to obtain the final denoised image.

[0059] Optionally, the dual attention unit includes a spatial attention extraction module and a channel attention extraction module.

[0060] Optionally, the output module is further configured to:

[0061] The denoised image is input into the spatial attention extraction module to obtain the spatial attention weight.

[0062] The denoised image is input into the channel attention extraction module to obtain the channel attention weight.

[0063] The spatial attention weights and channel attention weights are fused, and the fused weights are used to weight the features of the denoised image to obtain enhanced image reconstruction features.

[0064] On the one hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above-mentioned image denoising method based on Jacobian dynamic approximation.

[0065] On the one hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned image denoising method based on Jacobian dynamic approximation.

[0066] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0067] In the above scheme, firstly, for the complex noise existing in natural images, an image denoising strategy based on Jacobian dynamic approximation is proposed to gradually remove the noise in the image. Secondly, an end-to-end image denoising network based on Jacobian dynamic approximation is constructed, and an ordinary differential equation solving module based on Jacobian matrix is ​​designed to realize the forward derivation of the neural network. Through the multi-scale feature extraction module, the characteristics of the non-uniformly distributed noise in the image are obtained, and the dual attention structure is used to realize the enhancement of the reconstruction features. The image denoising method of the present invention can be successfully applied to the denoising of images with synthetic noise and real noise. Compared with the classical image denoising method, the method can achieve significant denoising performance improvement in both the subjective and objective quality of the reconstructed image.

[0068] The present invention regards the image denoising process as a forward differential dynamic system and designs an end-to-end image denoising network based on the ordinary differential equation solving strategy, namely the Jacobian dynamic approximation denoising network, which can better remove the synthetic noise and real noise in the noisy image and enhance the user's visual experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0070] Figure 1 It is a schematic diagram of the flow of an image denoising method based on Jacobian dynamic approximation provided by an embodiment of the present invention;

[0071] Figure 2 is a schematic diagram of an image denoising network provided by an embodiment of the present invention;

[0072] Figure 3 is a schematic diagram of a multi-scale feature extraction module provided by an embodiment of the present invention;

[0073] Figure 4 is a schematic diagram of a dual attention network structure provided by an embodiment of the present invention;

[0074] Figure 5 It is a sample diagram of denoising a Kodak24 grayscale image provided by an embodiment of the present invention;

[0075] Figure 6 is a sample diagram of denoising a BSD68 grayscale image provided by an embodiment of the present invention;

[0076] Figure 7 This is a sample diagram of denoising a Kodak24 color image provided by an embodiment of the present invention;

[0077] Figure 8 This is a sample diagram of denoising a BSD68 color image provided by an embodiment of the present invention;

[0078] Fig. 9 This is a sample diagram of real noise denoising in a SIDD data set provided by an embodiment of the present invention;

[0079] Fig.10 This is a sample diagram of real noise denoising in a DnD dataset provided by an embodiment of the present invention;

[0080] Fig.11 is a block diagram of an image denoising device based on Jacobian dynamic approximation provided by an embodiment of the present invention;

[0081] Fig.12 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0082] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0083] like Figure 1 As shown, an embodiment of the present invention provides an image denoising method based on Jacobian dynamic approximation, which can be implemented by an electronic device. Figure 1 The flowchart of the image denoising method based on Jacobian dynamic approximation is shown in FIG. 1 . The processing flow of the method may include the following steps:

[0084] S1. Obtain an image to be denoised.

[0085] S2. Input the image to be denoised into the shallow feature extraction unit of the image denoising network to obtain a shallow feature image.

[0086] Optionally, the shallow feature extraction unit in S2 is a 3×3 convolutional layer.

[0087] In a feasible implementation, a 3×3 convolutional layer is applied to extract shallow features of the image for subsequent feature learning.

[0088] S3. Input the shallow feature image into the denoising unit of the image denoising network to obtain the denoised image.

[0089] Optionally, the denoising unit in S3 includes a plurality of Jacobian blocks.

[0090] Optionally, multiple Jacobian blocks are cascaded using short connections.

[0091] In a feasible implementation, Figure 2 As shown in FIG. 1 , the image denoising network of Jacobian dynamic approximation constructs a deep network in the form of cascading multiple Jacobian blocks. Each Jacobian block acts as a solver of the dynamic system and dynamically fits the noise distribution in the image in the form of forward propagation.

[0092] The cascaded Jacobian block operation regards the general image denoising process as an iterative process of noise fitting, and each Jacobian block can be regarded as an iteration of fitting the complex noise in the image.

[0093] Shallow features use ODE (Ordinary Differential Equation) to realize dynamic modeling of image denoising process. The cascaded Jacobian blocks are used as solvers of ODE equations, where each Jacobian block is used to dynamically simulate noise. Two consecutive blocks are cascaded using short connections. In the kth Jacobian block, y k First, we get y through the multi-scale residual structure k+1 , then, respectively for y k and k+1 Perform nonlinear affine transformation and obtain f(y k ) and f(y k+1 ), the Jacobian solver is based on y k ,y k+1 、f(y k ) and f(y k+1 ) Solve for the Jacobian matrix for each iteration.

[0094] The Jacobian block is based on the premise that image denoising is considered as a dynamic process. Considering the complexity of noise, the denoising process is considered as: x = y + n. is a clear image containing N pixels, and n is the synthetic noise or real noise in the image. The mapping function is learned by iteration To approximate the noise n, the iterative process is shown in the following formula (1):

[0095]

[0096] Among them, y k represents the image after iterative denoising at the kth step, and y0 is the initial noisy image input.

[0097] The noise dynamic approximation process can be expressed by Euler discretization, as shown in the following equation (2):

[0098]

[0099] h is the step size of Euler discretization. It can be seen that the above formula is equivalent to the form of ODE equation, so the image dynamic denoising process can be regarded as a dynamic system of ODE. The general method can construct an integral solver based on the convolution layer, as shown in the following formula (3):

[0100]

[0101] However, the numerical integration in this solution process is computationally intensive and takes a long time to train, which limits the application of ODE networks. In addition, the noise perturbations in real images cannot be modeled using additive Gaussian white noise, and small changes in the input space also limit the learning efficiency of the model.

[0102] To solve the above problems, the present invention represents image denoising as an iterative denoising process, in which each denoising block can be regarded as a differential network. The cascaded Jacobian blocks are designed to form a dynamic network to fit the noise distribution of the image in a dynamic iterative manner.

[0103] Specifically, the shallow feature image is input into the denoising unit of the image denoising network, and the denoised image obtained includes:

[0104] S31. Let k=1.

[0105] S32, input the shallow feature image y0 into the kth Jacobian block of the denoising unit to obtain the denoised image y k .

[0106] S33, the denoised image y k Input into the k+1th Jacobian block of the denoising unit to obtain the denoised image y k+1 .

[0107] Optionally, in S33, the denoised image y k Input into the k+1th Jacobian block of the denoising unit to obtain the denoised image y k+1 include:

[0108] S331, the denoised image y k The multi-scale feature extraction module is input into the Jacobian block to obtain the multi-scale global features of the image.

[0109] Optionally, the multi-scale feature extraction module in S331 includes three residual blocks with convolution kernel sizes of 3×3, 5×5 and 7×7 respectively.

[0110] In a feasible implementation, Figure 3 As shown in the figure, a multi-scale residual structure is applied to obtain rich hierarchical features and realize multi-scale deep feature fusion; the non-uniformly distributed noise features in the image are mined through a multi-scale feature extraction module.

[0111] The residual structure consists of three residual blocks with convolution kernel sizes of 3×3, 5×5, and 7×7. Multi-scale residual blocks can expand the receptive field and better extract global features.

[0112] S332, input the multi-scale global features into the convolution layer of the Jacobian block for feature fusion to obtain the predicted image after noise removal

[0113] In a feasible implementation, the extracted multi-scale global features are fused through a convolutional layer to predict the image after noise removal, as shown in the following formula (4):

[0114]

[0115] Among them, y k is the image after denoising in step k; W represents the convolution kernel parameter; Represents the image after noise removal predicted by the 3×3 convolution kernel; Represents the image after noise removal predicted by the 5×5 convolution kernel; represents the image after noise removal predicted by the 7×7 convolution kernel; b represents the convolution kernel bias.

[0116] S333, according to image y k ,image And the Jacobian solution module of the Jacobian block to obtain the denoised image y k+1 .

[0117] In a feasible implementation, when y is knownk and Based on this, the Jacobian solution module is used to remove noise and obtain the image y after removing one step of noise. k+1 The denoising process includes the following specific steps:

[0118] 1. Each Jacobian block can be regarded as a denoiser. For the kth Jacobian block, y k+1 This is the “clear” image output by the kth Jacobian block. The denoising process can be viewed as the following equation (5):

[0119]

[0120] in, Represents y k Pixel I in represents the true clear pixel value of the i-th point, is the noise disturbance value of the i-th pixel, Represents a denoiser.

[0121] 2. Denoiser According to Taylor's formula, it can be expanded into the following formula (6):

[0122]

[0123] in, is an infinitesimal quantity.

[0124] The denoising process can be transformed into solving the first-order partial derivative of the image It can be further expressed in the form of Jacobian matrix, as shown in the following formula (7):

[0125]

[0126] However, the above formula requires that It needs to be differentiable everywhere, which is difficult to guarantee during the back propagation of the network. In order to realize the forward solution of the Jacobian matrix, according to the definition of the derivative, Transformed into the form of limit approximation, as shown in the following formula (8):

[0127]

[0128] Among them, y k+1 Predictable replace.

[0129] 3. To obtain and Use convolutional layer-ReLu-convolutional layer to transform y k andk+1 Processing to obtain and Finally, the denoising result of the kth Jacobian block can be expressed as follows (9):

[0130]

[0131] Among them, y k+1 is the image after denoising in the k+1th step; y k is the image after denoising in step k; is the mapping function.

[0132] S34, let k=k+1; repeat the above step S33 until the preset number of iterations is reached, and output the denoised image y k+1 .

[0133] S4. Input the denoised image into the dual attention unit of the image denoising network to obtain the final denoised image.

[0134] In a feasible implementation, Figure 4 As shown in the figure, the attention mechanism is applied to enhance the features, obtain the final denoised image, and improve the performance of image reconstruction.

[0135] Optionally, the step of inputting the denoised image into a dual attention unit of an image denoising network in S4 to obtain a final denoised image includes:

[0136] The denoised image is input into the dual attention unit of the image denoising network to obtain the enhanced image reconstruction features, and the enhanced image reconstruction features are input into the single-layer convolution unit to obtain the final denoised image.

[0137] Optionally, the dual attention unit includes a spatial attention extraction module and a channel attention extraction module.

[0138] The denoised image is input into the dual attention unit of the image denoising network, and the enhanced image reconstruction features obtained include:

[0139] The denoised image is input into the spatial attention extraction module to obtain the channel attention weights.

[0140] In a feasible implementation, a global pooling operation is performed on the input feature f in each feature channel, as shown in the following formula (10):

[0141]

[0142] Among them, w, h represent the dimension of the feature, f c (i, j) represents the eigenvalue of the cth channel of feature f with coordinates (i, j), z cRepresents the feature statistics of the cth channel. Apply convolution operation and Sigmoid activation function to extract channel attention weight, as shown in the following formula (11):

[0143] s c =σ(W2*(δ(W1*z c +b1))+b2) (11)

[0144] Among them, σ represents the Sigmoid function, W1 and W2 distributions represent different convolution kernel parameters, b1 and b2 represent different convolution kernel biases, and δ represents the ReLU function.

[0145] The denoised image is input into the channel attention extraction module to obtain the spatial attention weight.

[0146] In a feasible implementation, for the input feature f, the spatial attention is calculated by performing maximum pooling and average pooling operations along the channel dimension distribution, as shown in the following formula (12):

[0147] z s =[MaxPool(f),AvgPool(f)] (12)

[0148] Among them, [,] represents the cascade operation, MaxPool is the maximum pooling, AvgPool is the average pooling, z s is the spatial feature statistic. After the convolution operation and the Sigmoid function, the spatial attention weight is obtained as shown in the following formula (13):

[0149] s s =σ(W3*z s +b3) (13)

[0150] W3 represents the convolution kernel parameter, and b3 represents the convolution kernel bias.

[0151] The spatial attention weights and channel attention weights are fused, and the fused weights are used to weight the features of the denoised image to obtain enhanced image reconstruction features.

[0152] In a feasible implementation, the spatial attention weight and the channel attention weight are fused to weight the feature f as the reconstruction feature, as shown in the following formula (14):

[0153] f DAS =W4*[s c ·f,s s ·f]+b4 (14)

[0154] W4 represents the convolution kernel parameter, and b4 represents the convolution kernel bias.

[0155] In a feasible implementation, the network needs to be trained after the end-to-end network is constructed. The data set is set as follows: the training set used for synthetic image denoising is 800 high-definition images with a resolution of 2K in the DIV2K super-resolution reconstruction data set, and images synthesized by adding Gaussian white noise with different noise levels (expressed by the noise standard deviation σ), and random cropping is used to achieve data enhancement; the training set used for real image denoising is the SIDD Medium Dataset data set. In each training batch, 16 image blocks of size 64×64 are input for synthetic image denoising; 16 image blocks of size 128×128 are input for real image denoising. The present invention uses the ADAM algorithm to optimize the network parameters, setting β1=0.9, β2=0.999, ∈=10 -8 ; The initial learning rate is set to 10 -4 , in 1×10 4 The number of iterations is halved using 15 cascaded Jacobian blocks, implemented in PyTorch using a GeForce RTX 1080Ti GPU card.

[0156] The present invention uses RMSE (Root Mean Square Error) as the loss function in model training:

[0157]

[0158] In the testing phase, RMSE and PSNR (Peak Signal to Noise Ratio) are used as objective evaluation indicators, as shown in Table 1 the denoising results of Kodak24 and BSD68 (grayscale images), Table 2 the denoising results of Kodak24 and BSD68 (color images), Table 3 the objective comparison results on the SIDD dataset, and Table 4 the objective comparison results on the DnD dataset, where the optimal result is shown in bold and the suboptimal result is underlined.

[0159] Table 1

[0160]

[0161] Table 2

[0162]

[0163]

[0164] Table 3

[0165]

[0166] Table 4

[0167]

[0168] The test set of the synthetic noise denoising experiment uses the BSD68 and Kodak24 datasets, which both contain color images and grayscale images, and adds images composed of Gaussian white noise of different noise levels to the original clear images. The denoising results of JADN (Jacobian Approximation Denoising Network) and the comparison algorithm on grayscale images are shown in Table 1. On the Kodak24 dataset, the average PSNR value of JADN is 0.05dB (σ=30), 0.1dB (σ=50), and 0.11dB (σ=70) higher than that of RIDnet (Recursive Information Distillation Network); on the BSD68 dataset, the average PSNR value of JADN is 0.04dB (σ=30), 0.06dB (σ=50), and 0.08dB (σ=70) higher than that of RIDnet.

[0169] Table 2 shows the denoising comparison results of color images: on the Kodak24 dataset, JADN's average PSNR values ​​are 0.24dB (σ=30), 0.37dB (σ=50), and 0.31dB (σ=70) higher than RIDnet; on the BSD68 dataset, JADN's average PSNR values ​​are 0.21dB (σ=30), 0.23dB (σ=50), and 0.26dB (σ=70) higher than RIDnet. Obviously, JADN has the best denoising performance, especially in color images and high noise conditions.

[0170] The test results on the SIDD dataset are shown in Table 3: JADN's average PSNR value is 0.82dB higher than RIDNet and 0.33dB higher than VDN, and its denoising performance on real images is better than that on synthetic noisy images. MIRnet uses a network with a very large number of parameters, and its PSNR value is 0.19dB higher than JADN.

[0171] The objective results on the DnD dataset are shown in Table 4. Since real noise does not necessarily obey statistical distribution characteristics, non-blind denoising algorithms such as FFDNet+ and DnCNN+ perform poorly, while the PSNR value of JADN is 0.38dB higher than that of RIDnet, 0.05dB higher than that of SADnet (Spatial-Adaptive Network for Single Image Denoising), but 0.24dB lower than that of MIRnet.

[0172] Table 5 compares the time and space complexity. It compares the number of parameters of different algorithms. JADN uses fewer parameters than SADnet and DAnet (Dual Attention Network), but achieves better performance in real noise denoising experiments. MIRnet performs slightly better than JADN+, but the number of model parameters is 2.6 times that of JADN+ and 7.7 times that of JADN.

[0173] Table 5

[0174]

[0175]

[0176] Table 6 Ablation experiment (BSD68 dataset, σ=50).

[0177] Table 6

[0178]

[0179] In the embodiment of the present invention, firstly, for the complex noise existing in natural images, an image denoising strategy based on Jacobian dynamic approximation is proposed to gradually remove the noise in the image. Secondly, an end-to-end image denoising network based on Jacobian dynamic approximation is constructed, and an ordinary differential equation solving module based on Jacobian matrix is ​​designed to realize the forward derivation of the neural network. Through the multi-scale feature extraction module, the characteristics of the non-uniformly distributed noise in the image are obtained, and the dual attention structure is used to realize the enhancement of the reconstruction feature. The image denoising method of the present invention can be successfully applied to the denoising of images with synthetic noise and real noise. Compared with the classical image denoising method, the method can achieve significant denoising performance improvement in both the subjective and objective quality of the reconstructed image.

[0180] The present invention regards the image denoising process as a forward differential dynamic system and designs an end-to-end image denoising network based on the ordinary differential equation solving strategy, namely the Jacobian dynamic approximation denoising network, which can better remove the synthetic noise and real noise in the noisy image and enhance the user's visual experience.

[0181] like Fig.11 As shown, an embodiment of the present invention provides an image denoising device 1100 based on Jacobian dynamic approximation. The device 1100 is applied to implement an image denoising method based on Jacobian dynamic approximation. The device 1100 includes:

[0182] The acquisition module 1110 is used to acquire the image to be denoised.

[0183] The extraction module 1120 is used to input the image to be denoised into the shallow feature extraction unit of the image denoising network to obtain a shallow feature image.

[0184] The denoising module 1130 is used to input the shallow feature image into the denoising unit of the image denoising network to obtain a denoised image.

[0185] The output module 1140 is used to input the denoised image into the dual attention unit of the image denoising network to obtain the final denoised image.

[0186] Optionally, the shallow feature extraction unit is a 3×3 convolutional layer.

[0187] Optionally, the denoising unit comprises a plurality of Jacobian blocks.

[0188] Optionally, the extraction module 1120 is further configured to:

[0189] S31. Let k=1.

[0190] S32, input the shallow feature image y0 into the kth Jacobian block of the denoising unit to obtain the denoised image y k .

[0191] S33, the denoised image y k Input into the k+1th Jacobian block of the denoising unit to obtain the denoised image y k+1 .

[0192] S34, let k=k+1; repeat the above step S33 until the preset number of iterations is reached, and output the denoised image y k+1 .

[0193] Optionally, multiple Jacobian blocks are cascaded using short connections.

[0194] Optionally, the denoising module 1130 is further configured to:

[0195] S331, the denoised image y k The multi-scale feature extraction module is input into the Jacobian block to obtain the multi-scale global features of the image.

[0196] S332, input the multi-scale global features into the convolution layer of the Jacobian block for feature fusion to obtain the predicted image after noise removal

[0197] S333, according to image y k ,image And the Jacobian solution module of the Jacobian block to obtain the denoised image yk+1 .

[0198] Optionally, the multi-scale feature extraction module includes three residual blocks with convolution kernel sizes of 3×3, 5×5 and 7×7 respectively.

[0199] Optionally, the denoised image y k+1 , as shown in the following formula (1):

[0200]

[0201] Among them, y k+1 is the image after denoising in the k+1th step; y k is the image after denoising in step k; is the mapping function.

[0202] Optionally, the output module 1140 is further configured to:

[0203] The denoised image is input into the dual attention unit of the image denoising network to obtain the enhanced image reconstruction features, and the enhanced image reconstruction features are input into the single-layer convolution unit to obtain the final denoised image.

[0204] Optionally, the dual attention unit includes a spatial attention extraction module and a channel attention extraction module.

[0205] Optionally, the output module 1140 is further configured to:

[0206] The denoised image is input into the spatial attention extraction module to obtain the spatial attention weight.

[0207] The denoised image is input into the channel attention extraction module to obtain the channel attention weight.

[0208] The spatial attention weights and channel attention weights are fused, and the fused weights are used to weight the features of the denoised image to obtain enhanced image reconstruction features.

[0209] In the embodiment of the present invention, firstly, for the complex noise existing in natural images, an image denoising strategy based on Jacobian dynamic approximation is proposed to gradually remove the noise in the image. Secondly, an end-to-end image denoising network based on Jacobian dynamic approximation is constructed, and an ordinary differential equation solving module based on Jacobian matrix is ​​designed to realize the forward derivation of the neural network. Through the multi-scale feature extraction module, the characteristics of the non-uniformly distributed noise in the image are obtained, and the dual attention structure is used to realize the enhancement of the reconstruction feature. The image denoising method of the present invention can be successfully applied to the denoising of images with synthetic noise and real noise. Compared with the classical image denoising method, the method can achieve significant denoising performance improvement in both the subjective and objective quality of the reconstructed image.

[0210] The present invention regards the image denoising process as a forward differential dynamic system and designs an end-to-end image denoising network based on the ordinary differential equation solving strategy, namely the Jacobian dynamic approximation denoising network, which can better remove the synthetic noise and real noise in the noisy image and enhance the user's visual experience.

[0211] Fig.12 1 is a schematic diagram of the structure of an electronic device 1200 provided in an embodiment of the present invention. The electronic device 1200 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 1201 and one or more memories 1202, wherein the memory 1202 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 1201 to implement the following image denoising method based on Jacobian dynamic approximation:

[0212] S1. Obtain an image to be denoised.

[0213] S2. Input the image to be denoised into the shallow feature extraction unit of the image denoising network to obtain a shallow feature image.

[0214] S3. Input the shallow feature image into the denoising unit of the image denoising network to obtain the denoised image.

[0215] S4. Input the denoised image into the dual attention unit of the image denoising network to obtain the final denoised image.

[0216] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, which can be executed by a processor in a terminal to perform the above-mentioned image denoising method based on Jacobian dynamic approximation. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0217] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0218] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image denoising method based on Jacobian dynamic approximation, characterized in that: The method comprises: S1, obtaining the image to be denoised; S2, inputting the image to be denoised into a shallow feature extraction unit of an image denoising network to obtain a shallow feature image; S3, inputting the shallow feature image into a denoising unit of an image denoising network to obtain a denoised image; S4, inputting the denoised image into the dual attention unit of the image denoising network to obtain a final denoised image; The denoising unit in S3 includes a plurality of Jacobian blocks; The shallow feature image is input into the denoising unit of the image denoising network, and the denoised image is obtained including: S31, set k=1; S32, input the shallow feature image y0 into the kth Jacobian block of the denoising unit to obtain the denoised image y k ; S33, the denoised image y k Input into the k+1th Jacobian block of the denoising unit to obtain the denoised image y k+1 ; S34, let k=k+1; repeat the above step S33 until the preset number of iterations is reached, and output the denoised image y k+1 .

2. The method according to claim 1, characterized in that The shallow feature extraction unit in S2 is a 3×3 convolutional layer.

3. The method according to claim 1, characterized in that The multiple Jacobian blocks are cascaded in a short connection manner.

4. The method according to claim 1, characterized in that The denoised image y in S33 is k Input into the k+1th Jacobian block of the denoising unit to obtain the denoised image y k+1 include: S331, the denoised image y k Input to the multi-scale feature extraction module of Jacobian block to obtain the multi-scale global features of the image; S332: Input the multi-scale global features into the convolution layer of the Jacobian block for feature fusion to obtain a predicted image after noise removal. S333, according to the image y k ,image And the Jacobian solution module of the Jacobian block to obtain the denoised image y k+1 .

5. The method according to claim 4, characterized in that The multi-scale feature extraction module in S331 includes three residual blocks with convolution kernel sizes of 3×3, 5×5 and 7×7 respectively.

6. The method according to claim 4, characterized in that The denoised image y in S333 k+1 , as shown in the following formula (1): Among them, y k+1 is the image after denoising in the k+1th step; y k is the image after denoising in step k; is the mapping function.

7. The method according to claim 1, characterized in that The step of inputting the denoised image into the dual attention unit of the image denoising network in S4 to obtain the final denoised image includes: The denoised image is input into a dual attention unit of an image denoising network to obtain enhanced image reconstruction features, and the enhanced image reconstruction features are input into a single-layer convolution unit to obtain a final denoised image.

8. The method according to claim 7, characterized in that The dual attention unit includes a spatial attention extraction module and a channel attention extraction module; The step of inputting the denoised image into the dual attention unit of the image denoising network to obtain enhanced image reconstruction features comprises: Inputting the denoised image into a spatial attention extraction module to obtain a spatial attention weight; Inputting the denoised image into a channel attention extraction module to obtain a channel attention weight; The spatial attention weights and channel attention weights are fused, and the fused weights are used to weight the features of the denoised image to obtain enhanced image reconstruction features.

9. An image denoising device based on Jacobian dynamic approximation, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be denoised; An extraction module, used for inputting the image to be denoised into a shallow feature extraction unit of an image denoising network to obtain a shallow feature image; A denoising module, used for inputting the shallow feature image into a denoising unit of an image denoising network to obtain a denoised image; An output module, used for inputting the denoised image into a dual attention unit of an image denoising network to obtain a final denoised image; The denoising unit includes a plurality of Jacobian blocks; The shallow feature image is input into the denoising unit of the image denoising network, and the denoised image is obtained including: S31, set k=1; S32, input the shallow feature image y0 into the kth Jacobian block of the denoising unit to obtain the denoised image y k ; S33, the denoised image y k Input into the k+1th Jacobian block of the denoising unit to obtain the denoised image y k+1 ; S34, let k=k+1; repeat the above step S33 until the preset number of iterations is reached, and output the denoised image y k+1 .

Images