A network automatic search method for image denoising and an image denoising method

Abstract

The present invention discloses a network automatic search method for image denoising and an image denoising method, belonging to the technical field of image processing. When constructing a network structure for image denoising, the present application uses the Feature module as the first layer of the CNN network structure to ensure that the input image has sufficient features for the deep modules to learn. At the same time, the Transition module transforms the feature dimensions of the feature map, and the Dropout module prevents overfitting of the network, thereby obtaining a better Gaussian denoising effect. In addition, the present application adopts an automatic structure search method based on a genetic algorithm, which can automatically adjust the automatically designed network structure according to different noise level problems in the data set, avoiding the time cost problem of manual intervention and greatly improving the speed and performance of processing the denoising task.

Description

Technical Field

[0001] The present invention relates to a network automatic search method and an image denoising method for image denoising, belonging to the technical field of image processing. Background Art

[0002] In recent years, Convolutional Neural Networks (CNNs) have achieved great success in processing computer vision tasks and have become the mainstream method for solving computer vision problems such as image classification, object detection, and object tracking. From initially stacking network layers to make the network deeper, to later residual structures and dense connection structures that fuse features of different depth network layers, and then to structures that fuse width feature of the network, such as Figure 1 Figures 2, 3, and 4 respectively show the residual network structure ResNet, the dense connection network structure DenseNet, and the GoogleNet (Inception) structure that fuses width features. Obviously, the CNN network structure has become the key to affecting the performance of the CNN model.

[0003] Image denoising refers to the process of reducing noise in digital images. In reality, digital images are often affected by imaging device and external environmental noise interference during the digitization and transmission processes, and are called noisy images or images with noise. The presence of noise will affect subsequent further processing of the image, such as image classification, object recognition, and tracking, etc.

[0004] For the image denoising task, CNN networks for the image denoising task have been proposed, including the BM3D algorithm and the WNNM algorithm based on the non - linear similarity principle, the EPLL algorithm based on the generative model, the MLP algorithm, the CSF algorithm, the TNRD algorithm, and the DnCNN algorithm based on the independent training principle. However, the traditional manually designed CNN network structure for image denoising not only requires a large amount of CNN - related professional knowledge and rich experience, but also different CNN network structures need to be designed for different data sets, which greatly limits the development of the CNN structure for image denoising.

[0005] To solve the above problems, algorithms for automatically designing CNN network structures have been proposed. These algorithms can automatically search for CNN structures with superior performance without the need for human knowledge and experience. These algorithms can be roughly divided into two categories: semi-automatic CNN structure search algorithms represented by algorithms such as Genetic CNN, Hierarchical Evolution, EAS, Block-QNN-S, etc., and fully automatic CNN structure search algorithms represented by Large-scale Evolution, CGP-CNN, NAS, MetaQNN, IPPSO, AE-CNN, etc. It is worth mentioning that both of these two types of algorithms are based on evolutionary computing algorithms or reinforcement learning algorithms. For example, Genetic CNN, Hierarchical Evolution, CGP-CNN, Large-scale Evolution, AE-CNN, and IIPSO are based on evolutionary computing algorithms, and NSA, MetaQN, EAS, and Block-QNN-S are based on reinforcement learning algorithms. However, these algorithms are all used to solve image classification tasks. The last module of the network structure corresponding to the image classification task is a classifier, and the output is one-dimensional data representing the image classification result; while for image denoising, the last part of its corresponding network structure is a convolutional module, and the output result is two-dimensional data representing the noise of the image. Therefore, the existing automatic search methods for network structures for image classification cannot be directly used to solve the problem of image denoising.

[0006] Moreover, for image denoising, especially the design of CNN structures for Gaussian denoising tasks belongs to an optimization problem. By optimizing the network model M, the loss function L(M, T) is minimized on the training dataset T. The CNN model M can be represented by the network internal structure encoding vector λ and is learned from a specific learning algorithm A on the training set T, that is, M = A(λ, T). Then the goal of CNN structure design is to find an optimal network structure solution λ * , such that M * = A(λ * , T), and satisfies the minimum of L(M * , V) on the validation set V. It is represented by the following mathematical formula:

[0007] λ * = arg λ min L(T, M) = arg λ min f(λ, A, T, V, L) (1)

[0008] The objective function f receives the vector λ representing the network structure encoding as input and outputs the corresponding loss value. The training set T and the validation set V are divided from the training set according to a certain ratio and satisfy By minimizing the loss value of the objective function f, the optimal λ can be obtained, and then the network model M can be constructed. Finally, the generalization ability of M is verified on the test set. In theory, algorithms such as gradient descent can be used to optimize the network structure λ, but it is very difficult to achieve in practice for the following reasons:

[0009] 1. Optimizing the loss value of the objective function f essentially involves finding a specific λ, which belongs to a combinatorial optimization problem. That is, it is necessary to calculate the noise reduction index values of all network structures, and the computational cost is relatively high.

[0010] 2. The structure vector λ representing the network model M is a discrete encoding and cannot be processed by traditional methods for handling continuous functions. Summary of the Invention

[0011] To better solve the image noise reduction problem, the present invention provides a network automatic search method for image noise reduction and an image noise reduction method. Aiming at the problem that the traditional network structure for processing Gaussian noise reduction tasks based on CNN is just a simple linear structure, which greatly limits the Gaussian noise reduction effect, this application proposes a deep structure based on ResNet and DenseNet, as well as an Inception width structure. At the same time, a Feature feature extraction module, a Transition feature layer transformation module, and a Dropout noise reduction module are introduced, making full use of the depth and width structures of the CNN network structure, and greatly improving the effect of the CNN network in processing Gaussian noise reduction tasks.

[0012] A network automatic search method for image noise reduction, the method comprising:

[0013] Step 1. Perform variable-length linear encoding on the CNN structure based on the ResNet module, DenseNet module, Inception module, Feature module, Transition module, and Dropout module to obtain several CNN networks with different structures. The obtained several CNN networks with different structures are respectively used as individuals in the initial population P0. The first layer of each CNN network individual is a Feature module, and the Transition module is a superposition of Conv + Rlu + BN layers;

[0014] Step 2. Divide the publicly available image noise reduction data set according to a predetermined ratio, and calculate the fitness value of each CNN network individual in the population on the divided data set;

[0015] Step 3. According to the fitness value of each CNN network individual, perform crossover and mutation operations to generate offspring;

[0016] Step 4. The CNN network individuals of the offspring and the parents generate a new population P through natural selection;

[0017] Step 5. Repeat Steps 2 - 4 until the termination condition is met. The finally obtained CNN network individual is the network structure searched for image denoising.

[0018] Optionally, calculating the fitness value of each CNN network individual in the population on the partitioned dataset in Step 2 includes:

[0019] Taking the peak signal - to - noise ratio (PSNR) obtained by training each CNN network individual on the partitioned dataset for 30 epochs as the fitness value of the corresponding CNN network individual.

[0020] Optionally, the Feature module includes two branches, namely ConvRelu1, ConvRelu2, and ConvRelu3; ConvRelu3 corresponds to a convolution kernel size of 3*3, padding of 1, stride of 1, the number of input channels is c, and the number of output channels is 32.

[0021] Optionally, the Transition module adopts Conv, BN, and ReLU operations, where the convolution kernel size of Conv is 1*1, padding is 0, and stride is 1.

[0022] Optionally, the image denoising dataset disclosed in Step 2 is the Set12 and BM3D datasets.

[0023] Optionally, the predetermined ratio is 1:5.

[0024] Optionally, before calculating the fitness value of each CNN network individual in the population on the partitioned dataset, it further includes partitioning the test set and the training set at a ratio of 1:4 on the partitioned dataset.

[0025] This application also provides an image denoising method, which uses the network structure for image denoising searched by the above - mentioned method to perform image denoising processing.

[0026] The beneficial effects of the present invention are:

[0027] By introducing the Resnet and DenseNet depth modules, the Inception width module, the Feature feature learning module, the Dropout module, and the Transition module, various structural information of the CNN can be fully utilized to better solve the Gaussian noise reduction problem. When constructing the network structure for image noise reduction in this application, the Feature module is used as the first layer of the CNN network structure to ensure that the input image has sufficient features for the deeper modules to learn. At the same time, the Transition module transforms the feature dimensions of the feature map, and the Dropout module prevents overfitting of the network, thereby obtaining a better Gaussian noise reduction effect. In addition, this application adopts an automatic structure search method based on genetic algorithms, which can automatically adjust the automatically designed network structure according to different noise level problems in the dataset, avoiding the time cost problem of manual intervention and greatly improving the speed and performance of processing the noise reduction task. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0029] Figure 1 It is a schematic diagram of the ResNet structure of the residual network in the CNN network.

[0030] Figure 2 It is a schematic diagram of the DenseNet structure of the densely connected network in the CNN network.

[0031] Figure 3 It is a schematic diagram of the Inception structure of the width network in the CNN network.

[0032] Figure 4 It is a schematic diagram of the linear CNN structure encoding structure based on the depth and width modules of this application.

[0033] Figure 5 It is a schematic diagram of the Feature structure in this application.

[0034] Figure 6 It is a schematic diagram of the CNN network structure automatically designed by the network automatic search method FBE-CNN provided in an embodiment of this application on the Set12 and BSD68 datasets with a noise level of 15.

[0035] Figure 7It is a CNN network structure automatically designed by the network automatic search method FBE-CNN provided in an embodiment of the present application on the Set12 and BSD68 data sets with a noise level of 25.

[0036] Figure 8 It is a CNN network structure automatically designed by the network automatic search method FBE-CNN provided in an embodiment of the present application on the Set12 and BSD68 data sets with a noise level of 50. Detailed implementation manners

[0037] To make the objectives, technical solutions and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0038] Embodiment 1:

[0039] This embodiment provides a network automatic search method for image denoising, and the method includes:

[0040] Step 1. Based on the ResNet module, DenseNet module, Inception module, Feature module, Transition module and Dropout module, perform variable-length linear coding on the CNN structure to obtain several different structures of CNN networks. The obtained several different structures of CNN networks are respectively used as individuals in the initial population P0. The first layer of each CNN network is a Feature module, and the Transition module is a superposition of a convolutional layer + Relu activation function + batch normalization layer.

[0041] Step 2. Divide the publicly available image denoising data set according to a predetermined ratio, and calculate the fitness value of each CNN network individual in the population on the divided data set; use the peak signal-to-noise ratio PSNR obtained by training each CNN network individual for 30 epochs on the divided data set as the fitness value of the corresponding CNN network individual;

[0042] Step 3. Perform crossover and mutation operations according to the fitness value of each CNN network individual to generate offspring;

[0043] Step 4. The CNN individuals of the offspring and the parent generation generate a new population P through natural selection;

[0044] Step 5. Repeat steps 2-4 until the termination condition is met.

[0045] Embodiment 2

[0046] This embodiment provides an image denoising method, which uses the network automatic search method given in Embodiment 1 to search for a network structure for image denoising, and the method includes:

[0047] Step 1. Based on the ResNet module, DenseNet module, Inception module, Feature module, Transition module, and Dropout module, perform variable-length linear encoding on the CNN structure to obtain several CNN networks with different structures. Take the obtained several CNN networks with different structures as individuals in the initial population P0. The first layer of each CNN network is the Feature module, and the Transition module is a superposition of Conv+Rlu+BN layers.

[0048] Step 2.1: Divide the Gaussian denoising dataset Set12 and BM3D dataset at a ratio of 5. On the divided dataset, divide the test set and training set at a ratio of 1:4. Train for 30 epochs on the divided dataset to obtain the peak signal-to-noise ratio PSNR of each CNN individual on the divided dataset as the fitness of the individual.

[0049] Step 3. According to the fitness values of each CNN network individual, perform crossover and mutation operations to generate offspring.

[0050] Step 4. The CNN individuals of the offspring and parents generate a new population P through natural selection.

[0051] Step 5. Repeat Steps 2-4 until the termination condition is met.

[0052] The above method uses deep modules such as ResNet and DenseNet, introduces the Inception width module, and at the same time proposes the Feature module as the first layer of the CNN network structure to ensure that the input image has sufficient features for the deep modules to learn. The Transition module transforms the feature dimension of the feature map, and the Dropout module prevents overfitting of the network. Among them, the structures of the ResNet module, DenseNet module, and Inception module are Figure 1 , 2, 3 are similar, as Figure 5 shows the Feature module used in the present invention. The Transition module is a superposition of Conv+Rlu+BN layers. The present invention performs linear encoding based on the above modules and connects them into various different CNN network individuals, as Figure 4 shown.

[0053] To prove the effectiveness of the present invention, this embodiment conducts experimental verification on the Gaussian denoising dataset Set12 and BSD68 dataset, and at the same time compares with current mainstream Gaussian denoising methods, such as the BM3D algorithm and WNNM algorithm based on the non-linear similarity principle, the EPLL algorithm based on the generative model, the MLP algorithm, CSF algorithm, TNRD algorithm, and DnCNN algorithm based on the independent training principle.

[0054] In the current mainstream Gaussian denoising methods mentioned above, for the introduction of the BM3D algorithm based on the non-linear similarity principle, please refer to the introduction in "K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse3-d transform-domain collaborative filtering,” IEEE Transactions on Image Processing, vol. 16, no. 8, pp. 2080–2095, 2007.";

[0055] For the introduction of the WNNM algorithm, please refer to the introduction in "S. Gu, L. Zhang, W. Zuo, and X. Feng, “Weighted nuclear norm minimization with application to image denoising,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2862–2869.";

[0056] For the introduction of the EPLL algorithm based on the generative model, please refer to the introduction in "D. Zoran and Y. Weiss, “From learning models of natural image patches to whole image restoration,” in 2011 International Conference on Computer Vision, 2011, pp. 479–486.";

[0057] For the introduction of the MLP algorithm based on the independent training principle, please refer to the introduction in "H. C. Burger, C. J. Schuler, and S. Harmeling, “Image denoising: Can plain neural networks compete with bm3d?” in 2012 IEEE Conference on Computer Vision and Pattern Recognition, 2012, pp. 2392–2399.";

[0058] For the introductions of the CSF algorithm and the TNRD algorithm, please refer to the introductions in "U. Schmidt and S. Roth, “Shrinkage fields for effective image restoration,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2774–2781." and "Y. Chen and T. Pock, “Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, pp. 1256–1272, 2017.";

[0059] For the introduction of the DnCNN algorithm, please refer to the introduction in "K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising,” IEEE Transactions on Image Processing, vol. 26, no. 7, pp. 3142–3155, 2017."

[0060] The experimental results on the Gaussian denoising dataset Set12 are shown in Table 1 below:

[0061] Table 1 Comparison of the effects of FBE-CNN and traditional Gaussian denoising methods on Set12

[0062] BM3D WNNM EPLL MLP CSF TNRD DnCNN FBE-CNN θ = 15 32.372 32.696 32.138 – 32.318 32.502 32.839 32.49 θ = 25 29.969 30.257 29.692 30.027 29.837 30.055 30.378 30.28 θ = 50 26.722 27.052 26.471 26.783 – 26.812 27.165 27.19

[0063] In Table 1, θ represents the noise level. The larger the value of θ, the more difficult the denoising task. Three noise levels are adopted on the Set12 dataset, namely 15, 25, and 30. The corresponding values in each algorithm are the percentage values of the average noise ratio of PSNR at each noise level. The higher the PSNR value, the better the effect of processing the Gaussian denoising task. The present invention is an automatic structure search method based on a genetic algorithm, simply referred to as the FBE-CNN algorithm.

[0064] As can be seen from Table 1, the PSNR values of the present invention at the three noise levels are 32.9%, 30.28%, and 27.19% respectively.

[0065] Under the condition of a noise level of 15, the effect of the present invention is better than that of the BM3D method by 32.372%, the EPLL by 32.138%, and the TNRD by 32.502%.

[0066] At a noise level of 25, the effect of the present invention is better than that of the BM3D method by 29.969%, the WNNM method by 30.297%, the EPLL method by 29.629%, the MLP by 30.027%, the CSF by 29.837%, and the TNRD by 30.055%.

[0067] At a noise level of 50, the present invention achieves better results than all algorithms. Although at noise levels of 15 and 25, the effect of the present invention is not as good as that of DnCNN, at a noise level of 50, it is better than DnCNN. This shows that due to the adoption of a structure more complex than the traditional linear CNN structure in the present invention, which uses the depth modules of CNN, ResNet and DenseNet, as well as the width Inception module, it can automatically design a more complex structure and better solve the Gaussian noise reduction problem.

[0068] Table 2 shows the performance of the present invention on the Gaussian noise reduction dataset BSD68:

[0069] Table 2 Comparison of the effects of FBE-CNN and traditional Gaussian noise reduction methods on Set12

[0070] BM3D WNNM EPLL MLP CSF TNRD DnCNN FBE-CNN θ = 15 31.07 31.37 31.21 – 31.24 31.42 31.73 31.56 θ = 25 28.57 28.83 28.68 28.96 28.74 28.92 28.68 28.95 θ = 50 25.62 25.87 25.67 26.03 – 25.97 26.02 26.09

[0071] As shown in Table 2, the PSNR values of the FBE-CNN of the present invention at noise levels of 15, 20, and 50 are 31.56%, 28.95%, and 26.09% respectively.

[0072] At a noise level of 15, although the noise reduction effect of the present invention is not as good as that of DnCNN, it is better than that of the BM3D by 31.07%, the WNNM by 31.37%, the EPLL by 31.21%, the CSF by 31.24%, and the TNRD by 31.42%.

[0073] At noise levels of 25 and 50, the noise reduction effects achieved by the present invention are the best respectively.

[0074] Similar to the experimental results on the Set12 dataset, due to the adoption of ResNet and DenseNet depth modules, Inception width module, Feature feature extraction module, Transition feature conversion module, and Dropout overfitting prevention module in the present invention, it has a more complex structure compared to the traditional CNN noise reduction network based on simple linear modules. When facing difficult Gaussian noise reduction tasks, it can often achieve very good results. At the same time, the Gaussian noise reduction task on the BSD68 noise reduction dataset is more difficult than the Set12 task. Therefore, the present invention can achieve the best results at noise levels of 25 and 50, which also demonstrates the superiority of the present invention in solving complex Gaussian noise reduction problems.

[0075] Meanwhile, since the present invention adopts an automatic structure design algorithm based on genetic algorithms, for the Set12 and BSD68 datasets, the present invention can automatically adjust the automatically designed network structure according to different noise level problems in the datasets, avoiding the time cost problem of human intervention. The CNN Gaussian network structures automatically designed on the Set12 and BSD68 datasets are as Figure 6 , Figures 7 and 8 show that the CNN network structures designed at different noise levels are different, which indicates that the present invention can adaptively design the optimal CNN network structure for noise reduction tasks with different datasets and different noise levels without the intervention of human factors, greatly improving the speed and performance of processing noise reduction tasks.

[0076] Some steps in the embodiments of the present invention can be implemented by software, and the corresponding software program can be stored in a readable storage medium, such as an optical disc or a hard disk, etc.

[0077] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A network automatic search method for image denoising, characterized in that, The method includes: Step 1. Based on the ResNet module, DenseNet module, Inception module, Feature module, Transition module, and Dropout module, perform variable-length linear encoding on the CNN structure to obtain several CNN networks with different structures. Take the obtained several CNN networks with different structures as individuals in the initial population P0 respectively. The first layer of each CNN network individual is the Feature module, and the Transition module is a superposition of Conv+Rlu+BN layers; Step 2. Divide the publicly available image denoising dataset according to a predetermined ratio, and calculate the fitness value of each CNN network individual in the population on the divided dataset; Step 3. According to the fitness value of each CNN network individual, perform crossover and mutation operations to generate offspring; Step 4. The CNN network individuals of the offspring and the parent generation generate a new population P through natural selection; Step 5. Repeat steps 2-4 until the termination condition is met. The finally obtained CNN network individual is the searched network structure for image denoising; The Feature module includes two branches, namely ConvRelu1, ConvRelu2, and ConvRelu3; ConvRelu3 corresponds to a convolution kernel size of 3*3, padding of 1, stride of 1, output channels of c, and output channel Output Channel of 32; The Transition module adopts Conv, BN, and ReLU operations, where the convolution kernel size of Conv is 1*1, padding of 0, and stride of 1; The ResNet module is a residual structure; the DenseNet module is a densely connected structure; the Inception module is a width structure; the Transition module transforms the feature dimension of the feature map, and the Dropout module is used to prevent overfitting of the network.

2. The method according to claim 1, characterized in that, In step 2, calculating the fitness value of each CNN network individual in the population on the divided dataset includes: Taking the peak signal-to-noise ratio PSNR obtained by training each CNN network individual on the divided dataset for 30 epochs as the fitness value of the corresponding CNN network individual.

3. The method according to claim 2, characterized in that The publicly available image denoising dataset in step 2 is the Set12 and BM3D datasets.

4. The method according to claim 3, characterized in that, The predetermined ratio is 1:

5.

5. The method according to claim 4, wherein Before calculating the fitness value of each CNN network individual in the population on the divided dataset, it also includes dividing the test set and the training set according to a ratio of 1:4 on the divided dataset.

6. An image noise reduction method, characterized in that, The method adopts the network structure for image denoising searched by the method described in any one of claims 1-5.

Images