An image denoising method based on deep learning and image color space characteristics

Abstract

The present invention provides an image denoising method based on deep learning and image color space characteristics, relating to the field of image processing technology. Taking into account the imaging principles of images, the present invention utilizes the effect of noise on different color channels to perform preliminary image processing to enhance more image information and assist in network training. The resulting algorithm can denoise images containing both real noise and artificially synthesized noise while preserving image detail information. As a result, the network designed by the present invention can be trained to produce more universal denoising image results, improving the simplicity and accuracy of image denoising and having practical application value.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image denoising method based on deep learning and image color space characteristics. Background Art

[0002] Image denoising involves processing low-quality, noisy images to remove them, restore image details, and improve image quality. This is a low-level computer vision problem within the field of digital image processing, primarily addressing issues such as poor image quality and poor visual perception caused by limitations of imaging devices and network transmission losses. Before the widespread adoption of deep learning technology, many traditional image denoising algorithms, such as mean filtering, median filtering, Gaussian filtering, Fourier transforms, and wavelet transforms, all filtered noisy images in the spatial or transform domain to correct pixel values and generate a denoised image.

[0003] With the advancement and application of deep learning technology in computer vision, its application to image denoising has become a new approach. Compared to traditional image algorithms, deep learning algorithms learn the function that maps noisy images to clean images during training, achieving results that surpass those of traditional methods. Since their widespread application in the imaging field, convolutional neural networks have achieved significant success in high-level image processing tasks due to their efficient feature extraction capabilities. While convolutional neural networks have achieved excellent performance in image denoising, this model has also demonstrated their superior performance in detail recovery. Therefore, further research is needed on how to leverage deep learning to address image denoising.

[0004] Most traditional denoising algorithms rely on Gaussian filtering or wavelet transforms. These algorithms are not universally applicable to image denoising, are slow, and cannot flexibly handle images with varying noise levels, limiting their practical application. Furthermore, most existing deep learning-based algorithms denoise images containing artificially synthesized additive Gaussian white noise. Research on denoising real noisy images is limited. However, real-world noise is precisely the type of noise that needs to be addressed, as it is prevalent in real life. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention provides an image denoising method based on deep learning and the color space characteristics of images. Taking into account the imaging principles of images and leveraging the effect of noise on different color channels, the image is initially processed to enhance more image information and assist in network training. The resulting algorithm can denoise images containing both real and synthetic noise while preserving image detail. As a result, the network designed in the present invention can be trained to produce more universal denoised image results, improving the simplicity and accuracy of image denoising and possessing practical application value.

[0006] An image denoising method based on deep learning and image color space characteristics includes the following steps:

[0007] Step 1: Read the dataset and perform data augmentation on the noisy image;

[0008] Read noisy images from a public dataset of real and synthetic noise, randomly crop the noisy images into blocks of the same size and at the same position, then rotate the images by 90 degrees, 180 degrees, 270 degrees, 90 degrees and flip, 180 degrees and flip, 270 degrees and flip, horizontal flip, and vertical flip, with a probability of 50%.

[0009] Step 2: Build a network model;

[0010] The network model includes: CPBlock image preprocessing operation module, which provides more feature information for the processed image; Conv is a convolution layer with a convolution kernel of 3*3, which is used to change the number of channels of the image and generate feature maps for network training; BasicBlock is a basic module for extracting deep features;

[0011] The noise image block processed in step 1 is input into the network model. It is first preprocessed by CPBlock. The image generated by the preprocessing and the noise image block processed in step 1 are spliced between channels. After the channels are changed from 3 to 6, it passes through the first convolution layer to become 64 channels. A total of four BasicBlock modules are used for deep feature extraction. The number of channels after output is still 64. After the last convolution layer changes the 64 channels to 3 channels, it is added to the noise image block processed in step 1 to finally generate the image prediction after network denoising.

[0012] Step 3: Preprocess the input noisy image;

[0013] First, convert the noise image block processed in step 1 into HSV mode using the OpenCV library; perform Gaussian blur on the image, and then divide the image into several regions according to the value range of each color of H, S, and V; sum the values of the R, G, and B channels in each divided color region and calculate the average value; sort the average values of the three channels of R, G, and B, and use the gradients of the channels with the first and second average values for reference for the channel with the third average value, as shown in the following formula:

[0014]

[0015] For the channel with the second highest mean, the gradient of the channel with the highest mean is used as a reference, and the formula is as follows:

[0016]

[0017] Among them, the subscript low is the channel with the lowest mean, middle is the channel with the middle mean, high is the channel with the highest mean, and newValue low and newValue middle is the updated pixel value of the corresponding channel, mean low 、mean middle 、mean high Indicates the mean of the pixel values in the corresponding channel, value low 、value middle 、value high is the original pixel value of the corresponding channel;

[0018] The channel with the highest mean value is not modified. The above operation is performed on each divided color region, and finally a tensor with different values from the noise image but the same size is generated. This tensor is then concatenated with the noise image block processed in step 1 to form a 6-channel feature map, obtaining the preprocessed result.

[0019] Step 4: Exchange information between channels through the convolutional layer;

[0020] The preprocessed results are input into the first convolutional layer with a convolution kernel size of 3*3, which changes the channel from 6 to 64, so as to further exchange information between channels and extract shallow features.

[0021] Step 5: Extract deep features through the basic module to improve network performance;

[0022] The basic module has two branches along the network transmission direction. The first branch is used to extract global features, which consists of a convolution layer, an average pooling layer, a convolution layer, a ReLU activation layer, and a convolution layer. The second convolution layer is a 1*1 convolution to improve training speed and reduce computational consumption. The second branch is used to extract local features, which consists of a convolution layer, a ReLU activation layer, and a convolution layer. The output results of the two branches are multiplied, and the output is the sum of the input feature map and the product feature map.

[0023] Step 6: Set the number of times to repeat the basic module to learn deeper features;

[0024] Step 7: After the convolution layer converts the 64-channel image into a 3-channel image, it is added to the noise image block processed in step 1 to obtain the denoised image, completing the training;

[0025] Step 8: Calculate the loss function for the denoised image generated in step 7 and the corresponding noise-free clean image in the training set. The loss function is Among them, δ1 is L1Loss, Clean is a clean image without noise, For the denoised image generated in step 7, the selected optimization function is Adam, the parameters are the default parameters, and the initial value of the learning rate is 3*10 -5 .

[0026] The beneficial effects of adopting the above technical solution are:

[0027] The present invention provides an image denoising method based on deep learning and image color space characteristics, which can effectively remove real noise and artificially synthesized noise, reduce the damage to image details in the denoising process, and retain a certain amount of detail information; compared with existing methods, the model is simpler, the training consumes less computing resources, and it can denoise real noisy images, and the results are more robust and perform better. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is an overall flow chart of the image denoising method in an embodiment of the present invention;

[0029] Figure 2 This is a flowchart of image preprocessing in an embodiment of the present invention;

[0030] Figure 3 This is a network structure diagram in an embodiment of the present invention;

[0031] Figure 4 This is a graph showing the result of artificial noise denoising in an embodiment of the present invention;

[0032] Figure 5 This is a diagram of the real noise denoising result in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0034] Two datasets are used in this example. The first is the Smartphone Image Denoising Dataset (SIDD), a real image dataset publicly available on the Internet. The second is the Berkeley Segmentation Dataset (BSD500), an artificially synthesized noise dataset with artificially added additive white Gaussian noise. The noise levels are 5, 10, and 25.

[0035] An image denoising method based on deep learning and image color space characteristics, such as Figure 1 Shown is the main flow chart of the present invention, comprising the following steps:

[0036] Step 1: Read the dataset and perform data augmentation on the noisy image;

[0037] A total of 920 noisy images from a public dataset of real and synthetic noise were read. The noisy images were randomly cropped into image blocks of the same size of 48×48, and the image blocks were cropped at the same position as the clean images. The images were then rotated for data augmentation. The operations of rotating 90 degrees, rotating 180 degrees, rotating 270 degrees, rotating 90 degrees and flipping, rotating 180 degrees and flipping, rotating 270 degrees and flipping, horizontal flipping, and vertical flipping were selected with a probability of 50%. Before each round of training, the 920 images were randomly cropped and rotated 20 times to increase the number of noisy images during training and perform data augmentation.

[0038] Step 2: Build a network model;

[0039] Establish a network model, the overall network structure is as follows Figure 3 As shown in the figure. x is the noisy image block processed in step 1; CPBlock is the image preprocessing module. The processed image provides more feature information. The specific operation is shown in step 3. Conv is the convolution layer with a 3*3 convolution kernel. It is used to change the number of channels of the image and generate feature maps for network training. BasicBlock is the basic module used to extract deep features. The module details will be detailed in step 5. Predict is the generated denoised image.

[0040] The noise image block x processed in step 1 is input into the network model. It is first preprocessed by CPBlock. The image generated by the preprocessing and the noise image block x processed in step 1 are spliced between channels. After the channel is changed from 3 to 6, it passes through the first convolution layer to become 64 channels. A total of four BasicBlock modules are used for deep feature extraction. The number of channels after output is still 64. After the last convolution layer changes the 64 channels to 3 channels, it is added to the noise image block x processed in step 1 to finally generate the image predict after network denoising.

[0041] Step 3: Preprocess the input noisy image;

[0042] like Figure 2 As shown, the noise image initially read by the network is in RGB mode, while the HSV format can well divide the image into ten colors: black, white, gray, red, orange, yellow, green, cyan, blue, and purple. In order to facilitate the image color region segmentation, the noise image block processed in step 1 is first converted to HSV mode using the OpenCV library; at the same time, in order to reduce the impact of noise on the segmented area, the image is Gaussian blurred 5*5, and then the image is divided into several areas according to the value range of each color of H, S, and V; the values of the three channels R, G, and B in each divided color area are summed and the average value is calculated; it can be considered that the color channel with a higher mean value is less affected by noise. The average values of the three channels R, G, and B are sorted, and for the channel with the third highest average value, the gradients of the channels with the first and second highest average values are used as a reference to add more information. The formula is as follows:

[0043]

[0044] For the channel with the second highest mean, the gradient of the channel with the highest mean is used as a reference, and the formula is as follows:

[0045]

[0046] Among them, the subscript low is the channel with the lowest mean, middle is the channel with the middle mean, high is the channel with the highest mean, and newValue low and newValue middle is the updated pixel value of the corresponding channel, mean low 、mean middle 、mean high Indicates the mean of the pixel values in the corresponding channel, value low 、value middle 、value high is the original pixel value of the corresponding channel;

[0047] For the channel with the highest mean value, there is no change range in other channels for reference, so its value is not modified. For each divided color area, the above operation is performed, and finally a tensor with different values from the noise image but the same size is generated. This tensor is spliced with the noise image block processed in step 1 to form a 6-channel feature map to obtain the preprocessed result.

[0048] Step 4: Exchange information between channels through the convolutional layer;

[0049] The preprocessed results are input into the first convolutional layer with a convolution kernel size of 3*3, which changes the channel from 6 to 64, so as to further exchange information between channels and extract shallow features.

[0050] Step 5: Extract deep features through the basic module to improve network performance;

[0051] like Figure 1 As shown in the BasicBlock in the figure, the basic module has two branches along the network transmission direction. The first branch is used to extract global features, which includes a convolution layer, an average pooling layer, a convolution layer, a ReLU activation layer, and a convolution layer. The second convolution layer is a 1*1 convolution to improve training speed and reduce computational consumption. The second branch is used to extract local features, which includes a convolution layer, a ReLU activation layer, and a convolution layer. The output results of the two branches are multiplied, and the output is the sum of the input feature map and the product feature map.

[0052] Step 6: Set the number of times to repeat the basic module. In this embodiment, it is four times in total. Repeat the extraction multiple times to learn deeper features.

[0053] Step 7: After the convolution layer converts the 64-channel image into a 3-channel image, it is added to the noise image block processed in step 1 to obtain the denoised image, completing the training;

[0054] Step 8: Calculate the loss function for the denoised image generated in step 7 and the corresponding noise-free clean image in the training set to help the network train better. The loss function is Among them, δ1 is L1Loss, Clean is a clean image without noise, For the denoised image generated in step 7, the selected optimization function is Adam, the parameters are the default parameters, and the initial value of the learning rate is 3*10 -5 .

[0055] At the beginning of the design of this invention, the complexity of real image noise and the principle of image imaging were taken into consideration. During the imaging process, its distribution in the three channels of R, G, and B is different, and the noise distribution in each color area is also different. Therefore, the input dataset images are preliminarily processed before network training to further increase the information that the noise image can provide.

[0056] The learning ability is strong and the module used for feature extraction in the entire network adopts a relatively simple residual network structure. This can avoid overfitting caused by too many feature extraction layers and too deep a network. At the same time, it can also ensure that the network is sufficiently trained to achieve good denoising effects for various noisy images without the need for additional training of models for other specific noises.

[0057] In the embodiment of the present invention, during training, a large number of noisy images are input, and preliminary image processing is first performed using the noise distribution patterns of each color channel to increase image information. At the same time, a deep learning network model is used for feature extraction, ensuring that the characteristics of noise at different levels can be learned by the network. This allows for flexible processing. Figure 4 As shown, the real noise denoising effect is as follows Figure 5 The denoising results for each data set are shown in Table 1.

[0058]

[0059] The above description is merely a preferred embodiment of the present disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also encompass other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by mutually replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. An image denoising method based on deep learning and image color space characteristics, characterized in that: The following steps are involved: Step 1: Read the dataset and perform data augmentation on the noisy image; Step 2: Build a network model; The network model includes: CPBlock image preprocessing operation module, which provides more feature information for the processed image; Conv is a convolution layer with a convolution kernel of 3*3, which is used to change the number of channels of the image and generate feature maps for network training; BasicBlock is a basic module for extracting deep features; The noise image block processed in step 1 is input into the network model. It is first preprocessed by CPBlock. The image generated by the preprocessing and the noise image block processed in step 1 are spliced between channels. After 3 channels are converted to 6 channels, it passes through the first convolution layer and converts 6 channels to 64 channels. It passes through the BasicBlock module four times for deep feature extraction. The number of channels after output is still 64 channels. After the last convolution layer converts 64 channels to 3 channels, the completed 3 channels and the noise image block processed in step 1 are added to finally generate the image prediction after network denoising. Step 3: Convert the noise image block processed in step 1 to HSV mode using the OpenCV library; perform Gaussian blur on the image, and then divide the image into several regions according to the value range of each color of H, S, and V; sum the values of the R, G, and B channels in each divided color region and calculate the average value; sort the average values of the R, G, and B channels, and use the gradients of the channels with the first and second average values for reference for the channel with the third average value, as shown in the following formula: For the channel with the second highest mean, the gradient of the channel with the highest mean is used as a reference. The formula is as follows: ; Among them, the subscript is the channel with the lowest mean, is the channel with the middle mean ranking, is the channel with the highest mean value, and They are the updated pixel values of the channel with the lowest mean and the updated pixel values of the channel with the middle mean ranking, 、 、 They represent the mean of the pixel values in the channel with the lowest mean, the mean of the pixel values in the channel with the middle mean, and the mean of the pixel values in the channel with the highest mean, respectively. 、 、 They are the original pixel values of the channel with the lowest mean, the original pixel values of the channel with the middle mean, and the original pixel values of the channel with the highest mean; The channel with the highest mean value is not modified. Step 3 is performed on each divided color region, and a tensor with a different value from the noise image but the same size is generated. The generated tensor and the noise image block processed in step 1 are concatenated into a 6-channel feature map to obtain the preprocessed result. Step 4: Exchange information between channels through the convolutional layer; Step 5: Extract deep features through the basic module to improve network performance; Step 6: Set the number of times to repeat the basic module to learn deeper features; Step 7: After the convolution layer converts the 64-channel image into a 3-channel image, it is added to the noise image block processed in step 1 to obtain the denoised image, completing the training; Step 8: Calculate the loss function for the denoised image generated in step 7 and the corresponding noise-free clean image in the training set to complete the image denoising.

2. The image denoising method based on deep learning and image color space characteristics according to claim 1, characterized in that: The step 1 specifically includes: reading a noise image from a public dataset of real noise and artificially synthesized noise, randomly cropping the noise image into image blocks of the same size, and cropping the image blocks at the same position, and then rotating the image, selecting the operations of rotating 90 degrees, rotating 180 degrees, rotating 270 degrees, rotating 90 degrees and flipping, rotating 180 degrees and flipping, rotating 270 degrees and flipping, horizontal flipping, and vertical flipping with a probability of 50%.

3. The image denoising method based on deep learning and image color space characteristics according to claim 1, characterized in that: The step 4 is specifically as follows: inputting the preprocessed result into the first convolution layer, the size of the convolution kernel is 3*3, and changing it from 6 channels to 64 channels, so as to further exchange information between channels and extract shallow features.

4. The image denoising method based on deep learning and image color space characteristics according to claim 1, characterized in that: The basic module described in step 5 has two branches along the network transmission direction. The first branch is used to extract global features, which is a convolution layer, an average pooling layer, a convolution layer, a ReLU activation layer, and a convolution layer, where the second convolution layer is a 1*1 convolution; the second branch is used to extract local features, which is a convolution layer, a ReLU activation layer, and a convolution layer; the output results of the two branches are multiplied, and the output is the sum of the input feature map and the product feature map.

5. The image denoising method based on deep learning and image color space characteristics according to claim 1, characterized in that: The loss function in step 8 is ,in, is L1Loss, For a clean image without noise, For the denoised image generated in step 7, the selected optimization function is Adam, and the initial value of the learning rate is 3*10 -5 .

Images