A Lightweight Convolutional Neural Network for Salt-and-Pepper Noise Labeling

Abstract

This solution discloses a lightweight convolutional neural network method for salt-and-pepper noise labeling in the field of intelligent image processing technology, comprising: S1, adding random noise to normal image blocks to generate a noisy image-noise mask pair dataset; S2, constructing a lightweight convolutional neural network model for noise labeling, including multiple repeated depthwise separable convolutions; S3, training the noisy image-noise mask pair dataset using the network neural model to obtain the noise mask. This invention is the first to apply a lightweight convolutional neural network to salt-and-pepper noise labeling, significantly reducing the computational complexity of the network and the false positive rate of noise labeling. The network uses a smaller network depth and a simpler convolutional structure, achieving excellent noise mask restoration results.

Description

Technical Field

[0001] This invention belongs to the field of intelligent image processing technology, and specifically relates to a lightweight convolutional neural network method for salt-and-pepper noise labeling. Background Technology

[0002] With the rapid growth of digital imaging and video applications, image denoising remains a challenging task. Digital images are susceptible to salt-and-pepper noise during sensor acquisition, storage, and transmission, and this noise needs to be removed for better subsequent processing. Noise marking and noise removal are two key steps in the salt-and-pepper denoising process. Currently, the industry has conducted extensive research on the noise removal step and achieved good results.

[0003] In essence, noise calibration has a direct impact on the performance of salt-and-pepper noise reduction. Compared to the noise removal step, research on the noise calibration step is relatively lacking in the industry. Traditional salt-and-pepper noise calibration methods include the following three types.

[0004] (a) Pole calibration method

[0005]

[0006] The principle of the pole calibration method is relatively simple. It marks points with a gray level of 0 or 255 in the input noisy image as noise points, and other points as normal points. The method is shown in (1), where BJ i,j This outputs noise markers; noise points are marked as 0, and normal points as 1. (img) ij This represents the grayscale value of the pixel in the i-th row and j-th column of the input image. The image has a total of H rows and W columns, so the value of i is an integer between 1 and H, and the value of j is an integer between 1 and W.

[0007] (b) Extreme value image patch calibration method

[0008] The principle of extreme value image block calibration method is to scan the pixels with gray levels of 0 or 255 in the image, take a k*k window centered on it, count the number of pixels with gray levels of 0 and 255, and output the label value according to the relationship between these numbers.

[0009] N0 = card{p|win m,n =0},m,n=[1,k], (2)

[0010] The first step of this method is to count the number of extreme point pixels in the sliding window. If the gray level of the scanned pixel is 0, then count the number of pixels with a gray level of 0 in the sliding window, N0, as shown in (2). Here, win represents a sliding window of size k*k, which is usually taken as 5*5.

[0011] N'0=k 2-N0, (3)

[0012] The number of pixels with a gray level of non-zero in the window is shown in (3), where k 2 This represents the total number of pixels in the window.

[0013] If the scanned pixel has a gray level of 255, the method for calculating the number of pixels with a gray level of 255 and those without is similar.

[0014]

[0015] The extreme value image block marking method is shown in (4). Each pixel unit in the scanned image is considered a noise point if there are fewer pixels with a gray level of 0 in the sliding window when the gray level is 0. If there are fewer pixels with a gray level of 255 in the sliding window when the gray level is 255, the pixel is considered a noise point. In other cases, the scanned point is considered a normal point.

[0016] (c) Mean calibration method

[0017]

[0018]

[0019] The mean calibration method determines whether a scanned point is noise by measuring the average gray level within a sliding window. First, the average gray level of the pixels within the sliding window is calculated, as shown in formulas (5) and (6). Each pixel in the scanned image is then analyzed. When the gray level is 0, if the average gray level of the pixels within the sliding window is relatively large, it indicates that the scanned point is distorted and is therefore identified as noise. When the gray level is 255, if the average gray level of the pixels within the sliding window is relatively small, it indicates that the scanned point is distorted and is therefore identified as noise. In other cases, the scanned point is considered a normal point.

[0020] Due to its simple principle and ease of operation, the extremum calibration method is the most commonly used labeling method. It marks pixels with a grayscale value of 0 or 255 as noise points and other pixels as normal points. If the original image itself contains many extrema, these extrema, which are actually normal pixels, will be misclassified as noise. To solve this problem, the industry has proposed extremum image block labeling, which determines whether a scanned point is noise by comparing the number of normal points and extrema in a 5x5 window. Although this method reduces misclassifications at low density, it still has many misclassifications for high-density noise. In addition, the industry has proposed the mean calibration method, which often uses the average grayscale of pixels in a 3x3 window for noise identification. However, this method will fail when there are a large number of extrema in the background of the input image. Summary of the Invention

[0021] This invention aims to improve the accuracy of salt-and-pepper noise labeling and proposes a lightweight convolutional neural network-based salt-and-pepper noise labeling method.

[0022] This solution includes a lightweight convolutional neural network method for salt-and-pepper noise labeling, comprising:

[0023] S1. Generate a noisy image-noise mask pair dataset by adding random noise to normal image patches;

[0024] S2. Build a lightweight convolutional neural network model with noise labeling, including multiple repeated depthwise separable convolutions.

[0025] S3. The network neural model is used to train the noisy image-noise mask dataset to obtain the noise mask.

[0026] Furthermore, the lightweight convolutional neural network model includes two regular convolutions and n repeated depthwise separable convolutions between the two regular convolutions, where n ≥ 5; wherein the regular convolution at the beginning generates 32 to 128 convolution tensors for the input noisy image; and the regular convolution at the end generates one noise mask for all the input convolution tensors.

[0027] Furthermore, the conventional convolution at the beginning first processes Ni input channels using a 3×3 convolution kernel, followed by batch normalization and ReLU activation. Then, a 1×1 convolution kernel is used to process all channels sequentially, resulting in No processed channels; this step is called point convolution. Finally, after batch normalization and activation, No channels are output.

[0028] Furthermore, in step S1, the 91images dataset is selected as the original data, and the color image is converted into a black and white image with a resolution of (200~300)×(200~300).

[0029] Furthermore, in step S1, several image blocks are extracted from each image with a step size of less than 50. Random density salt and pepper noise is added to these image blocks to obtain noisy image blocks. The positions where noise is added are marked to distinguish them from other positions, generating a real noise mask corresponding to the noisy image. Finally, the noisy image blocks and the noise mask are combined to form a one-to-one noise image-noise mask pair dataset.

[0030] Furthermore, 5 ≤ n ≤ 15.

[0031] Furthermore, the regular convolution at the beginning generates 64 convolution tensors from the input noisy image.

[0032] Furthermore, after the color image is converted to a black and white image, the resolution is adjusted to 200×200.

[0033] The beneficial effects of this invention are as follows: This invention proposes a lightweight convolutional neural network model for noise labeling, which reduces the network depth by half compared to traditional network models. While reducing the network depth, depthwise separable convolution is used instead of conventional convolution, resulting in an order-of-magnitude reduction in computational complexity and the number of parameters. Experimental results show that the computational complexity of the proposed network is reduced by 160 times compared to traditional networks, and the misclassification rate of the noise mask obtained through training is reduced by 94.79%, 94.79%, and 83.65% compared to extremum labeling, mean labeling, and extremum image patch labeling, respectively. This invention is the first to apply a lightweight convolutional neural network to salt-and-pepper noise labeling, significantly reducing the network's computational complexity and the misclassification rate of noise labeling.

[0034] Based on the above optimization schemes, the network adopts a smaller network depth and a simpler convolutional structure, which can achieve a good noise mask restoration effect and avoid the waste of computing resources to a great extent. Attached Figure Description

[0035] Figure 1 The process of constructing a dataset of noisy images and noise masks;

[0036] Figure 2 A lightweight convolutional neural network model for training noise masks;

[0037] Figure 3 Depthwise separable convolution;

[0038] Figure 4 Schematic diagram of image patch extraction method;

[0039] Figure 5 Noisy image-noise mask pair;

[0040] Figure 6 Comparison of noise mask images. Detailed Implementation

[0041] To demonstrate the specific implementation method of the present invention, specific embodiments are provided. It is worth noting that the data used in the following steps will be specified, and modifications to the related data may not significantly affect the implementation effect and will still fall within the scope of protection of this patent. The steps of the present invention are performed sequentially as follows: Figure 1 As shown, the process includes: Step 1, dataset preparation, which includes image patching, generating noisy image-noise mask data pairs, and constructing a noisy image-noise mask dataset; Step 2, machine training, which includes building a lightweight convolutional neural network and training it to obtain a noise mask generator; and Step 3, labeling the input noisy images.

[0042] Step 1: Dataset Preparation

[0043] (1) Image patch extraction

[0044] Methods for extracting image blocks, such as Figure 4 As shown, in a clean image, a 70×70 resolution image patch is cropped starting from the top left corner with a stride of 20. For a 200×200 resolution image, five croppings are made horizontally and five vertically, scanning from left to right and top to bottom, resulting in 25 70×70 image patches. To fully represent the image features, the resolution of the image patches should not be too small. To avoid an excessively large number of training parameters, the resolution of the image patches should not be too large. Therefore, this paper chooses a 70×70 image patch resolution.

[0045] (2) Generate noisy image-noise mask data pairs

[0046] To train a noise mask, a noisy image needs to be input into a convolutional neural network. The network's output is compared with the original noise mask to generate a loss function. The output is continuously improved through iterative training using this loss function. Therefore, the noisy image-noise mask fed to the training network is crucial to the data. This paper adds random density salt-and-pepper noise to the generated 70×70 image patches to generate a noisy image. Simultaneously, the location where noise is added is marked as 1, while pixels without added noise are marked as 0, thus generating the noise mask. It is noteworthy that there is a one-to-one correspondence between the noisy image and the noise mask, forming a noise image-noise mask pair. Furthermore, to enhance the robustness of the dataset, the density of the noise added to each image patch is randomly determined, with the noise density ranging from 0.1 to 0.9. Figure 5 These are noise image-noise mask pairs corresponding to several image patches. It can be seen that the noise image is obtained by adding noise to the image patch. The noise mask marks the location where noise is added as a white dot and the normal pixel as a black dot.

[0047] (3) Constructing a noisy image-noise mask pair dataset

[0048] Based on the image patch extraction and noise mask generation described above, a noisy image-noise mask pair dataset is constructed. This invention uses the 91images dataset as the source data, converting color images to black and white and adjusting their resolution. Twenty-five image patches are extracted from each image with a stride of 20. Random density salt-and-pepper noise is added to these patches to obtain noisy image patches. Simultaneously, the locations where noise is added are marked as 1, and other locations as 0, generating a real noise mask corresponding to the noisy image. Finally, the noisy image patches and noise mask patches are combined to form a one-to-one noise image-noise mask pair, as shown below. Figure 5 As shown.

[0049] Step 2: Machine Training

[0050] (1) Building a lightweight convolutional neural network

[0051] In the Tensorflow platform, calling Keras components follows... Figure 2 , Figure 3 The convolutional neural network model shown is used to build the network.

[0052] (2) Training was conducted to obtain a noise mask generator.

[0053] The mean squared error (MSE) is used as the loss function, with an initial learning rate of 0.001. If the loss function value does not decrease after 20 training epochs, it is decreased by 0.2. The generated noisy image-noise mask pair dataset is imported, with 8 pairs of data per batch, and iterative training is performed for 50 epochs to obtain a noise mask generator.

[0054] Step 3: Label the input noisy image

[0055] Import an image with salt and pepper noise, process it using the noise mask generator described above, and output a noise mask image, from which you can see the distribution of noise points in each pixel unit.

[0056] Implementation effect

[0057] (1) Marked image comparison

[0058] Figure 6 For the noise mask images generated by different methods, three representative images were selected for the experiment. Kodim18 contains a large number of dark pixels and a small number of bright pixels; Kodim20 contains a large number of bright pixels and a small number of dark pixels in the image details; and Kodim24 contains a small number of bright pixels and a large number of pixels with mediocre gray levels. To detect the mask quality under different noise densities, low, medium, and high densities of salt-and-pepper noise were added to Kodim20, Kodim24, and Kodim18, respectively. Figure 6 In the noise mask image, noise points are marked with black dots, and normal points are marked with white dots. It is worth noting that since the noise points in the original noise image are uniformly distributed, Figure 6 The more uniform the distribution of black dots in the noise mask image shown, the more it conforms to objective reality, and the higher the quality of the noise mask. From Figure 6 It can be seen that the extremum labeling method, by marking extreme pixels as noise, is prone to misjudgment when there are many extreme points in the image itself, such as... Figure 6 (b3) and Figure 6In (c3), a large number of sky pixels are marked as noise, and the noise mask image is severely affected by the extreme pixels in the original image, resulting in widespread misclassification. Furthermore, the mean calibration method determines noise based on the difference between the average grayscale of pixels in the window and the extreme values; this method is prone to failure when the distribution of extreme pixels is concentrated in certain areas of the image. Therefore, Figure 6 (b5) and Figure 6 (c5) contains a large number of misclassified points. The extreme value image block calibration method identifies noise by comparing the number of extreme pixels and normal pixels in the window. Compared to the two methods above, it better utilizes the characteristics of noise distribution, and the generated mask is almost no longer affected by extreme pixels in the original image, achieving a relatively ideal effect, such as... Figure 6 (b4) and Figure 6 As shown in (c4). Compared to these traditional methods, the CNN calibration method proposed in this paper uses machine learning to grasp the characteristics of noise distribution, making it an intelligent noise mask generation method. Figure 6 (a6) Figure 6 (b6) and Figure 6 As shown in (c6), the noise mask image is almost no longer affected by the extreme pixels in the original image, especially Figure 6 (a6) exhibits a perfectly uniform noise distribution, which is very close to the ideal noise distribution, further demonstrating the strong robustness of the proposed method. Furthermore, Figure 6 The eaves lines in (c2) no longer appear Figure 6 (c6) explains that the method is no longer affected by image edges and will not produce false judgments.

[0059] (2) Comparison of false positive rates

[0060] Table 1 Comparison of False Positive Rates

[0061]

[0062] From a subjective visual perspective, it is evident that CNN noise masks offer higher quality. To objectively compare the false positive rates of various noise mask generation methods, this paper adds various levels of noise to the Kodim image, with noise densities ranging from 0.1 to 0.9. The false positive rate is the ratio of the difference between the number of true noise points and the number of labeled noise points in an image to the number of true noise points; a lower false positive rate indicates better noise mask quality. The comparative data is shown in Table 1. Consistent with the above analysis, the extremum labeling method tends to label extreme points as noise, resulting in a high false positive rate. The mean labeling method tends to fail in image regions with large areas of extreme points, thus also resulting in a high false positive rate. Compared to the former two, the extreme image block labeling method labels noise points based on the relationship between the number of extreme points and normal points in the window, better capturing the characteristics of noise distribution, thereby improving the false positive rate. The deep CNN labeling method proposed in this paper comprehensively extracts the characteristics of noise distribution through machine learning, achieving the best noise mask quality. Table 1 shows that the false positive rate of the noise mask generated by the method in this paper is reduced by 94.79%, 94.79%, and 83.65% compared with the extreme point label, mean label, and extreme image block label, respectively.

[0063] (3) Comparison of Convolutional Neural Network Complexity

[0064] Table 2 Comparison of Network Complexity

[0065] network MFLOPs Number of parameters .h5 file size (KB) Liang CNN 17372.03 3,572,993 43,712 Xing CNN 2717.66 558,977 6,840 This article is from CNN. 16.82 38,401 723

[0066] Liang CNN contains as many as 32 residual layers and median filter layers, resulting in extremely high computational complexity and parameter count. Xing CNN has 17 regular convolutional layers, and its computational complexity and parameter count remain high. Compared to traditional salt-and-pepper denoising convolutional neural networks, this paper reduces the network depth and uses depthwise separable convolutions instead of regular convolutions in the intermediate layers. These two factors significantly reduce the network complexity. The industry uses MFLOPs (the number of floating-point multiply-accumulators) to evaluate the computational complexity of convolutional neural networks. As shown in Table 2, the computational complexity of the network model proposed in this paper is 160 times lower than that of Xing CNN. In addition, the number of parameters and the size of the .h5 file are reduced by orders of magnitude compared to traditional networks.

Claims

1. A pepper noise labeling method of a lightweight convolutional neural network, characterized by, The application relates to a method for generating a noise mask of a noise image, comprising the following steps: S1, adding salt and pepper noise in a normal image block to generate a noise image-noise mask pair data set; S2, building a lightweight convolutional neural network model adopting noise point marking, comprising a plurality of repeated depth separable convolutions; S3, training the noise image-noise mask pair data set by using the network neural model to obtain a noise mask; the lightweight convolutional neural network model comprises two normal convolutions and n repeated depth separable convolutions between the two normal convolutions, and n>=5; wherein the normal convolution at the head end generates 32-128 convolution tensors for the input noise image; the normal convolution at the tail end generates one noise mask for all the input convolution tensors.

2. The method of claim 1, wherein: The normal convolution at the head end first processes Ni input channels by using a 3*3 convolution kernel, then performs batch normalization and ReLU activation, sequentially processes all channels by using a 1*1 convolution kernel to obtain No processed channels, and the step is called point convolution; finally, No channels are output after batch normalization and activation.

3. The method of claim 2, wherein: In the S1, the 91images data set is selected as original data, color images are converted into black and white images, and the resolution is adjusted to (200-300)*(200-300).

4. The method of claim 3, wherein: In the S1, a plurality of image blocks are cut from each image with a step less than 50, random density salt and pepper noise is added to the image blocks to obtain noise image blocks, the positions of the added noise are marked to be different from other positions, and a real noise mask corresponding to the noise image is generated; Finally, the noise image blocks and the noise mask blocks are combined to form a one-to-one corresponding noise image-noise mask pair data set.

5. The method of claim 1, wherein: The 5<=n<=15.

6. The method of claim 1, wherein: The normal convolution at the head end generates 64 convolution tensors for the input noise image.

7. The method of claim 3, wherein: After the color image is converted into a black and white image, the resolution is adjusted to 200*200.

Images