CIS image data noise reduction method
The combination of median filtering and a deep convolutional neural network addresses CIS noise issues under ultra-low illumination, enhancing noise reduction and processing speed by accurately estimating object movement and preserving image details.
Patent Information
- Application Number
- CN202410925717.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-10
- Publication Date
- 2024-12-13
AI Technical Summary
Contact Image Sensors (CIS) suffer from large noise under ultra-low illumination, with traditional noise reduction algorithms being ineffective for moving objects or dynamic scenes, and existing methods fail to adequately address the complex noise composition and distribution, especially in night vision scenarios.
A method combining median filtering with a deep convolutional neural network (DCNN) for CIS image data, utilizing a training process with a sliding window, normalization, and optimization techniques to enhance noise reduction, particularly under low light conditions.
The method effectively reduces noise in CIS images, enhances signal-to-noise ratio, and accelerates processing speed by accurately estimating object movement and preserving image details, overcoming motion blur and improving noise reduction efficacy.
Smart Images
Figure CN119130836A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image data noise reduction processing, and in particular relates to a CIS image data noise reduction method. Background Art
[0002] Contact Image Sensor (CIS), used in scanners, is a device that closely arranges photosensitive units to directly collect light information reflected from the scanned manuscript. CIS has the characteristics of simple structure, good shock resistance, simple control, no need for preheating, and low production cost. When CIS is working, the light emitted by the LED light source array is directly irradiated onto the surface of the object to be scanned. The light reflected from its surface is focused by the self-focusing rod lens array, imaged on the photoelectric sensor array (usually MOS devices), and converted into electric charge for storage. The light intensity at different locations is different, resulting in different light intensities received by sensor units (i.e., pixels of CIS) at different locations. After a series of processing, these charge signals are converted into digital signals to form image data. However, CIS has very large noise under ultra-low illumination, including read noise, thermal noise, dark noise, shot noise, etc. The composition and distribution of the noise are very complex, and the image signal mixed in the noise is very weak. In this case, the noise reduction effect of traditional image noise reduction algorithms is very limited. At the same time, traditional noise reduction algorithms over-rely on noise reduction of similar image blocks, and are almost powerless for moving objects or moving scenes under extremely weak light, let alone dynamic scenes in ultra-low illumination. Summary of the invention
[0003] In view of the above shortcomings, the present invention discloses a CIS image data denoising method, designs a special video denoising deep convolutional neural network to perform video denoising, thereby improving the denoising effect, accelerating the video processing speed, etc.
[0004] The present invention is achieved by adopting the following technical solutions:
[0005] A CIS image data noise reduction method comprises the following steps:
[0006] (1) A CIS sensor collects nighttime video image data, and performs median filtering on the nighttime video image data. The formula used for the median filtering is: g(x,y)=med{f(xm,yn)|(m,n)∈W}, where g(x,y) is the pixel value of the image at the coordinate (x,y) after the median filtering, f(x,y) is the pixel value of the original image at the coordinate (x,y), W represents a sliding window or neighborhood surrounding the pixel (x,y), (m,n) is an offset relative to (x,y) within the window W, and med represents a median operation;
[0007] (2) dividing the nighttime video image data processed in step (1) into three parts: a training set, a validation set, and a test set;
[0008] (3) constructing a deep convolutional neural network, wherein the deep convolutional neural network includes an input layer, a convolutional layer, a pooling layer, a fully connected layer and an output layer, and the activation function of the deep convolutional neural network includes any one or two of ReLU, sigmoid and tanh; normalizing the data in the training set processed in step (2) and inputting it into the input layer for training the deep convolutional neural network, and using the structural similarity index (SSIM) to measure the difference between the predicted image and the true noise-free image; training the model through multiple iterations of back propagation and gradient descent until convergence;
[0009] The calculation formula of SSIM is as follows: SSIM(I clean ,I noisy )=(2μ I μ J +C1)(2σ IJ +C2) / (μ I 2 +μ J 2 +C1)(σ I 2 +σ J 2 +C2), where I clean and I noisy are the original noise-free image and the image after adding noise; μ I and μ J I clean and I noisy The mean value, σ I 2 and σ J 2 I clean and I noisy The variance, σ IJ isI clean and I noisy The covariance of , C1 and C2 are constants used to avoid division by zero and maintain stability;
[0010] (4) using the test set obtained in step (1) and using the peak signal-to-noise ratio (PSNR) to evaluate the training model in step (3); inputting the validation set obtained in step (1) into the training model for processing and evaluation, adjusting the number of network layers and filters according to the evaluation results, and obtaining the optimal deep convolutional neural network model;
[0011] The calculation formula of the PSNR is as follows: PSNR = 10log 10 (L2 / MSE), where L is the maximum possible grayscale value of the image and MSE is the mean square error;
[0012] (5) Apply the optimal deep convolutional neural network model obtained in step (4) to perform denoising on the CIS image data. First, perform median filtering on the CIS image data according to the method described in step (1), and then use the deep convolutional neural network model to process the CIS image data to obtain the denoised image data.
[0013] Furthermore, the data ratio of the training set, validation set and test set described in step (2) is 7:1:2.
[0014] Furthermore, in step (1), the nighttime video image data is sharpened, and the sharpening is performed by any one of Sobel sharpening and Prewitt sharpening.
[0015] The formula used for Sobel sharpening is: Sobel (I) = ∑ i,j Gx×I i,j +Gy×I i,j , where I is the original image, Gx and Gy are the Sobel operators in the horizontal and vertical directions;
[0016] The formula used for the Prewitt sharpening is: Prewitt (I) = ∑ i,j Px×I i,j +Py×I i,j , where I is the original image, Px and Py are the Prewitt operators in the horizontal and vertical directions.
[0017] Furthermore, the size of the filtering window in step (1) is 3×3.
[0018] Furthermore, the deep convolutional neural network in step (3) also includes an optimizer, which is used to adjust the weights in the network to minimize the loss function, and the optimizer is any one of a momentum optimizer, an Adagrad optimizer and an Adam optimizer.
[0019] Furthermore, the normalization process in step (3) adjusts the image pixel values to a range of 0 to 1.
[0020] Furthermore, the deep convolutional neural network model obtained after step (4) is subjected to layer pruning using the ONNX tool.
[0021] Compared with the prior art, this technical solution has the following beneficial effects:
[0022] 1. Aiming at the problem of large noise in CIS images collected under ultra-low illumination and the very complex composition and distribution of noise, the present invention constructs a processing model based on a deep convolutional neural network, and combines the median filter processing technology to obtain a method suitable for CIS night vision image noise reduction processing, which can accurately evaluate the distribution of noise, extract weak signals in the image, and fuse the image signals of the previous and next frames to enhance the current frame image. At the same time, the present invention can more accurately estimate the movement of objects and scenes, avoid the motion blur problem caused by traditional noise reduction, and thus achieve clear extremely weak light imaging. Thereby improving the noise reduction effect and accelerating the video processing speed.
[0023] 2. The present invention first uses median filtering to process night vision image data to remove part of the salt and pepper noise and thermal noise, and then uses a deep convolutional neural network to perform comprehensive noise reduction processing, which can not only improve the noise reduction effect but also reduce the processing time of the deep convolutional neural network.
[0024] 3. The present invention processes the collected image data using Sobel sharpening or Prewitt sharpening and then performs noise reduction processing. Sharpening the image data can enhance the edges and details in the image, making these structures easier to retain in the noise reduction process. Moreover, by enhancing the useful signals in the image, sharpening can improve the signal-to-noise ratio of the image to a certain extent, which helps the noise reduction algorithm to more effectively separate the signal and noise, thereby retaining more image details while removing noise, and can help the noise reduction algorithm converge faster, reducing the number of iterations required for the algorithm, thereby improving processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 It is a flow chart of the CIS image data noise reduction method described in Example 1. DETAILED DESCRIPTION
[0026] The present invention is further described by the following examples, but is not intended to be limiting of the present invention. Specific experimental conditions and methods not specified in the following examples are conventional methods commonly known to those skilled in the art.
[0027] Embodiment 1: A CIS image data noise reduction method, comprising the following steps:
[0028] (1) A CIS sensor collects nighttime video image data, and performs median filtering on the nighttime video image data. The formula used for the median filtering is: g(x,y)=med{f(xm,yn)|(m,n)∈W}, where g(x,y) is the pixel value of the image at the coordinate (x,y) after the median filtering, f(x,y) is the pixel value of the original image at the coordinate (x,y), W represents a sliding window or neighborhood surrounding the pixel (x,y), (m,n) is the offset relative to (x,y) in the window W, med represents a median operation, and the size of the filtering window is 3×3;
[0029] Performing a sharpening process on the nighttime video image data, wherein the sharpening process adopts Sobel sharpening;
[0030] The formula used for Sobel sharpening is: Sobel (I) = ∑ i,j Gx×I i,j +Gy×I i,j , where I is the original image, Gx and Gy are the Sobel operators in the horizontal and vertical directions;
[0031] (2) dividing the nighttime video image data processed in step (1) into three parts: a training set, a validation set, and a test set; the data ratio of the training set, the validation set, and the test set is 7:1:2;
[0032] (3) constructing a deep convolutional neural network, wherein the deep convolutional neural network includes an input layer, a convolutional layer, a pooling layer, a fully connected layer and an output layer, and the activation function of the deep convolutional neural network is ReLU; normalizing the data in the training set processed in step (2) and inputting it into the input layer for training the deep convolutional neural network, and using the structural similarity index (SSIM) to measure the difference between the predicted image and the true noise-free image; training the model through multiple iterations of back propagation and gradient descent until convergence;
[0033] The calculation formula of SSIM is as follows: SSIM(I clean ,I noisy )=(2μ I μ J +C1)(2σ IJ +C2) / (μ I 2 +μ J 2 +C1)(σ I 2 +σ J 2 +C2), where I clean and I noisyare the original noise-free image and the image after adding noise; μ I and μ J I clean and I noisy The mean value, σ I 2 and σ J 2 I clean and I noisy The variance, σ IJ isI clean and I noisy The covariance of , C1 and C2 are constants used to avoid division by zero and maintain stability;
[0034] The normalization process adjusts the image pixel value to a range of 0 to 1; the deep convolutional neural network also includes an optimizer, which is used to adjust the weights in the network to minimize the loss function, and the optimizer is a momentum optimizer;
[0035] (4) using the test set obtained in step (1) and using the peak signal-to-noise ratio (PSNR) to evaluate the training model in step (3); inputting the validation set obtained in step (1) into the training model for processing and evaluation, adjusting the number of network layers and filters according to the evaluation results, and using the ONNX tool to perform layer pruning to obtain the optimal deep convolutional neural network model;
[0036] The calculation formula of the PSNR is as follows: PSNR = 10log 10 (L 2 / MSE), where L is the maximum possible grayscale value of the image and MSE is the mean square error;
[0037] (5) Apply the optimal deep convolutional neural network model obtained in step (4) to perform denoising on the CIS image data. First, perform median filtering on the CIS image data according to the method described in step (1), and then use the deep convolutional neural network model to process the CIS image data to obtain the denoised image data.
[0038] The signal-to-noise ratio of the night vision video image processed by the method described in this embodiment can reach more than 40dB.
[0039] Embodiment 2: A CIS image data noise reduction method, comprising the following steps:
[0040] (1) A CIS sensor collects nighttime video image data, and performs median filtering on the nighttime video image data. The formula used for the median filtering is: g(x,y)=med{f(xm,yn)|(m,n)∈W}, where g(x,y) is the pixel value of the image at the coordinate (x,y) after the median filtering, f(x,y) is the pixel value of the original image at the coordinate (x,y), W represents a sliding window or neighborhood surrounding the pixel (x,y), (m,n) is the offset relative to (x,y) in the window W, med represents a median operation, and the size of the filtering window is 3×3;
[0041] The nighttime video image data is sharpened, and the sharpening process adopts Prewitt sharpening; the formula adopted by the Prewitt sharpening is: Prewitt (I) = ∑ i,j Px×I i,j +Py×I i,j , where I is the original image, Px and Py are the Prewitt operators in the horizontal and vertical directions;
[0042] (2) dividing the nighttime video image data processed in step (1) into three parts: a training set, a validation set, and a test set; the data ratio of the training set, the validation set, and the test set is 7:1:2;
[0043] (3) constructing a deep convolutional neural network, wherein the deep convolutional neural network includes an input layer, a convolutional layer, a pooling layer, a fully connected layer and an output layer, and the activation functions of the deep convolutional neural network include ReLU and sigmoid; normalizing the data in the training set processed in step (2) and inputting it into the input layer for training the deep convolutional neural network, and using the structural similarity index (SSIM) to measure the difference between the predicted image and the true noise-free image; training the model through multiple iterations of back propagation and gradient descent until convergence;
[0044] The calculation formula of SSIM is as follows: SSIM(I clean ,I noisy )=(2μ I μ J +C1)(2σ IJ +C2) / (μ I 2 +μ J 2 +C1)(σ I 2 +σ J 2 +C2), where I clean and I noisyare the original noise-free image and the image after adding noise; μ I and μ J I clean and I noisy The mean value, σ I 2 and σ J 2 I clean and I noisy The variance, σ IJ isI clean and I noisy The covariance of , C1 and C2 are constants used to avoid division by zero and maintain stability;
[0045] The normalization process adjusts the image pixel value to a range of 0 to 1; the deep convolutional neural network also includes an optimizer, which is used to adjust the weights in the network to minimize the loss function, and the optimizer is an Adagrad optimizer;
[0046] (4) using the test set obtained in step (1) and using the peak signal-to-noise ratio (PSNR) to evaluate the training model in step (3); inputting the validation set obtained in step (1) into the training model for processing and evaluation, adjusting the number of network layers and filters according to the evaluation results, and using the ONNX tool to perform layer pruning to obtain the optimal deep convolutional neural network model;
[0047] The calculation formula of the PSNR is as follows: PSNR = 10log 10 (L 2 / MSE), where L is the maximum possible grayscale value of the image and MSE is the mean square error;
[0048] (5) Apply the optimal deep convolutional neural network model obtained in step (4) to perform denoising on the CIS image data. First, perform median filtering on the CIS image data according to the method described in step (1), and then use the deep convolutional neural network model to process the CIS image data to obtain the denoised image data.
[0049] The signal-to-noise ratio of the night vision video image processed by the method described in this embodiment can reach more than 40dB.
[0050] Embodiment 3: A CIS image data noise reduction method, comprising the following steps:
[0051] (1) A CIS sensor collects nighttime video image data, and performs median filtering on the nighttime video image data. The formula used for the median filtering is: g(x,y)=med{f(xm,yn)|(m,n)∈W}, where g(x,y) is the pixel value of the image at the coordinate (x,y) after the median filtering, f(x,y) is the pixel value of the original image at the coordinate (x,y), W represents a sliding window or neighborhood surrounding the pixel (x,y), (m,n) is the offset relative to (x,y) in the window W, med represents a median operation, and the size of the filtering window is 3×3;
[0052] The nighttime video image data is sharpened, and the sharpening process adopts Sobel sharpening; the formula adopted by the Sobel sharpening is: Sobel (I) = ∑ i,j Gx×I i,j +Gy×I i,j , where I is the original image, Gx and Gy are the Sobel operators in the horizontal and vertical directions;
[0053] (2) dividing the nighttime video image data processed in step (1) into three parts: a training set, a validation set, and a test set; the data ratio of the training set, the validation set, and the test set is 7:1:2;
[0054] (3) constructing a deep convolutional neural network, wherein the deep convolutional neural network includes an input layer, a convolutional layer, a pooling layer, a fully connected layer and an output layer, and the activation function of the deep convolutional neural network is tanh; normalizing the data in the training set processed in step (2) and inputting it into the input layer for training the deep convolutional neural network, and using the structural similarity index (SSIM) to measure the difference between the predicted image and the true noise-free image; training the model through multiple iterations of back propagation and gradient descent until convergence;
[0055] The calculation formula of SSIM is as follows: SSIM(I clean ,I noisy )=(2μ I μ J +C1)(2σ IJ +C2) / (μ I 2 +μ J 2 +C1)(σ I 2 +σ J 2 +C2), where I clean and I noisy are the original noise-free image and the image after adding noise; μ I and μJ I clean and I noisy The mean value, σ I 2 and σ J 2 I clean and I noisy The variance, σ IJ isI clean and I noisy The covariance of , C1 and C2 are constants used to avoid division by zero and maintain stability;
[0056] The normalization process adjusts the image pixel value to a range of 0 to 1; the deep convolutional neural network also includes an optimizer, which is used to adjust the weights in the network to minimize the loss function, and the optimizer is an Adam optimizer;
[0057] (4) using the test set obtained in step (1) and using the peak signal-to-noise ratio (PSNR) to evaluate the training model in step (3); inputting the validation set obtained in step (1) into the training model for processing and evaluation, adjusting the number of network layers and filters according to the evaluation results, and using the ONNX tool to perform layer pruning to obtain the optimal deep convolutional neural network model;
[0058] The calculation formula of the PSNR is as follows: PSNR = 10log 10 (L 2 / MSE), where L is the maximum possible grayscale value of the image and MSE is the mean square error;
[0059] (5) Apply the optimal deep convolutional neural network model obtained in step (4) to perform denoising on the CIS image data. First, perform median filtering on the CIS image data according to the method described in step (1), and then use the deep convolutional neural network model to process the CIS image data to obtain the denoised image data.
[0060] The signal-to-noise ratio of the night vision video image processed by the method described in this embodiment can reach more than 40dB.
[0061] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.
Claims
1. A CIS image data denoising method, characterized in that: The following steps are involved: (1) A CIS sensor collects nighttime video image data, and performs median filtering on the nighttime video image data. The formula used for the median filtering is: g(x,y)=med{f(xm,yn)|(m,n)∈W}, where g(x,y) is the pixel value of the image at the coordinate (x,y) after the median filtering, f(x,y) is the pixel value of the original image at the coordinate (x,y), W represents a sliding window or neighborhood surrounding the pixel (x,y), (m,n) is an offset relative to (x,y) within the window W, and med represents a median operation; (2) dividing the nighttime video image data processed in step (2) into three parts: a training set, a validation set, and a test set; (3) constructing a deep convolutional neural network, wherein the deep convolutional neural network includes an input layer, a convolutional layer, a pooling layer, a fully connected layer and an output layer, and the activation function of the deep convolutional neural network includes any one or two of ReLU, sigmoid and tanh; normalizing the data in the training set processed in step (2) and inputting it into the input layer for training the deep convolutional neural network, and using a structural similarity index to measure the difference between the predicted image and the true noise-free image; training the model by multiple iterations of back propagation and gradient descent until convergence; The calculation formula of SSIM is as follows: SSIM(I clean ,I noisy )=(2μ I μ J +C1)(2σ IJ +C2) / (μ I 2 +μ J 2 +C1)(σ I 2 +σ J 2 +C2), where I clean and I noisy are the original noise-free image and the image after adding noise; μ I and μ J I clean and I noisy The mean value, σ I 2 and σ J 2 I clean and I noisy The variance, σ IJ isI clean and I noisy The covariance of , C1 and C2 are constants used to avoid division by zero and maintain stability; (4) using the test set obtained in step (1) and using the peak signal-to-noise ratio to evaluate the training model in step (3); inputting the validation set obtained in step (1) into the training model for processing and evaluation, adjusting the number of network layers and the number of filters according to the evaluation results, and obtaining the optimal deep convolutional neural network model; The calculation formula of the PSNR is as follows: PSNR = 10log 10 (L 2 / MSE), where L is the maximum possible grayscale value of the image and MSE is the mean square error; (5) Apply the optimal deep convolutional neural network model obtained in step (4) to perform denoising on the CIS image data. First, perform median filtering on the CIS image data according to the method described in step (1), and then use the deep convolutional neural network model to process the CIS image data to obtain the denoised image data.
2. The CIS image data denoising method according to claim 1, characterized in that: The data ratio of the training set, validation set and test set described in step (1) is 7:1:
2.
3. The CIS image data denoising method according to claim 1, characterized in that: In step (1), the nighttime video image data is sharpened, and the sharpening process adopts any one of Sobel sharpening and Prewitt sharpening; The formula used for Sobel sharpening is: Sobel (I) = ∑ i,j Gx×I i,j +Gy×I i,j , where I is the original image, Gx and Gy are the Sobel operators in the horizontal and vertical directions; The formula used for the Prewitt sharpening is: Prewitt (I) = ∑ i,j Px×I i,j +Py×I i,j , where I is the original image, Px and Py are the Prewitt operators in the horizontal and vertical directions.
4. The CIS image data denoising method according to claim 1, characterized in that: The size of the filtering window in step (1) is 3×3.
5. The CIS image data denoising method according to claim 1, characterized in that: The deep convolutional neural network in step (3) also includes an optimizer, which is used to adjust the weights in the network to minimize the loss function. The optimizer is any one of a momentum optimizer, an Adagrad optimizer and an Adam optimizer.
6. The CIS image data denoising method according to claim 1, characterized in that: The normalization process described in step (3) adjusts the image pixel values to the range of 0 to 1.
7. The CIS image data denoising method according to claim 1, characterized in that: For the deep convolutional neural network model obtained after step (4), the ONNX tool is used to perform layer pruning.