Low-illumination camera real-time noise reduction method and device based on neural network
By acquiring and fusing image data under low illumination conditions, building a lightweight Unet neural network and quantizing it, the problems of poor noise reduction effect and high computational complexity in the existing technology are solved, and real-time and effective image noise reduction effect is achieved.
Patent Information
- Application Number
- CN202510322868.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-18
- Publication Date
- 2025-07-01
AI Technical Summary
The existing image noise reduction technology is poorly effective under low illumination conditions, which can easily lead to excessive smoothing of image details and textures, high computational complexity, difficult to meet real-time processing needs, and limited processing capabilities for mixed noise.
A lightweight noise reduction method based on neural network is adopted to acquire image data in darkrooms and bright environments, and weighted fusion is performed to obtain training samples. A Unet network structure including encoder and decoder is built, and the SE channel attention module is introduced. The L1 loss function is used for training, and the model is finally quantized and deployed to the edge device.
Real-time noise reduction for low-illumination images is achieved, image quality is significantly improved, details and texture information are retained, computational complexity is reduced, real-time processing needs are met, and the ability to handle multiple noise types is provided.
Smart Images
Figure CN120236085A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and particularly to a real-time noise reduction method and device for a low-illumination camera based on a neural network. Background Art
[0002] Image noise reduction is a key technology in the fields of computer vision and image processing. The purpose of this technology is to recover a clear and high-quality image from a noisy image. The source of noise is inevitable during the camera imaging process. Therefore, the impact of noise on the visual quality of the image is an unavoidable factor. Especially in low-illumination environments, images often have serious noise problems, which greatly affect the image quality and subsequent processing and analysis of image information.
[0003] Common types of noise include Gaussian noise and Poisson noise. Gaussian noise is also known as additive white noise, and its intensity follows a Gaussian distribution. Gaussian noise usually comes from the thermal noise of electronic components and appears as uniformly distributed random fluctuations in the image. The characteristic of Gaussian noise is that it is independent of the image signal and can be described by the mean and variance. Poisson noise is also known as shot noise, and its intensity follows a Poisson distribution. Poisson noise mainly comes from the quantum characteristics of photons and is particularly obvious under low-illumination conditions. Different from Gaussian noise, the intensity of Poisson noise is related to the intensity of the image signal. The weaker the signal, the higher the relative noise level. In actual low-illumination images, these two types of noise often exist simultaneously and may also contain other types of noise, such as salt-and-pepper noise, stripe noise, etc., making the noise reduction problem more complex.
[0004] Traditional noise reduction methods such as Gaussian filtering and bilateral filtering can suppress noise to a certain extent, but often lead to the loss of image details. For example, Gaussian filtering can effectively smooth Gaussian noise but will also blur the image edges; bilateral filtering can retain edge information to a certain extent, but its processing effect on strong noise is limited. Generally speaking, traditional noise reduction methods are not ideal.
[0005] In recent years, image noise reduction methods based on deep learning have made remarkable progress. For example, the DnCNN network improves the noise reduction performance of the convolutional neural network through residual learning and batch normalization; CBDNet is specifically designed for real noisy images and guides the noise reduction process by estimating the noise level. However, these methods still have the following defects when dealing with low-illumination images:
[0006] First, for different degrees of noise, the noise reduction effect is not stable enough;
[0007] Second, it is easy to cause excessive smoothing of image details and textures;
[0008] Third, the computational complexity is high and it is difficult to meet the requirements of real-time processing;
[0009] 4. Limited processing ability for mixed noise such as the combination of Gaussian noise and Poisson noise. Summary of the Invention
[0010] In view of the deficiencies of the prior art, the present invention provides a real-time noise reduction method and device for a low-light camera based on a neural network, which solves the problems of poor noise reduction effect and high computational complexity existing in the existing image noise reduction technology.
[0011] To achieve the above objectives, the present invention is realized through the following technical solutions: A real-time noise reduction method for a low-light camera based on a neural network specifically includes the following steps:
[0012] Step 1: Collect images at different illuminance levels in a darkroom as noise data, and collect images with relatively less noise and clear scenes in a bright environment as clean data.
[0013] Step 2: Use OpenCV to fuse the noise data and clean data to obtain training samples.
[0014] Step 3: Build a lightweight neural network and use the training samples for training until the final model is obtained.
[0015] Step 4: Quantify the final model.
[0016] Step 5: Deploy the quantized model to an edge device.
[0017] The present invention is further configured as follows: The method of using OpenCV to fuse the noise data and clean data in Step 2 includes:
[0018] After weight distribution for the noise data and clean data, weighted fusion of the noise data and clean data is performed according to the allocated weight ratio, where the weight ratio is 1:1.
[0019] The present invention is further configured as follows: The lightweight neural network in Step 3 includes three encoder layers and three decoder layers from top to bottom, and the topmost encoder is jump-connected to the bottommost decoder, the bottommost encoder is jump-connected to the topmost decoder, and the middle encoder is jump-connected to the middle decoder.
[0020] The present invention is further configured as follows: The encoder includes three downsampling modules, and each downsampling module includes a 3x3 depthwise separable convolutional layer, a batch normalization layer, a ReLU activation function, and a 2x2 max pooling layer connected in sequence.
[0021] The present invention is further configured that: the decoder includes three upsampling modules, and each upsampling module includes a 2x2 transposed convolution layer, a skip connection with the corresponding layer of the encoder, a 3x3 depthwise separable convolution layer, a batch normalization layer, and a ReLU activation function.
[0022] The present invention is further configured that: an SE channel attention module is further introduced into the encoder and the decoder, and the SE channel attention module sequentially includes a global average pooling layer, two fully connected layers, and a Sigmoid activation function from top to bottom, and a ReLU activation function is arranged between the two fully connected layers.
[0023] The present invention is further configured that: the lightweight neural network selects an L1 loss function:
[0024]
[0025] where y pred is the network prediction output, and y true is the training sample.
[0026] The present invention is further configured that: the method for quantizing the final model in step four includes:
[0027] Using the MinMax method to determine the zero point Z and the scaling factor S:
[0028]
[0029] where R max is the maximum value of the floating-point data, R min is the minimum value of the floating-point data, Q max is the maximum value of the fixed-point number, and Q min is the minimum value of the fixed-point number;
[0030] Quantizing the floating-point number R into a fixed-point number Q:
[0031]
[0032] R = (Q - Z) × S
[0033] The present invention also discloses a real-time noise reduction device for a low-light camera based on a neural network, including:
[0034] A data acquisition module, which is used to acquire noise data and clean data;
[0035] A data synthesis module, which is used to generate training samples according to the noise data and the clean data;
[0036] A neural network module, which is used as a lightweight neural network to perform noise reduction processing on images;
[0037] A model quantization module, which is used to convert the trained lightweight neural network into a fixed-point representation;
[0038] An image processing module, which is a lightweight neural network in fixed-point representation and is used to perform real-time noise reduction processing on the input low-light image.
[0039] The present invention provides a real-time noise reduction method and device for a low-light camera based on a neural network. It has the following beneficial effects:
[0040] The present invention obtains higher-quality and diverse training data by means of weighted fusion of noise data and clean data, and uses a lightweight neural network to effectively remove noise in low-light images while retaining the details and texture information of the images. It has a low computational complexity, effectively meets the requirements of real-time processing, has a significant noise reduction effect, can model various different low-light camera noises, can adapt to different noise types and low-light scenarios, and has the advantage of strong generalization ability. Description of the Drawings
[0041] Figure 1 It is a schematic flowchart of the present invention;
[0042] Figure 2 It is a schematic architecture diagram of the lightweight neural network of the present invention;
[0043] Figure 3 It is a schematic architecture diagram of the SE channel attention module of the present invention;
[0044] Figure 4 It is a schematic diagram of model quantization in the embodiment of the present invention. Detailed Embodiments
[0045] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention.
[0046] Please refer to Figures 1-4 , the embodiments of the present invention provide the following technical solutions: A real-time noise reduction method for a low-light camera based on a neural network, specifically including the following steps:
[0047] Step 1: Collect a group of basic black frames in a dark room to reflect the basic noise distribution of the camera under this illumination condition, and collect images at different illumination levels in the dark room as noise data;
[0048] In a bright environment, collect images with relatively less noise and clear scenes as clean data, so as to ensure the complex and changeable image environment and enhance the robustness of the network.
[0049] Step 2: Use OpenCV to fuse the noisy data and the clean data to obtain training samples. The number of training samples is preferably 10,000 - 20,000 pairs. Specifically:
[0050] Allocate a weight ratio of 1:1 to the noisy data and the clean data, and perform weighted fusion on the noisy data and the clean data according to the allocated weight ratio to achieve the effect of adding noise to the image.
[0051] Step 3: Build a lightweight neural network. The lightweight neural network adopts a three-layer Unet network structure. As shown in the appendix Figure 2 The lightweight neural network includes three layers of encoders and three layers of decoders from top to bottom. Moreover, the top encoder is skip-connected to the bottom decoder, the bottom encoder is skip-connected to the top decoder, and the middle encoder is skip-connected to the middle decoder. To enhance the network's perception ability of the features of the noise, the Convolution-BatchNorm-ReLU structure is used in the encoder and the decoder, and the SE channel attention mechanism, that is, the SE channel attention module, is introduced. As shown in the appendix Figure 3 The SE channel attention module includes a global average pooling layer, two fully connected layers, and a Sigmoid activation function from top to bottom. A ReLU activation function is set between the two fully connected layers, and depthwise separable convolution is used instead of standard convolution. Depthwise separable convolution is composed of Depthwise (DW) convolution and Pointwise (PW) convolution. This structure is similar to the conventional convolution and can be used to extract features. However, compared with the conventional convolution, its number of parameters and computational cost are lower, effectively solving the problem of high computational complexity of the existing model.
[0052] For the loss function part, the L1 loss, L2 loss, and perceptual loss are compared. The L1 loss is less sensitive to outliers than the L2 loss and can better handle the extreme noise that may exist in low-light images. The L1 loss can better retain the details and edge information of the image, which is particularly important for low-light image denoising. Finally, the L1 loss function is selected:
[0053]
[0054] where y pred is the network prediction output, and y true is the training sample.
[0055] And use the training samples for training until the final model is obtained. Specifically, use the PyTorch deep learning framework for model training. The main parameter settings are as follows: batch size: 32, learning rate 0.001, use the Adam optimizer, and the number of training epochs is 150.
[0056] Step 4: To deploy the final model to an edge device, the final model is quantized. Model quantization refers to converting the floating-point algorithm of a neural network into fixed-point. Model quantization requires finding the relationship of data mapping. The conversion formula from floating-point to fixed-point data includes:
[0057]
[0058] R = (Q - Z) × S
[0059] In the formula, Z is the zero point, S is the scaling factor, R is the floating-point number, and Q is the fixed-point number;
[0060] This mapping relationship is determined according to the two parameters S and Z. The MinMax solution methods include:
[0061]
[0062] In the formula, R max is the maximum value of the floating-point data, R min is the minimum value of the floating-point data, Q max is the maximum value of the fixed-point number, Q min is the minimum value of the fixed-point number;
[0063] Finally, 1000 validation set images are used for calibration, and the quantization parameters are adjusted to minimize the accuracy loss, completing the quantization calibration to obtain the final quantized model.
[0064] Step 5: Deploy the obtained final quantized model to the edge device.
[0065] During use, the collected feature images are input into the edge device, and the edge device can use the loaded final quantized model to perform noise reduction processing on the feature images.
[0066] As an optional embodiment, a real-time noise reduction device for a low-light camera based on a neural network includes:
[0067] A data acquisition module, which is used to acquire noise data and clean data;
[0068] A data synthesis module, which is used to generate training samples according to the noise data and clean data;
[0069] A neural network module, which is a lightweight neural network and is used to perform noise reduction processing on images;
[0070] A model quantization module, which is used to convert the trained lightweight neural network into fixed-point representation;
[0071] An image processing module, which is a lightweight neural network in fixed-point representation and is used to perform real-time noise reduction processing on the input low-light image.
[0072] It should be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variation thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device.
[0073] Although the embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A real-time noise reduction method for a low-light camera based on a neural network, characterized in that: The specific steps include: Step 1: Collect images at different illumination levels in a dark room as noise data, and collect images with relatively less noise and clear scenery in a bright environment as clean data; Step 2: Use OpenCV to fuse the noisy data and clean data to obtain training samples; Step 3: Build a lightweight neural network and use the training samples for training until the final model is obtained; Step 4: Quantify the final model; Step 5: Deploy the quantized model to the edge device.
2. The method for real-time noise reduction of a low-light camera based on a neural network according to claim 1, characterized in that: The method of using OpenCV to fuse the noise data and the clean data in step 2 includes: After weights are assigned to the noise data and the clean data, the noise data and the clean data are weightedly fused according to the assigned weight ratio, where the weight ratio is 1:
1.
3. The method for real-time noise reduction of a low-light camera based on a neural network according to claim 1, characterized in that: The lightweight neural network in step three includes three layers of encoders and three layers of decoders from top to bottom, and the top encoder is jump-connected with the bottom decoder, the bottom encoder is jump-connected with the top decoder, and the middle encoder is jump-connected with the middle decoder.
4. The method for real-time noise reduction of a low-light camera based on a neural network according to claim 3, characterized in that: The encoder includes three downsampling modules, each of which includes a 3x3 depthwise separable convolutional layer, a batch normalization layer, a ReLU activation function, and a 2x2 maximum pooling layer connected in sequence.
5. The method for real-time noise reduction of a low-light camera based on a neural network according to claim 4, characterized in that: The decoder includes three upsampling modules, which include a 2x2 deconvolution layer, a skip connection to the corresponding layer of the encoder, a 3x3 depth-separable convolution layer, a batch normalization layer and a ReLU activation function.
6. The method for real-time noise reduction of a low-light camera based on a neural network according to claim 5, characterized in that: The encoder and decoder also introduce an SE channel attention module, which includes a global average pooling layer, two fully connected layers and a Sigmoid activation function from top to bottom, wherein a ReLU activation function is set between the two fully connected layers.
7. The method for real-time noise reduction of a low-light camera based on a neural network according to claim 1, characterized in that: The lightweight neural network uses the L1 loss function: In the formula, y pred is the network prediction output, y true is the training sample.
8. The method for real-time noise reduction of a low-light camera based on a neural network according to claim 1, characterized in that: The method of quantifying the final model in step 4 includes: Use the MinMax method to determine the zero point Z and the scaling factor S: In the formula, R max is the maximum value of floating point data, R min is the minimum value of floating point data, Q max is the maximum value of a fixed-point number, Q min is the minimum value of a fixed-point number; Quantize a floating point number R to a fixed point number Q: R=(QZ)×S.
9. A real-time noise reduction device for a low-light camera based on a neural network according to the method of any one of claims 1 to 8, characterized in that: include: A data acquisition module, wherein the data acquisition module is used to collect noise data and clean data; A data synthesis module, wherein the data synthesis module is used to generate training samples according to the noise data and the clean data; A neural network module, wherein the neural network module is used as a lightweight neural network to perform noise reduction processing on the image; A model quantization module, wherein the model quantization module is used to convert a trained lightweight neural network into a fixed-point representation; The image processing module is a lightweight neural network represented by a fixed point, and is used to perform real-time noise reduction processing on an input low-light image.