Ultra-low illumination full-color night vision imaging method
Through the combination of deep learning convolutional neural network and generative adversarial network, the problem of poor image quality in ultra-low illumination environments is solved, and clear full-color high-definition images are obtained in extremely low-light environments, suitable for nighttime hours Surveillance and military vehicles and other scenarios.
Patent Information
- Application Number
- CN202410760496.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-13
- Publication Date
- 2024-11-12
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing night vision technology has problems such as low brightness, low contrast, noise, blur and glare in ultra-low illumination environments, making it difficult to obtain clear and recognizable full-color images in extremely low-light environments.
Deep learning convolutional neural network is used for exposure fusion processing, combined with median filtering and bilateral filtering for noise reduction processing, and convert RAW data into high-quality RGB images by generating adversarial networks, optimize the neural network structure to reduce model size and Accelerate processing speed.
It achieves the acquisition of full-color high-definition images in extremely low-light environments, improves imaging effects, reduces system power consumption and delay, and is suitable for night monitoring and military vehicles and other scenarios.
Smart Images

Figure CN118945485A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of night vision imaging, and in particular relates to an ultra-low illumination full-color night vision imaging method. Background Art
[0002] Night vision technology is a technology that uses photoelectric detection and imaging technology and equipment to convert (enhance) targets that are invisible to the human eye due to darkness into visible images. Existing night monitoring and night vision equipment mainly use infrared, laser and dual-spectrum imaging technology, but they generally have the following problems: 1. Supplementary light is required and the concealment is low; 2. The monitoring and detection distance is short, generally within 200 meters; 3. The image is fuzzy, the color is single, there is a trailing image, and it is difficult to identify; 4. It is easy to be damaged by strong light, the hardware is limited, and the cost is high. In recent years, one direction of night vision technology research and development is low-light thermal imaging fusion technology, which uses contour recognition to superimpose the two images obtained by low-light imaging and infrared thermal imaging, so that it can not only display low-light scenes but also identify hidden or camouflaged personnel or vehicles through thermal imaging; the second direction is to develop night vision devices that integrate other sensors, comprehensively display the data of various sensors, and can be used in severe weather conditions; in addition, there is also a type of low-light color night vision device, which uses the color information of visible light and the brightness contour information of infrared light to display low-light scenes like scenes with color information seen during the day, rather than just an overall green picture through infrared imaging. However, when the current night vision technology is applied to ultra-low illumination environments, there are problems such as low brightness, low contrast, noise, blur, glare, and partial overexposure of the image due to insufficient lighting, short exposure time, and uneven lighting. Summary of the invention
[0003] In view of the above shortcomings, the present invention discloses an ultra-low illumination full-color night vision imaging method, constructs a suitable deep neural network and algorithm for low-light night vision scenes, realizes reconstruction from RAW data to high-quality RGB images, and converts ultra-low illumination night vision images into clear and discernible full-color images.
[0004] The present invention is achieved by adopting the following technical solutions:
[0005] An ultra-low illumination full-color night vision imaging method comprises the following steps:
[0006] (1) Video image data is acquired through a CIS sensor, and then the video image data is subjected to exposure fusion processing using a deep learning convolutional neural network, where the deep learning convolutional neural network is represented as: merged =f CNN (I1,I2,...,I n ), where I1,I2,...,I n is the input image under different exposure settings, Imerged is the fused output image, f CNN A convolutional neural network model includes a plurality of convolutional layers, a pooling layer, an activation function ReLU and an upsampling layer;
[0007] The deep learning convolutional neural network is optimized by using neural network search, which specifically includes the following steps:
[0008] S11. Define a search space, where the search space includes a convolution layer, a maximum pooling layer, and a fully connected layer, where the number of convolution layers is 3 to 10, and the convolution kernel size is 3 to 5;
[0009] S12. Use the gradient-based method of the RNN controller to randomly sample the network structure, and then calculate the cross entropy loss function on the validation set for each selected architecture:
[0010] where y i is the true label, is the predicted label;
[0011] S13. Update the weights of the controller network using the policy gradient method according to the value of the cross entropy loss function, and then repeat the above steps until the network architecture with the best performance is found.
[0012] S14, then pruning the deep learning convolutional neural network processed by step S13 by using neuron pruning to obtain an optimized deep learning convolutional neural network;
[0013] (2) performing noise reduction processing on the image data after the exposure fusion processing in step (1), specifically, using a median filtering method to remove obvious impulse noise and salt and pepper noise, and then using a bilateral filtering method to perform noise reduction processing;
[0014] (3) The video image data subjected to the noise reduction process in step (2) is processed using a generative adversarial network, with the RAW image data being used as the input layer, and the output layer generating corresponding RGB image data and transmitting it to the display device.
[0015] Furthermore, the mean square error is used as the loss function in the deep learning convolutional neural network described in step (1).
[0016] Furthermore, in step (1), the optimized deep learning convolutional neural network is pruned using neuron pruning, which is based on the evaluation of the weights of neurons in the deep learning convolutional neural network, pruning 3% of neurons with the smallest weights, using the model parameters before pruning to initialize the pruned network, and fine-tuning it.
[0017] Furthermore, the weights of neurons in the deep learning convolutional neural network described in step (1) are obtained using a perceptron algorithm or a back propagation algorithm.
[0018] Furthermore, the weight of the bilateral filtering method in step (2) includes a spatial distance weight and a pixel value similarity weight; the spatial distance weight is expressed as:
[0019] [w_d(i,j,k,l)=\exp\left(-\frac{(ik)^2+(jl)^2}{2\sigma_d^2}\right)], where ((i,j)) is the coordinate of the neighboring pixel in the template, ((k,l)) is the coordinate of the center pixel of the template, and (\sigma_d) is the variance of the spatial distance weight;
[0020] The pixel value similarity weight is expressed as:
[0021] [w_r(i,j,k,l)=\exp\left(-\frac{|f(i,j)-f(k,l)|^2}{2\sigma_r^2}\right)], where (f(i,j)) is the pixel value at coordinate ((i,j)), and (\sigma_r) is the variance of the pixel value similarity weight;
[0022] The total weight of bilateral filtering is expressed as:
[0023] [w(i,j,k,l)=w_d(i,j,k,l)\cdot w_r(i,j,k,l)].
[0024] Furthermore, in the processing described in step (3), one or more combinations of binary cross entropy loss, least squares loss, gradient penalty loss and feature matching loss are used as loss functions.
[0025] Compared with the prior art, this technical solution has the following beneficial effects:
[0026] 1. Since deep neural network models are generally large, they are mainly reflected in the large number of parameters, large amount of calculation, and large amount of memory bandwidth usage. If not optimized, many deep neural networks cannot even process video data in real time on large servers. In order to enable deep neural networks to run in real time on terminal devices with lower computing power and power consumption, the present invention greatly compresses the adopted deep learning neural network and cuts off unnecessary convolution kernels to achieve the purpose of neural network pruning, thereby further reducing the size of the convolutional neural network and accelerating the speed of the neural network.
[0027] 2. The present invention adopts a method combining median filtering and bilateral filtering to perform noise reduction processing, thereby further improving the imaging effect. At the same time, the present invention adopts a generative adversarial network to process the data (RAW data) output by the CIS and reconstruct high-quality RGB image data, solving the problems of insufficient brightness, missing details, blur, low contrast, low saturation, color cast, etc. existing in the original RAW data.
[0028] 3. The night vision device configured based on the method of the present invention has the characteristics of fast speed, low latency and low power consumption, and can obtain full-color high-definition images in extremely weak light environments. It can be used not only in daily monitoring fields, but also in night surveillance of border and coastal defense and night driving of military vehicles, helicopters and drones. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 It is a flow chart of the ultra-low illumination full-color night vision imaging method described in Example 1.
[0030] Figure 2 This is the actual imaging effect of 2 kilometers in the night vision device test described in Example 1.
[0031] Figure 3 This is the actual imaging effect of 2 kilometers in the night vision device test described in Example 2.
[0032] Figure 4 This is the actual imaging effect of 5 kilometers in the night vision device test described in Example 2. DETAILED DESCRIPTION
[0033] The present invention is further described by the following examples, but is not intended to be limiting of the present invention. Specific experimental conditions and methods not specified in the following examples are conventional methods commonly known to those skilled in the art.
[0034] Example 1: Figure 1 As shown, an ultra-low illumination full-color night vision imaging method comprises the following steps:
[0035] (1) Video image data is acquired through a CIS sensor, and then the video image data is subjected to exposure fusion processing using a deep learning convolutional neural network, where the deep learning convolutional neural network is represented as: merged =f CNN (I1,I2,...,I n ), where I1,I2,...,I n is the input image under different exposure settings, I merged is the fused output image, f CNNA convolutional neural network model includes a plurality of convolutional layers, a pooling layer, an activation function ReLU and an upsampling layer; the deep learning convolutional neural network uses a mean square error as a loss function;
[0036] The deep learning convolutional neural network is optimized by using neural network search, which specifically includes the following steps:
[0037] S11. Define a search space, where the search space includes a convolution layer, a maximum pooling layer, and a fully connected layer, where the number of convolution layers is 3 to 10, and the convolution kernel size is 3 to 5;
[0038] S12. Use the gradient-based method of the RNN controller to randomly sample the network structure, and then calculate the cross entropy loss function on the validation set for each selected architecture:
[0039] where y i is the true label, is the predicted label;
[0040] S13. Update the weights of the controller network using the policy gradient method according to the value of the cross entropy loss function, and then repeat the above steps until the network architecture with the best performance is found.
[0041] S14, then pruning the deep learning convolutional neural network processed by step S13 by neuron pruning to obtain an optimized deep learning convolutional neural network, which is based on the evaluation of the weights of neurons in the deep learning convolutional neural network, pruning 3% of neurons with the smallest weights, using the model parameters before pruning to initialize the pruned network, and fine-tuning it, wherein the weights of neurons in the deep learning convolutional neural network are obtained by using a perceptron algorithm;
[0042] (2) performing noise reduction processing on the image data after the exposure fusion processing in step (1), specifically, using a median filtering method to remove obvious impulse noise and salt and pepper noise, and then using a bilateral filtering method to perform noise reduction processing, wherein the weight of the bilateral filtering method includes a spatial distance weight and a pixel value similarity weight; the spatial distance weight is expressed as:
[0043] [w_d(i,j,k,l)=\exp\left(-\frac{(ik)^2+(jl)^2}{2\sigma_d^2}\right)], where ((i,j)) is the coordinate of the neighboring pixel in the template, ((k,l)) is the coordinate of the center pixel of the template, and (\sigma_d) is the variance of the spatial distance weight;
[0044] The pixel value similarity weight is expressed as:
[0045] [w_r(i,j,k,l)=\exp\left(-\frac{|f(i,j)-f(k,l)|^2}{2\sigma_r^2}\right)], where (f(i,j)) is the pixel value at coordinate ((i,j)), and (\sigma_r) is the variance of the pixel value similarity weight;
[0046] The total weight of bilateral filtering is expressed as:
[0047] [w(i,j,k,l)=w_d(i,j,k,l)\cdot w_r(i,j,k,l)];
[0048] (3) The video image data subjected to the noise reduction process in step (2) is processed using a generative adversarial network. In the processing, a combination of least squares loss and feature matching loss is used as a loss function. The RAW image data is used as an input layer, and the output layer generates corresponding RGB image data and transmits it to a display device.
[0049] The image signal processor chip obtained by the method described in this embodiment is combined with a CIS sensor, an optical lens, and a display device to manufacture a night vision device, and its technical indicators can meet the following standards:
[0050] (1) Minimum light intensity that can be adapted: <0.0001Lux;
[0051] (2) System video image signal-to-noise ratio: 40dB (@0.01Lux, F1.0), 28dB (@0.001Lux, F1.0)
[0052] (3) System low-light video size and frame rate: 30fps
[0053] (4) System power consumption: ≤6W
[0054] (5) Adaptable temperature range: -20℃ to 60℃
[0055] At the same time, the night vision device was used for field testing. A certain place in Tibet (3797.628 meters above sea level) was selected for real-scene testing. The test environment was a rainy day. The test time was from 10 pm to 12 noon. The test results can be seen in the picture. Figure 2 .
[0056] Embodiment 2: A method for ultra-low illumination full-color night vision imaging, comprising the following steps:
[0057] (1) Video image data is acquired through a CIS sensor, and then the video image data is subjected to exposure fusion processing using a deep learning convolutional neural network, where the deep learning convolutional neural network is represented as: merged =fCNN (I1,I2,...,I n ), where I1,I2,...,I n is the input image under different exposure settings, I merged is the fused output image, f CNN A convolutional neural network model includes a plurality of convolutional layers, a pooling layer, an activation function ReLU and an upsampling layer; the deep learning convolutional neural network uses a mean square error as a loss function;
[0058] The deep learning convolutional neural network is optimized by using neural network search, which specifically includes the following steps:
[0059] S11. Define a search space, where the search space includes a convolution layer, a maximum pooling layer, and a fully connected layer, where the number of convolution layers is 3 to 10, and the convolution kernel size is 3 to 5;
[0060] S12. Use the gradient-based method of the RNN controller to randomly sample the network structure, and then calculate the cross entropy loss function on the validation set for each selected architecture:
[0061] where y i is the true label, is the predicted label;
[0062] S13. Update the weights of the controller network using the policy gradient method according to the value of the cross entropy loss function, and then repeat the above steps until the network architecture with the best performance is found.
[0063] S14, then pruning the deep learning convolutional neural network processed by step S13 by neuron pruning to obtain an optimized deep learning convolutional neural network, which is based on the evaluation of the weights of neurons in the deep learning convolutional neural network, pruning 3% of neurons with the smallest weights, using the model parameters before pruning to initialize the pruned network, and fine-tuning it, and the weights of the neurons in the deep learning convolutional neural network are obtained by using a back propagation algorithm;
[0064] (2) performing noise reduction processing on the image data after the exposure fusion processing in step (1), specifically, using a median filtering method to remove obvious impulse noise and salt and pepper noise, and then using a bilateral filtering method to perform noise reduction processing, wherein the weight of the bilateral filtering method includes a spatial distance weight and a pixel value similarity weight; the spatial distance weight is expressed as:
[0065] [w_d(i,j,k,l)=\exp\left(-\frac{(ik)^2+(jl)^2}{2\sigma_d^2}\right)], where ((i,j)) is the coordinate of the neighboring pixel in the template, ((k,l)) is the coordinate of the center pixel of the template, and (\sigma_d) is the variance of the spatial distance weight;
[0066] The pixel value similarity weight is expressed as:
[0067] [w_r(i,j,k,l)=\exp\left(-\frac{|f(i,j)-f(k,l)|^2}{2\sigma_r^2}\right)], where (f(i,j)) is the pixel value at coordinate ((i,j)), and (\sigma_r) is the variance of the pixel value similarity weight;
[0068] The total weight of bilateral filtering is expressed as:
[0069] [w(i,j,k,l)=w_d(i,j,k,l)\cdot w_r(i,j,k,l)];
[0070] (3) The video image data subjected to the noise reduction process in step (2) is processed using a generative adversarial network. In the processing, a gradient penalty loss is used as a loss function. The RAW image data is used as an input layer, and the output layer generates corresponding RGB image data and transmits it to a display device.
[0071] The image signal processor chip obtained by the method described in this embodiment is combined with a CIS sensor, an optical lens, and a display device to manufacture a night vision device, and its technical indicators can meet the following standards:
[0072] (1) Minimum light intensity that can be adapted: <0.0001Lux;
[0073] (2) System video image signal-to-noise ratio: 40dB (@0.01Lux, F1.0), 28dB (@0.001Lux, F1.0)
[0074] (3) System low-light video size and frame rate: 30fps
[0075] (4) System power consumption: ≤6W
[0076] (5) Adaptable temperature range: -20℃ to 60℃
[0077] At the same time, the night vision device was used for field testing, and a certain place in Tibet (4873.828 meters above sea level) was selected for real-scene testing; the test environment was completely dark, the sky was dark, it was rainy, and there was almost no light; the test time was from 9 pm to 11 pm; the test results are shown in the picture Figure 3 and Figure 4 .
[0078] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.
Claims
1. An ultra-low illumination full-color night vision imaging method, characterized in that: The following steps are involved: (1) Video image data is acquired through a CIS sensor, and then the video image data is subjected to exposure fusion processing using a deep learning convolutional neural network, where the deep learning convolutional neural network is represented as: merged =f CNN (I1,I2,...,I n ), where I1,I2,...,I n is the input image under different exposure settings, I merged is the fused output image, f CNN A convolutional neural network model includes a plurality of convolutional layers, a pooling layer, an activation function ReLU and an upsampling layer; The deep learning convolutional neural network is optimized by using neural network search, which specifically includes the following steps: S11. Define a search space, where the search space includes a convolution layer, a maximum pooling layer, and a fully connected layer, where the number of convolution layers is 3 to 10, and the convolution kernel size is 3 to 5; S12. Use the gradient-based method of the RNN controller to randomly sample the network structure, and then calculate the cross entropy loss function on the validation set for each selected architecture: where y i is the true label, is the predicted label; S13. Update the weights of the controller network using the policy gradient method according to the value of the cross entropy loss function, and then repeat the above steps until the network architecture with the best performance is found. S14, then pruning the deep learning convolutional neural network processed by step S13 by using neuron pruning to obtain an optimized deep learning convolutional neural network; (2) performing noise reduction processing on the image data after the exposure fusion processing in step (1), specifically, using a median filtering method to remove obvious impulse noise and salt and pepper noise, and then using a bilateral filtering method to perform noise reduction processing; (3) The video image data subjected to the noise reduction process in step (2) is processed using a generative adversarial network, with the RAW image data being used as the input layer, and the output layer generating corresponding RGB image data and transmitting it to the display device.
2. The ultra-low illumination full-color night vision imaging method according to claim 1, characterized in that: The method comprises the following steps: in the deep learning convolutional neural network described in step (1), the mean square error is used as the loss function.
3. The ultra-low illumination full-color night vision imaging method according to claim 1, characterized in that: In step (1), the optimized deep learning convolutional neural network is pruned by neuron pruning, which is based on the evaluation of the weights of neurons in the deep learning convolutional neural network, pruning 3% of neurons with the smallest weights, using the model parameters before pruning to initialize the pruned network, and fine-tuning it.
4. The ultra-low illumination full-color night vision imaging method according to claim 1, characterized in that: The weights of neurons in the deep learning convolutional neural network described in step (1) are obtained using a perceptron algorithm or a back propagation algorithm.
5. The ultra-low illumination full-color night vision imaging method according to claim 1, characterized in that: The weights of the bilateral filtering method in step (2) include a spatial distance weight and a pixel value similarity weight; the spatial distance weight is expressed as: [w_d(i,j,k,l)=\exp\left(-\frac{(ik)^2+(jl)^2}{2\sigma_d^2}\right)], where ((i,j)) is the coordinate of the neighboring pixel in the template, ((k,l)) is the coordinate of the center pixel of the template, and (\sigma_d) is the variance of the spatial distance weight; The pixel value similarity weight is expressed as: [w_r(i,j,k,l)=\exp\left(-\frac{|f(i,j)-f(k,l)|^2}{2\sigma_r^2}\right)], where (f(i,j)) is the pixel value at coordinate ((i,j)), and (\sigma_r) is the variance of the pixel value similarity weight; The total weight of bilateral filtering is expressed as: [w(i,j,k,l)=w_d(i,j,k,l)\cdot w_r(i,j,k,l)].
6. The ultra-low illumination full-color night vision imaging method according to claim 1, characterized in that: In the processing described in step (3), one or more combinations of binary cross entropy loss, least squares loss, gradient penalty loss and feature matching loss are used as loss functions.
