A low-light-level night-vision image enhancement method and embedded enhancer
By using a pre-trained enhancement model and an illuminance and reflectance component estimation network to adaptively enhance low-light night vision images, the problem of low image quality under dim light conditions is solved, and efficient and natural image enhancement effects are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HOHAI UNIV
- Filing Date
- 2023-03-31
- Publication Date
- 2026-06-02
AI Technical Summary
The image quality of low-light night vision images taken under dim lighting conditions is difficult to improve. Existing techniques such as high ISO, long exposure, and flash have problems such as noise amplification, blurring, and the introduction of bright spots, and there is a lack of effective image enhancement solutions.
The pre-trained enhancement model, including an illuminance component estimation network, a reflectance component estimation network, and an enhancement network, is used to adaptively enhance low-light night vision images by learning from a large amount of data. The enhancement model is lightweight and can be implemented on embedded devices.
The enhancement model can adaptively improve the quality of low-light night vision images, preserve original details, suppress noise, and has high processing capabilities, making it suitable for embedded devices.
Smart Images
Figure CN116342427B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a low-light night vision image enhancement method and an embedded enhancer, belonging to the field of AI image processing. Background Technology
[0002] Shooting high-quality images in low-light conditions is generally very difficult. While setting high ISO, long exposure, and using flash can improve image brightness, these methods also have their drawbacks. For example, high ISO increases the image sensor's sensitivity to light, amplifying noise and resulting in a low signal-to-noise ratio. Long exposure is limited to static scenes; it can cause blurring in non-static scenes. Using flash can illuminate the environment, but this often introduces unexpected bright spots and unbalanced lighting, degrading image quality. In recent years, low-light night photography has seen significant development, but there are still no effective solutions for enhancing low-light night photography. Summary of the Invention
[0003] This invention provides a low-light night vision image enhancement method and an embedded enhancer, which solves the problems disclosed in the background art.
[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:
[0005] A method for enhancing low-light night vision images includes:
[0006] Acquire low-light night vision images to be enhanced;
[0007] The low-light night view image to be enhanced is input into a pre-trained enhancement model to obtain the enhanced low-light night view image. The enhancement model includes an illuminance component estimation network, a reflectance component estimation network, and an enhancement network. The illuminance component estimation network is used to estimate the illuminance component from the input low-light night view image. The reflectance component estimation network is used to estimate the reflectance component based on the illuminance component and the input low-light night view image. The enhancement network is used to reconstruct the enhanced low-light night view image based on the illuminance component, the reflectance component, and the input low-light night view image.
[0008] The illumination component estimation network consists of two sets of first convolutional kernels, a global mean pooling layer, and two sets of second convolutional kernels connected sequentially from input to output.
[0009] The first convolution kernel is a 3×3 convolution kernel, and the second convolution kernel is a 1×1 convolution kernel. The number of first and second convolutions in each group is 3.
[0010] The reflection component estimation network includes a connection unit, three sets of dilated convolution kernels, and one set of first convolution kernels connected sequentially from input to output; wherein, the connection unit is used to connect the illuminance component and the input low-light night view image.
[0011] The dilated convolution kernel is a 3×3 dilated convolution kernel, and the number of dilated convolution kernels in each group is 8; the first convolution kernel is a 3×3 convolution kernel, and the number of first convolution kernels in each group is 3.
[0012] The enhanced network is a UET network.
[0013] When training the augmentation model, the training set includes pairs of indoor images and pairs of outdoor images. The pairs of indoor images include low-light night view images and normal lighting images taken in the same scene after image processing. The pairs of outdoor images include low-light night view images after image processing and corresponding low-light night view images without image processing.
[0014] An embedded enhancer includes an embedded microcontroller, which is loaded with a program corresponding to a low-light night vision image enhancement method, and the embedded microcontroller executes the low-light night vision image enhancement method.
[0015] When the embedded microcontroller loads the program corresponding to the low-light night vision image enhancement method, it converts the enhancement model to RKNN format and replaces the global adaptive average pooling in the enhancement model with convolution, activation function and average pooling.
[0016] When concatenating the outputs of the illuminance component estimation network and the reflectance component estimation network, the output of the illuminance component estimation network is upsampled.
[0017] The beneficial effects achieved by this invention are as follows: The enhancement model composed of an illuminance component estimation network, a reflectance component estimation network, and an enhancement network can automatically learn the parameters of the environment and equipment by learning from a large amount of data, thereby achieving adaptive enhancement of low-light night vision images; and the enhancement model of this invention is a lightweight model that can be implemented on embedded devices and has high processing efficiency. Attached Figure Description
[0018] Figure 1 A flowchart of a low-light night vision image enhancement method;
[0019] Figure 2 This is a schematic diagram of an optical imaging model;
[0020] Figure 3 To enhance the framework diagram of the model;
[0021] Figure 4 This is a schematic diagram of the structure of the illuminance component estimation network;
[0022] Figure 5 This is a schematic diagram of the network structure for estimating the reflection components. Detailed Implementation
[0023] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.
[0024] like Figure 1 As shown, a low-light night vision image enhancement method includes the following steps:
[0025] Step 1: Obtain the low-light night view image to be enhanced.
[0026] Step 2: Input the low-light night view image to be enhanced into the pre-trained enhancement model to obtain the enhanced low-light night view image. The enhancement model includes an illuminance component estimation network, a reflectance component estimation network, and an enhancement network. The illuminance component estimation network is used to estimate the illuminance component from the input low-light night view image. The reflectance component estimation network is used to estimate the reflectance component based on the illuminance component and the input low-light night view image. The enhancement network is used to reconstruct the enhanced low-light night view image based on the illuminance component, the reflectance component, and the input low-light night view image.
[0027] The above method employs an enhancement model composed of an illuminance component estimation network, a reflectance component estimation network, and an enhancement network. By learning from a large amount of data, it can automatically learn the parameters of the environment and equipment, thereby achieving adaptive enhancement of low-light night vision images. Furthermore, the enhancement model of this invention is a lightweight model that can be implemented on embedded devices and has high processing efficiency.
[0028] The optical imaging model of low-light night vision images mainly includes two parts of light source information: illuminance component and reflectance component. For example... Figure 2 As shown, the illuminance component contains the approximate outline and brightness distribution of objects in the image. This component does not originate from the radiation of the objects themselves, but rather from light scattering in the surrounding environment. Background scattered light is the main reason for the reduced contrast in low-light night vision images. The reflection component originates from the object itself, containing the image's detailed texture and representing its essential attributes. It includes all edge detail colors and, in insufficient light, can result in blurred details, color distortion, and increased noise.
[0029] Based on the above theory, it is possible to construct, as follows Figure 3The enhancement model shown uses a convolutional neural network to fit multiple components of the imaging model. Specifically, it includes an illuminance component estimation network, a reflectance component estimation network, and an enhancement network. The illuminance component estimation network estimates the illuminance component from the input low-light night vision image. The reflectance component estimation network estimates the reflectance component based on the illuminance component and the input low-light night vision image. The enhancement network reconstructs the enhanced low-light night vision image based on the illuminance component, reflectance component, and the input low-light night vision image. These three networks only need to achieve their respective objectives, and their structures can vary.
[0030] As an embodiment of the present invention, the specific structure of the above-mentioned illuminance component estimation network can be as follows: Figure 4 As shown, it includes two sets of first convolutional kernels, a global mean pooling layer, and two sets of second convolutional kernels connected sequentially from input to output; wherein, the first convolutional kernel is a 3×3 convolutional kernel, the second convolutional kernel is a 1×1 convolutional kernel, and the number of each first and second convolutional kernels is 3. The activation function for the convolution operation is the parameterized modified linear unit PReLU.
[0031] The advantages of PReLU over ReLU are as follows:
[0032] Mitigating the Vanishing Gradient Problem: ReLU's gradient is 0 when x < 0, causing some neurons in the network to fail to update, which exacerbates the vanishing gradient problem. PReLU, on the other hand, allows for a small slope in the negative interval, mitigating this issue. Stronger Expressive Power: PReLU can better fit the non-linear characteristics of data distributions and more accurately capture data details, making it better suited for complex, high-dimensional datasets. Greater Robustness: PReLU is more robust than ReLU, handling a wider range of data and providing more stable model training, thus improving the model's generalization ability.
[0033] In summary, PReLU can improve model performance. Using PReLU in neural networks can train models faster and more accurately, thereby improving the model's accuracy and robustness.
[0034] As an embodiment of the present invention, the specific structure of the reflection component estimation network can be as follows: Figure 5 As shown, it includes a connection unit connected sequentially from input to output, three sets of dilated convolution kernels, and one set of first convolution kernels; wherein, the connection unit is used to connect the illuminance component and the input low-light night view image; wherein, the dilated convolution kernel is a 3×3 dilated convolution kernel, and the number of each set of dilated convolution kernels is 8; the first convolution kernel is a 3×3 convolution kernel, and the number of each set of first convolution kernels is 3.
[0035] The aforementioned enhancement network can employ a UNET network. The advantages of using the UNet network for image enhancement are as follows:
[0036] Preserving Image Details: The UNet network can preserve image details and remove noise and other interference, thus improving image clarity. Therefore, UNet can be used to enhance the quality of low-quality images, such as medical images or low-resolution images. Learning More Complex Patterns: UNet can transform complex images into specific forms, allowing it to learn more complex patterns, such as artistic patterns or textures, which can be used to enhance artistic or photographic images. Relatively Simple Network Structure: The encoder-decoder structure used in the UNet network achieves very good performance with fewer parameters and computational resources. Therefore, UNet can achieve good image enhancement with short runtime. Adaptable to Different Tasks: The UNet network is suitable for various types of image enhancement tasks, such as denoising, super-resolution, image restoration, and image enhancement. Therefore, it can be applied to many fields, including medical imaging, computer vision, and graphics processing.
[0037] In summary, the UNet network is a powerful and flexible image enhancement tool that can enhance various types of images through effective feature extraction and preservation of detail information, as well as a simple and robust network structure.
[0038] The enhanced model of the present invention has the following advantages:
[0039] 1. Better preservation of original image details: Enhanced models can better preserve the details of the original image, which is conducive to a more natural and realistic image quality and a more acceptable visual effect.
[0040] 2. Higher Image Enhancement Performance: The enhancement model achieves higher image enhancement performance. This is because the design of the illumination component estimation and reflectance component estimation in the network can better integrate feature information, resulting in better image enhancement results.
[0041] 3. Greater robustness: Night vision images often contain a lot of noise, and the ability to handle this noise is key to good enhancement results. Enhanced models can better suppress noise and are more robust.
[0042] 4. High efficiency: The enhanced model's network structure can be implemented through full convolution, making it a lightweight model that can be implemented on embedded devices and has high processing efficiency.
[0043] The above model requires training before it can be used. To construct training samples, a total of six cameras were used in the image processing: two ordinary USB cameras, two large-aperture cameras, and two slow-shutter cameras. Among them, the images captured by the ordinary USB cameras were the darkest and had less noise. The large-aperture cameras could capture brighter images in low-light night vision scenes by increasing the amount of light entering the camera, but the images captured had more noise and varying degrees of color cast. The slow-shutter cameras increased the amount of light entering the camera by slowing down the shutter speed, but this also resulted in a lower frame rate, severe motion blur, and unclear details.
[0044] The shooting scenarios were divided into indoor and outdoor. During the shooting process, the ambient light was controlled to maintain the light intensity at 0Lx, 0.1Lx, 0.2Lx, 0.4Lx, 0.6Lx, and 200Lx, respectively. 0–0.6Lx represents a low-light night vision environment, and 200Lx represents a normal lighting environment. With the camera position fixed indoors, one low-light night vision image was captured simultaneously with one normal lighting image to create paired datasets later. Outdoors, low-light night vision images were captured directly at night, and subsequently processed manually using Photoshop to create paired datasets.
[0045] For outdoor low-light night vision images without paired images and indoor paired images with insufficient quality, manual processing is performed using Photoshop. Specifically, this involves adjusting brightness and contrast, exposure, curves, color levels, sharpening, reducing noise, and removing haze. These images are then used to create a paired training set for model training.
[0046] Based on the above processing, the training set includes pairs of indoor images and pairs of outdoor images. The pairs of indoor images include low-light night vision images and normal lighting images taken in the same scene after image processing. The pairs of outdoor images include low-light night vision images after image processing and corresponding low-light night vision images without image processing.
[0047] To further enrich the scene images, publicly available low-light night view image sets such as LOL and SID (all of which are paired images, similar to outdoor paired images) were added to the training set. Based on the training set, the model can be trained by adjusting parameters such as the number of convolution and pooling layers and the size of the convolution kernel, so as to achieve a balance between processing speed and processing effect, that is, to be as fast as possible and as effective as possible.
[0048] In application, the low-light night view image to be enhanced is acquired, and the image is input into the trained enhancement model to obtain the enhanced low-light night view image.
[0049] The above method is merely a theory. To apply this method, it needs to be further combined with an embedded microcontroller to construct an embedded enhancer. This embedded enhancer includes an embedded microcontroller, which is loaded with a program corresponding to the low-light night vision image enhancement method. The embedded microcontroller executes the low-light night vision image enhancement method.
[0050] The low-light night vision image enhancement method primarily involves loading the model. This model is designed and trained using the PyTorch neural network framework. PyTorch is a Python-first deep learning framework that not only achieves powerful GPU acceleration but also supports dynamic neural networks. If the model needs to be deployed to a microcontroller and accelerated via an NPU, it needs to be further converted to RKNN format. RKNN is the model type used by the Rockchip NPU platform, with model files ending in .rknn. Rockchip provides a complete Python tool for model conversion, making it easy for users to convert their self-developed algorithm models into RKNN models.
[0051] During the conversion process, directly converting the model from PyTorch format to RKNN will result in conversion failure and the model will not run properly. Therefore, it is necessary to first convert the model to ONNX format, and then convert it from ONNX format to RKNN format.
[0052] The model uses global adaptive average pooling, which is not supported by RKNN. If average pooling is used as a replacement, the same result as the source operator can be achieved by configuring parameters, but it is still not supported by RKNN. Therefore, the effect of the source operator is finally achieved by using three operators: convolution, activation function and average pooling.
[0053] Specifically, the original model used global adaptive average pooling to output a 1*1 feature map for illuminance component estimation. However, RKNN does not support this operator. Using standard average pooling with specified kernel_size, stride, and padding as a substitute is also not supported. Therefore, several layers of convolution, activation functions, and average pooling are used to downsample to 1*1.
[0054] After the model conversion was successful, the running results on the microcontroller were inconsistent with the running results before the conversion. After screening, it was found that the tensor splicing results were inconsistent. Therefore, when splicing the outputs of the illuminance component estimation network and the reflectance component estimation network, the output of the illuminance component estimation network was upsampled.
[0055] Specifically, the 1*1 feature map obtained from the illuminance component estimation needs to be concatenated with the 3*512*512 feature map from the reflectance component estimation network to obtain a 4*512*512 feature map. This requires expanding the 1*1 feature map to 512*512, which is simple in PyTorch, but it causes problems when converted to RKNN. Therefore, convolution is used to upsample to 512*512 before concatenation.
[0056] The aforementioned embedded enhancer is small in size, easy to carry, and applicable to a wide range of scenarios. Furthermore, the aforementioned embedded microcontroller can be a common microcontroller used in existing cameras, which is optimized for cameras and has greater practical value.
[0057] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0058] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0059] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0060] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0061] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of the claims of the present invention pending approval.
Claims
1. A method for enhancing low-light night vision images, characterized in that, include: Acquire low-light night vision images to be enhanced; The low-light night view image to be enhanced is input into a pre-trained enhancement model to obtain the enhanced low-light night view image. The enhancement model includes an illuminance component estimation network, a reflectance component estimation network, and an enhancement network. The illuminance component estimation network is used to estimate the illuminance component from the input low-light night view image. The reflectance component estimation network is used to estimate the reflectance component based on the illuminance component and the input low-light night view image. The enhancement network is used to reconstruct the enhanced low-light night view image based on the illuminance component, the reflectance component, and the input low-light night view image. The illuminance component estimation network includes two sets of first convolutional kernels, a global mean pooling layer, and two sets of second convolutional kernels connected sequentially from input to output; the reflection component estimation network includes a connection unit, three sets of dilated convolutional kernels, and one set of first convolutional kernels connected sequentially from input to output. The connection unit is used to connect the illuminance component and the input low-light night view image.
2. The low-light night vision image enhancement method according to claim 1, characterized in that, The first convolution kernel is a 3×3 convolution kernel, and the second convolution kernel is a 1×1 convolution kernel. The number of first and second convolutions in each group is 3.
3. The low-light night vision image enhancement method according to claim 1, characterized in that, The dilated convolution kernel is a 3×3 dilated convolution kernel, and the number of dilated convolution kernels in each group is 8; the first convolution kernel is a 3×3 convolution kernel, and the number of first convolution kernels in each group is 3.
4. The low-light night vision image enhancement method according to claim 1, characterized in that, The enhanced network is a UET network.
5. The low-light night vision image enhancement method according to claim 1, characterized in that, When training the augmentation model, the training set includes pairs of indoor images and pairs of outdoor images. The pairs of indoor images include low-light night view images and normal lighting images taken in the same scene after image processing. The pairs of outdoor images include low-light night view images after image processing and corresponding low-light night view images without image processing.
6. An embedded enhancer, characterized in that, It includes an embedded microcontroller, which is loaded with a program corresponding to any one of the methods of claims 1 to 5, and the embedded microcontroller executes any one of the methods of claims 1 to 5.
7. The embedded enhancer according to claim 6, characterized in that, When the embedded microcontroller loads the program corresponding to any of the methods in claims 1 to 5, it converts the augmented model into RKNN format and replaces the global adaptive average pooling in the augmented model with convolution, activation function and average pooling.
8. The embedded enhancer according to claim 7, characterized in that, When concatenating the outputs of the illuminance component estimation network and the reflectance component estimation network, the output of the illuminance component estimation network is upsampled.